I don't really think this is an issue. Most testers are careful to use small and neutral books, and they should turn off all engine books. Just pay attention to the testing conditions for any rating list that you're reading.
Best analysis engine/ test suites rankings
Sort:
I dont know how they test the engines, but on this site you can get some testsuites http://www.rybkachess.com/index.php?auswahl=Downloads
The results on this page are very old. You can tell by the engine names. (Houdini 1.5a, Stockfish 1.8, Critter .8, etc.) But the site does offer 13 or so different test suites to download.
It would be cool if someone could update the results with today's engines.
Here's another page for test suites.
Thanks for the link. MrEdCollins. The results are a bit dated but still offer a pretty good indicator if we assume the programs at the top didn't get much worse.
I agree it would be good to see these sorts of rankings more often though.
I think GM nunn or someone also had a set of positions that different programs would start playing against eachother from. The positions were often after several book moves. The programs couldn't use book moves and both programs would play the black and the white side against eachother so there was no favoritism.
These sorts of tests are much more valuable to me for what I use engines for than matches where the opening book might have a big impact.
If you're looking for opening test suites, here's some to get you started.
http://computer-chess.org/doku.php?id=computer_chess:wiki:download:index
And some information:
http://chessprogramming.wikispaces.com/Test-Positions
I found another large opening test suite that might interest someone. I'm not sure if you can really call it a test suite, as it doesn't have best moves - maybe more of an opening book of positions. Here's a Talkchess thread about the 8moves_GM.pgn suite. It looks like the Stockfish testers are going to start using this, except that the suite contains some bad moves that need to be culled out. If you go to the Google Groups link provided, you can read about what was culled out, and there are links to the cleaned up versions.
You can just download the 8moves_v3_epd.zip or the 8moves_v3_pgn.zip file and use that (34,700 positions), or you can download the book.tar file (gz extension) and manually combine the sf_good.epd and sf_bad_critter_good.epd files to create a larger suite (44,624 positions).
http://www.talkchess.com/forum/viewtopic.php?t=48874
Its easy to find which program is rated the highest but this may often be due to the opening book they select.
Computer chess enthusiasts used to test engines based on certain starting positions without opening books. Do they still do things like that? If so where can I find the data?
I think this might be a better way to find which engine is the best for analyzing my games.