1.3 million master games

MrEdCollins

Often I see players looking for or requesting chess databases.  With just a little bit of work, you can download and create your own chess database, containing more than 1.3 million master games.

Each week, The Week In Chess website puts together the latest master games from around the globe.  For example, the very latest file (twic973) contains more than 3,000 master games.  Prior week files can be downloaded from the twic site, or from other sites.

I noticed The Week In Chess website has the past 74 files still available to download.  (900 - 973)  I did a Google search and found these websites that contain prior twic files:

Files 211 - 400
Files 400 - 779
Files 739 - 964
Files 900 - 973

When you visit these sites, you will almost certainly want to download thes files in batch mode.  (Unless you're retired with nothing to do and a LOT of time on your hands.)  The wonderful Firefox add-on utility called DownThemAll makes downloading multiple files a snap.

One suggestion is to download all of the games in PGN format, since whatever GUI / database you're using, it most certainly can handle PGN files.  Download all of the PGN files into one folder and then use the old DOS copy command to create one, large PGN file:

copy *.pgn all-in-one.pgn

The above command will create a single file called all-in-one.pgn containing the games from all of the pgn files in the folder.

Once you have all of the game in a single pgn file, you can then load this file into your favorite chess GUI  (Chessbase, ChessDB, Scid vs PC, etc.)

For those of you who use the wonderful database Scid vs. PC, I suggest you use the included command-line utility "pgnscid.exe" to convert this large PGN file into a much smaller file, (a binary format) that Scid can read.  The binary file will load MUCH faster.  Using my example above, the syntax to use the utility would be:

pgnscid.exe all-in-one.pgn

This utility will create an "all-in-one.sg4" file from your all-in-one.pgn file.  That's your new database!

Together, all of The Week In Chess files mentioned above, (211 - 973) will give you more than 1.3 million chess games.  (Actually, it will give you more than this, but after I asked Scid vs PC to remove all of the duplicates, my result is about 1.3 million games.)

Enjoy.

DrSpudnik

I have always wondered what good it would do anyone to have so many games. You can't go over more than a few hundred, and quantity would just about swamp quality, making study odd.

Scottrf

It's not about looking at them all though, just having sufficient games in different lines you want to look at without downloading each time.

That and the statistics you get.

MrEdCollins

Scott's correct.  One can never look at all of them and that's not the point.

Let's assume I'm interested in studying the French Defense.  Maybe lately my opponents all seem to be playing the Exchange Variation, so I'm interested in viewing games with that line.  With a database of a few million games, I can then filter it and ask it to, for example, pull up all of the French Defense Exchange Variation games, played within the last few years, where Black won, in under 30 moves.

By then stepping through these games, I can get some general ideas of what I should be doing, where my pieces should go, etc.  I can also look at statistics.  How often does Black win or draw when move x is played, vs move y?

Or maybe in working on my endings, and I'm currently interested in rook and pawn endings.  I can filter the large database and just pull up all of the games that reached such an ending.

ollave

One or two of those URLs aren't quite right; a couple include chess.com as an unwanted prefix. The links below should work, and are also present as text for your cutting & pasting pleasure.

Files 211-400 http://www.chess.co.uk/twic/twicp1.html (**)

Files 400-779 http://www.chess.co.uk/twic/twicp.html (*)

Files 739-964 http:/www.tandilajedrez.com.ar/twic/

FIles 920-973 http://www.theweekinchess.com/twic

Enjoy. Or ignore. Betcha more of you download them than ever use them!

(*) Links to individual .pgn files not working. I don't know if this is a permanent thing or not; they redirect and the site they redirect to doesn't have them in the specified location. Maybe they got taken down; @MrEdCollins were you able to download them?

[ Edited to correct files present at last URL; they start at 920, not 900. ]

[ Edited again for the (*) note above. ]

[ And again for (**). ]

MrEdCollins

Excellent eye.  Thanks.

It took several tries (chess.com wasn't allowing me to edit them.  I'd edit them but the edit didn't take) but I finally fixed it... I think.

Thanks again.

NimzoRoy

I've downloaded TWIC from #400 up to update my CB BIG DB and recently discovered some other DBs I merged into the BIG DB from several online sites contain a fair amt of "GM draws" games so short as to be meaningless, duplicate games and "games" with no moves or corrupted files, So I've tried to "fix" my BIG DB by using tools to at least remove duplicate games as well as corrupted game files. 

I'm not saying TWIC has any of these problems but since each issue has anywhere from several  hundred to several thousand games it couldn't hurt to scan each file for errors and duplicates if possible. Offhand I think TWIC is pretty reliable and now I'm sorry I added any other DBs to my CB BIG DB since some of them are far from reliable - but hey they were free so who am I to complain? Just saying...

BTW Mark Crowther is a great guy, he must be to make TWIC available for free. I told him (well emailed him)  I'd be willing to pay $20/yr to continue getting TWIC every wk after he asked readers awhile back if they'd be willing to pay an annual fee - although it's still free so far. I think he also mentioned he might just quit doing TWIC altogether but this was awhile ago so my memory may be fuzzy here.

NimzoRoy
DrSpudnik wrote:

I have always wondered what good it would do anyone to have so many games. You can't go over more than a few hundred, and quantity would just about swamp quality, making study odd.

Then again you're making presumptions about stuff you haven't actually done (it sounds like) such as using a multi-million game DB. I started using CB BIG DB 2011 a few years ago and it's great for looking up openings and getting a feel for what's popular, what isn't, what looks like it works and what doesn't by going thru the data fields inc. average rating of players using an opening, variation, system etc; their average performance rating based on their results with a specific line, the highest-rated players to employ a line, the W-L-D statistics of a move,  the last date it was played (ie in a major tnmt) and so on. It's also really handy once you start getting further and further away from "main lines" down into obscure moves, that's when I start looking at entire games - when it appears that  my opponent is following out a "role model" game (my term here) move for move - which I sometimes do myself until I decide to try an "improvement" or my opponent does

Most of my opening books are way outta date, my "latest" ECOs are 2nd ed (they're up to 4th and 5th editions now) so basically the DB is my own "ECO" and for opinions, evaluations etc I go to MCO-15 (2008) and/or NCO (1999) but keep in mind even ECO is far from "complete" a lot of footnote moves in ECO have been played in hundreds of IM and GM games not to mention lines  that aren't even represented by one FN - but sometimes by several hundred or more games in a DB.

On the other hand, I recall that until very recently we all got along without DBs because there weren't any, so it's your call: You seem to be doing fine as it is from looking at your online rating. You could download SCID a free DB and try using it or seeing if it's worth the trouble of learning how to use it or not. I didn't know about until after I bought a used copy of CB 9 which I definitely am getting my money's worth out of or I would've tried it myself first.

ghillan

sorry to open an old thread but i miss the following twics

from 1 to 209

and from 817 to 854

in PGN formats. could someone point from where i can get them? I googled a lot , but was unable to find them.

MrEdCollins

I've never ever been able to find 1 to 209 in a pgn format.  One of the large databases I downloaded awhile back claimed they were all there in that database, but I guess I had no way to know for sure.

You can download 817-854 here, at my website.  I compressed them just a moment ago and created that file for you, with the wonderful 7 zip utility.  Let me know when you have it, so I can then delete the file from my server.

ghillan

TNX!

DunnoItAll

1-209 are definitely in mine if you downloaded it.

(which you can now get from here: http://chessdownloads.weebly.com/scid-vs-pc.html)

ghillan

@dunnotall. Tnx but  i wanted the weelky PGN files, not a merged file in some proprierary format.

@MrEdCollins: files downloaded, but triing to unzip some files i discovered that 2 of my files are corrupted. Could you (or anyone else )be so kind to share also 892 and 901? tnx in advance.

DunnoItAll

That's fine.  I was just letting Ed now that 1-209 were definitely in the file he got from me since he was not sure.

MrEdCollins

http://www.edcollins.com/twic892.pgn
http://www.edcollins.com/twic901.pgn

ghillan

Done, tnx Ed.

nameno1had

I thing I noticed about Arena's database and elo estimation is that it needs a broad spectrum of rated games all from the same type of chess, to accurately give elo estimates. That is a good enough reason for me to want a great number of games, and even games of players that arent necessarily all that good....

GrahamAstor

Thank you for posting (and correcting) these links. However, all I get is a redirect to Chess & Bridge - which seems a likely place to find these pgn files; but how do I actually get to the download areas, e.g for Files 211-400?

Many thanks in advance,

    Graham....

ghillan

chess & bridge had an archive of chose stuff bit not anymore. those olf stugg are no longer available in official sites, but you can find them in iternet if you look accuratelly. maybe someone here will be able to post a link for you.

MrEdCollins

Graham, if you send me your e-mail address in a private message, I will compress and e-mail you files twic211.pgn through twic400.pgn.

And yes, you're correct... the original link I posted ten months ago has changed... that site no longer has them.