Convert Mega2009 to PGN?

Hi,

Has anyone who has CB Mega Database 2009 managed to convert the entire thing into a single PGN file?

I've tried 2 methods with CB10 - export to PGN and create new PGN +copy/paste - and both crash after about 60% of the export or import process. Export to text file doesn't work either, as it does not create proper PGN text.

I've tried Chess Assistant Lite it also can't handle making the PGN.

SCID can't do it either.

Notepad and Wordpad can't handle the quantity of text, so I can't use them to copy/paste.

I managed to make two PGNs containing 50% of the Mega Database 2009 each. So I need to combine them. I've tried every PGN utility I can find online, but they either don't work or they are DOS-based and I can't figure out how to use them. Using the command prompt and doing *.pgn > <new file name>.pgn produces a file which cannot be read by ChessBase or properly by SCID.

The merge PGN utility provided on this site under Downloads also does not work.

Can anyone either help me produce a single PGN file or at least correctly merge the two halves that I have?

Cheers!

For a text editor, you can download (free) Edit Pad Lite, I dont think it has a size restriction.

http://www.editpadpro.com/editpadlite.html.

Try opening the two pgn files in Openoffice.org, and then paste one into the other, and save as a text file with a pgn extender.

Thanks guys.

The OpenOffice didn't work. I did however find an excellent text editor that opened these huge text files very quickly: EmEditor. Unfortunately Windows couldn't handle copying the 56 million lines to the clipboard...

Eventually I found a file merge utility called Simple File Joiner which merged the two PGNs perfectly. However, the single file is still not recognized as a valid file by ChessBase10 and SCID crashed when trying to open it.

My reason for getting a large PGN is so I could do things in SCID that ChessBase can't do. For some reason SCID only reads PGN files and not CBH files.

I can get SCID to open a PGN database of around 2 million games. So that's better than nothing I guess :-)

So I got a mail from ChessBase and they told me that PGNs are limited by size, so I guess my proposed 4-million-game PGN breached that limit.

I'm now working on making my Mega2009 database even bigger. Will come back on that.

What is it that SCID can do that Chessbase can't?

I like the player reports better. And the pieces. And auto-replay. So far :-)

Thanks to a member of this site I have got ChessBase Mega2009 converted in its entirety into a single SCID database - yay!! :-)

Sure.

1) In ChessBase, make 2 equal-sized PGNs out of Mega2009 (then use the Check Integrity tool to make sure they are sound).

2) Close ChessBase. Open SCID.

3) Go to: Windows > (open) Database Switcher

4) Open part 1 of your Mega2009 PGN.

5) Make a new SCID database (.si4 file) and name it e.g. Megascid.

6) Drag-and-drop Mega2009 part 1 into Megascid.

7) Then from Database Switcher right-click on Mega2009 part 1 and close it.

8) Open part 2 of your Mega2009 PGN.

9) Drag-and-drop Mega2009 part 2 into Megascid.

10) Then from Database Switcher right-click on Mega2009 part 2 and close it.

11) Close the Database Switcher. You should now have Mega2009 as a single SCID database file! (file name: "megascid.si4")

I have found SCID searches in Mega2009 much faster than ChessBase10 does.

Is this PGNed database available for download somewhere?

Nope. You need to buy Mega2009 database from ChessBase first ;-)

I am not so sure that this is correct.

The database itself (the dataformat and its order) are undoubtedly covered by copyrights, but the information contained within the database itself is not - it is in the public domain.  The status of the collection of games is also probably of one in the public domain since it is largely composed of publicly available games.  I believe that If somebody therefore takes the Mega2009 database and exports it into the PGN format, he/she could safely make it pubicly available for download (if only because it would be impossible to prove that this collection of games was exported from Mega2009 or any other database )

Now, I understand that there are corporations out there who would want to sqeeze every last penny out of each one of us to maximze their profits, but we don't really need to buy into their arguments.

Besides, corporations need to come to terms with the reality that we are living in an Internet-connected world and start to revise their business models. This guy for example has, I think, a much smarter business model:

http://www.chessgameslinks.lars-balzer.info/data-information.html

He offers the service of a DVD with all the databases, but also makes all the databases available online.

As for Mega2009 is already available for download on the web anyway .

Considering all this, I would say that converting the Mega2009 into PGN (or, even better, the SCID4.0 format) and making it available to the rest of the chess players out there - in particular those who could not affod to pay such money to begin with! - is a terrific idea, no?

RN

PS: some good readings

http://www.chesshistory.com/winter/extra/copyright.html
http://www.chessvibes.com/columns/copyright-on-chess-moves-shadows-on-the-wall/
http://www.chesscafe.com/text/kibitz17.txt

Megabase includes annotations by famous players which are presumably under copyright.  Also, Lars Balzer does not make all the databases available online.  Many of the web sites where he got his data from are now defunct, and as far as I know, his web site does not host any pgn data at all.

@rigamagician: I have to admit that I did not check whether all of Lars databases were online .  However, I do presume that he does not make a claim to the ownership of those now defunct missing from his website.  The point I was trying to make is that I think that charing users for a service (a DVD) or a technoloy (a database structure) is legally speaking legit (morally - I think that nothing should prevent people from sharing anything and everything they possibly can share, but that's me).  In an ideal world I wish CB made their full collection available for download each year for free as a sign of support and as a service to the community. I would gladly voluntarily donate a litle something to keep this service going on each year.  What I do *not* like to hear is that I *have* to pay before being able to access this info.  I would not even agree to pay 1 symbolic dollar for Megabase, on principle (but that is me, again, I am a little weird I have to admit ).

@Gonnosuke: has CB actually paid each player for his/her annotations or have they collected them from public sources?

Chessbase pays grandmasters such as Robert Huebner for their annotations.  The value added to Megabase also comes from their standardization of names and such.  There are a number of large database archives available for free: Jose, Chess Analysis Project, Icofybase, ChessDB, 1.74 Million Base, etc.

Fair enough.  Point take.  Thanks for these explanations!

Can you please expand more on the Chess Analysis Project? I am familiar with the other bases but not with that one.

Is the analysis based on collected comments by analysts or is it an engine dump?

I don't like engine dumps, imho the most useful annotation is that by a good human analyst with many comments like x is also interesting y variation is unclear etc.

My problem with that approach is that an engine simply cannot find all the ideas & plans tried by humans during the last 60 years in a given position during the x seconds it is evaluating the position. Furthermore I doubt in general that engines can spot deep plans no matter how much time they run, like e.g. the Karpov-Nunn plan in Be2 Najdorf (Q-side pawn advance).

 Also, again imho, a centipawn value tells very little about the position unless is is like +2.00 or -2.00.

However these are the sort of games that have been immensely useful in my opening preparation, looking at engine centipawn vals instead of following the great variety of plans humans have tried is like castrating imagination in opening play.

To add on top of that, I am pretty disappointed by informant's laconic annotations, I wonder what sort of publication is best for some good *human* analysis in recent games (not a personal opinion but I have heard people complaining that nowadays NIC is also an engine dump). If you have any suggestions I'm all ears.

I was referring to the Chess Analysis Project mainly as a huge source of PGN games.  I agree that the assessments are pretty subjective to say the least.  If you don't like Informant and New In Chess, it's hard to make constructive suggestions.