SCID vs PC

Sort:
EscherehcsE
SkyMarshal wrote:

Look at this thread in other site

http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=7852

they complain about the same exact number in which my computer fails to continue. If it was due to 1GB, every time I try it should stop at a different number, very close but different, because the processes that are running at that moment. I dont think it has anything to do with that. The problem is that I cant figure out what it is! 

I think you may be right about this. In my 2.5 million game database, there weren't very many different round numbers (much less than 262 thousand). So I guess you'll have to keep the round number below the limit if you're going to use Scid vs. PC. Either delete the round number tags, or renumber them with fewer numbers, idk. (If the round numbers aren't important, many times people will just put a question mark in the round number tag field.)

NightKingx

I dont really know why the number of the rounds affects the program, but since computers are impossible to fully understand, I guess that changing the number of the rounds is a solution. But how can I delete at once, and fast, the information about the rounds in 2,6 million games? 

Maybe you are wondering why I am not using Arena. Well, I have been using it for some time now, but to do what I want to do: I have the pgn and I have to tell it to convert it to a book, which would take around 4 days. And then, once created, since the book contains so many games/moves, it is extremely slow and the program tends to crush. I dont understand why that happens, because as said, in Lucas Chess this is instantaneous. If you know a way to use Arena in the way I wanted to use Scid vs pc, that will do just fine :) 

MrEdCollins

By the way, I don't recommend Notepad.  It's very basic and very limiting and yes, it doesn't open large files.  Instead, I recommend something like TextPad for all of your text editing needs.   In addition to TextPad, there are lots of other good text editors out there, many of which are free.  They have lots more options and features than Notepad.

I have pgn file of 4.8 million games and TextPad opens it just fine.  Now, with so little RAM on your system you may have a problem opening large files with ANY text editor.  (I have 14 GB of RAM on my system here at home.)

If you have any computer programming skills you could write a short little program to edit your pgn file, and remove the ROUND tag or edit it or whatever.  An Excel macro could do it, for example.  This way you wouldn't have to worry about opening it at all.

HGMuller

I think this is a known limitation of SCID / SCID vs PC. The PGN format has a "round" tag, and the compressed format of SCID databases store the round number in 18-bits. That limits it to 2 to the power 14 = 256K.

So it is not so much the total number of games in the database that is limited, but the maximum round number. As soon a SCID encounters a round number larger than 256K it will choke on it, because it cannot encode it. If you do a million games that all have round numbers below 100, there is no problem at all.

NightKingx

MrEd: If I had programming skills, I would write a chess program that can satisfy my needs Laughing. Just what I need, doesn't look too complicated Wink but I am afraid I know nothing about it.

HGMuller: Yes, you are right. It is strange that there are games with a round number bigger than 256k isnt it? I have opened the pgn and I see that there are too much games with an round number bigger than that number. If I dont find a solution, I guess I'll have to erase a quarter of the databse...

EscherehcsE

OK, I think I have a solution, as long as the Round numbers don't mean anything to you. Norm Pollock has a set of pgn command-line utilities named 40H-PGN Utility Suite. One of them is named tagNull, and it will set all occurrences of a specified tag name in a pgn file to its null value (a question mark in this case).

Here's Norm's web page:  http://www.hoflink.com/~npollock/chess.html

 

Download the file named "40H-PGN-20151002.7z". Notice that it's in 7-zip format. You can extract the archive using 7-zip, available here:  http://www.7-zip.org/

 

After extracting all of the individual 40H-PGN files, copy your pgn database into that folder. Then open up a DOS command prompt and change the directory to the folder with the 40H-PGN files and your database.

Then type the command:

tagNull UltraBook.pgn Round

then press Enter. It may take 5 to 10 minutes or so to do its business. When it's done, all of the Round tags should look like this:

[Round "?"]

 

You can use GigaEdit to look at the modified pgn file. GigaEdit is the only free text editor I know of that can handle super large files without taking forever to open.

https://gigaedit.codeplex.com/

 

Oh, one last item to mention. I think Norm Pollock's Readme file cautions against deleting the Round tag, so it's better to just change them to the null value.

NightKingx

I thought that would work, but at the moment of writing the command, it says it is not a valid win32 application. So I thought the problem with that is my computer which is running on a 32 bit OS. I tried it in a 64 bits, but the message was the same. What I am going to do is, keeping intact the original book, create a copy an operate with that one. I am going to delete the games with the bigger round number and it will work. Of course I will lose some games, but it is the best I have been able to come up with.

EscherehcsE
SkyMarshal wrote:

I thought that would work, but at the moment of writing the command, it says it is not a valid win32 application. So I thought the problem with that is my computer which is running on a 32 bit OS. I tried it in a 64 bits, but the message was the same. What I am going to do is, keeping intact the original book, create a copy an operate with that one. I am going to delete the games with the bigger round number and it will work. Of course I will lose some games, but it is the best I have been able to come up with.

I'm not sure what could be causing that error message. It seems you've ruled out OS incompatibility (or at least you tried it on a 64-bit OS). It could be hardware incompatibility, except you tried it on two PCs.

You could check the hash of the tagNull.exe file to see if it's corrupted. Right-click on the tagNull.exe file, then click on "Properties", then "Checksums".

My MD5 hash is:

8b334b64308c68ab902b1b6f053b4c85

And my SHA-1 hash is:

a529d0856b67a8bac1fe8f67b77399f9615edb8a

 

If your tagNull.exe file isn't corrupted, then I guess I've run out of ideas. Frown

NightKingx

When I downloaded the 40H-pgn program, it was in a 32 bit computer, and then, that same download is what I used in a 64 bit computer. I mean, I didnt downloaded it again from the other computer. Maybe the site detected it was a 32 bit and gave me the 32 bit version, and then it didnt work in the 64 because it was still the 32 version I was running. I dont know if that works like this, but even if it would, that wouldnt explain why the donwload gives a 32 bit program that doesnt work in a 32 bit computer. Anyway, I am creating now a new pgn without all the games that exceed the round number limit, since deleting them from the pgn I already have seems impossible for both computers (maybe the problem is arena, I dont know, but I rather create the pgn again.). 

I guess you like challenges Tongue Out. Thanks for your help, I'll see if when I get the new pgn I can actually make it work with Scid.

EscherehcsE
SkyMarshal wrote:

When I downloaded the 40H-pgn program, it was in a 32 bit computer, and then, that same download is what I used in a 64 bit computer. I mean, I didnt downloaded it again from the other computer. Maybe the site detected it was a 32 bit and gave me the 32 bit version, and then it didnt work in the 64 because it was still the 32 version I was running. I dont know if that works like this, but even if it would, that wouldnt explain why the donwload gives a 32 bit program that doesnt work in a 32 bit computer...

I *think* that 32-bit programs should still run in 64-bit machines. Anyway, when I ran tagNull on my 64-bit machine, my Windows Task Manager said that tagNull was running as a 32-bit application.

Anyway, good luck with your project. Smile

NightKingx

Kind of good news. I have now my database in Scid format. I had my computer combining the pgns I wanted in Lucas Chess, and after 12 hours it hasn't finished yet. At the beggining it started pretty well, 200 000 games in half an hour, but then it started to slow down for no reason and it became at the ratio of 100 000 per hour. I decided to try to merge the png files with the DOS console, and it took it like 15 seconds, lol. Then I created the Scid File too and it is true that it doesnt take 25 minutes to load in Scid, the difference is that for every move I make it does take it 1 second or 2 to catch up instead of doing it instantly. I guess I'll have to buy a new computer if I want it to run faster. It works better than Lucas Chess and Arena, so I'll be using it.

Thank you all for your help and support Laughing

MrEdCollins
SkyMarshal wrote:

Thank you all for your help and support 

EscherehcsE certainly deserves most of the thanks. 

If you haven't seen it already, be sure to see my tribute page for more Scid vs PC tips.