Building Chess.com: Part 3 - Growing Pains

Well, we had a few really interesting things happen in the last few days, so I thought I would post on them!

Today when we woke up one of our servers was acting funky. It was up and running, but it wasn't allowing us to login. We weren't sure what had happened. We thought is was either some mysterious glitch that would be fixed by rebooting, or it was one of our new and lesser-known programmers doing something evil! After the machine was rebooted, it turns out it was neither, but still something nefarious - the box was attacked with a scripted attack meant to bring the site down. Basically this means someone programs a bunch of machines to overload the site with requests and that overwhelms the hardware and it fails. Much like what can happen to us when we login to Outlook in the morning and see 500+ emails... system failure. Anyway, it didn't cause any damage, and it failed to bring the site down (mostly due to Igor's amazing setup work). Still, we're now preparing for greater redundancy on the site.

Another interesting thing from today was that we had a scheduled hardware outage. It was supposed to be for 60 minutes while we added some new ram and additional CPUs to two of our servers. Why does it take 60 minutes? Well, in part because you have to shut them down and let them cool off before you can get inside! So, power down, install chip, and.... the screen reads: CPU MISMATCH! What??? No. We ordered the right stuff (we are using Dell PowerEdge 1950s. Must be something else. So I quickly call Dell and it turns out it was a "stepping" issue, where the processors were the same KIND in every way, but were different versions made on different dates, etc. Somehow that made it fail. So what we did instead was take the two processors from the original servers (we were upgrading 2 identical machines) and put their processors together, and then the two new processors in the other machine, and then we got the site back up. But that was more like 2-3 hours later! (By the way, we added 3 hours on to all chess games in progress!) Oh great, right as I am in the middle of writing this, the site goes back down. Something is wrong. And, I don't know quite what it is yet. Ah, ok, we just got it figured out - APC was just acting up (APC = Alternative PHP Cache). So now the site is back up for good (fingers crossed and knocking on wood).

Ah, there is another story that happened last month that I didn't report that is pretty interesting. One morning I woke up (yes, things ALWAYS go bad when I am asleep, which isn't enough mind you...), and there were 200+ spam reports in my inbox, meaning that somebody had been spamming members on our site! So I looked into it and found that they were spamming our site to promote another chess website. They had sent a few hundred messages. So I contacted that site and they flat-out denied it. Being the nice and trusting guy I am, I moved on figuring it would be over. Next day, SAME THING! But even MORE spam. The sad thing is that he had to put in the CAPTCHA each time he sent a message (you know, type those graphical letters on the colored background to make sure he is human). He literaly did this for HOURS. This time we do server log queries and find out the the spammer had used 2 different accounts that were both registered within 5 minutes of, and using the same IP address, as the owner of the other chess website. I confronted him and he STILL denied it, even though the evidence was there. And then he tried to block my IP so that I couldn't even get to his site (which didn't work, but hey, nice try). Anyway, his site never took off at all the the clown recently sold the site for a few bucks. But what a slimeball - a spammer, a liar, and a sellout all in one! I guess as Chess.com grows we will continue to attract attention - both good and bad.

Well, back to the grindstone. We are working on some really cool products coming out soon and I need to design what the interfaces will look like. We also are getting some more hardware soon (one being a beefy quad-core with 8gb ram), so I need to spec out those servers and then put the squeeze on my Dell rep for a good deal (we may start leasing...). Finally, I need to write the business plan so I can raise a little money to keep this ship sailing!

Wish us luck!

- Erik