I just now needed to shut off access to the API for an IP address that was attempting to pull all PGNs, for all games, apparently for all players. This is decidedly against the goals of the API, which is to help developers build tools that help players enjoy and learn more from their games on chess.com. I hate blocking people, but I need to protect the site.
Downloading all games is foolish — even at the max rate limit of 3 archives per second, this would take 3 years of continuous operation... during which time enough games would have been played to require another 18 months to download. During which time enough games would be played... It might never end.
The person who created this Python script did not create a user-agent string that allowed us to contact him. I want to call attention to this text in the documentation:
In some cases, if we detect abnormal or suspicious activity, we may block your application entirely. If you supply a recognizable user-agent that contains contact information, then if we must block you application we will attempt to contact you to correct the problem.
Maybe it wasn't clear that downloading the whole library is a bad idea, but we couldn't contact this developer to explain that and work out a solution. What if we made it more clear?
Developers with working code, let's discuss some options. Would this work for you?
We establish a total limit per day. The limit should be about 3–5x higher than anyone currently uses for their programs.
This could be a limit on the total number of requests (which might penalize applications that consume a lot of small endpoints frequently), or the total bandwidth (which might penalize downloading large archives). Which is better?
We would place the information about your daily allowance in the response headers. When you get to 90% of the limit, a 429 "Too Many Requests" response will be delivered 50% of the time; at 100%, a 429 response is delivered to every request until the next day
In this way, you will know what's allowed and when you are getting close. If this happens often and you are building tools for the Chess.com players, then we can work with you to get special, higher rate limits.
What do you think? Comments or ideas welcome!