Bug in rate limiting

Eric,

It sounds like you are doing this correctly and there is a bug somewhere on the network. I've reviewed our server logs, and we only see records of delivering 429 coded responses 767 times in the past month, all more than a week ago. So your hunch may be right, and your requests might be getting caught by our anti-DDoS filters.

In order to check, I'd appreciate your help:

• Can you provide the exact response including all of the headers, when you receive this error?
• Can you provide sample URLs, your IP address, or your custom user-agent string so we can filter out your specific requests and see what's happening?

Thanks!

It is very possible that it’s just my code. I will provide you with the IP address but I need to do it in a more private formate than a public forum thread. I have not implemented any custom user agent functionality yet so I cannot provide that and I do not save my request. 

What i I can tell you is that for my username I did not encounter these issues I only encountered them when I was downloading from a user who had a lot more games. 

I was downloading for my chess coach with the username TheButcher. If you search your logs for requests last night early morning that contained that username you might be able to find them. It through the 429 after my code got through about 2000 of his games. 

We looked at our logs, and based on your description, we believe we found your 43 requests between January 24th 2018, 00:59:51 and January 24th 2018, 02:02:38 (PST, California time, where the servers are), covering one request for the games archives list and then 42 requests for each archive listed there. Since this looks like a complete scan of his games archives and we replied with data and a "200 Ok" code for each of these, we think that the problem is between you and our servers, or in the code you are using. We scanned the CDN events for possible blocks on your IP (again, assuming the IP based on the above URL requests), and found nothing.

I think the most likely place to look now is your code. First, if you get this again, can you verify that you are receiving a "429 Too Many Requests" response? Many other response codes might give you zero results, so we should look at the response code directly to make sure we are treating this problem correctly. Also, any other headers in the response would greatly help us determine if there is something we missed on our side.

Some of the larger responses were 3Mb in size. It's possible your code ran out of memory processing these.

These larger responses also took over 3 seconds to generate on the server, so if your code has a timeout of 5 seconds or less then it may have just stopped listening before the response arrived.

The user-agent string is empty, and sometimes this looks suspicious to proxy servers in between you and us. Our CDN did not record such a problem, which implies that if something is blocking this empty user-agent as "suspicious" then it might be your ISP or a proxy it routes through. Customizing the user-agent string is super useful for debugging things like this, too!

the exception of as 100% a 429 I was debugging when it happened. I will customize the user agent and see if it helps. 

That's troubling, since it's not in our logs. When you received a 429, did you then request the same URL again until you received the data?

I'll see whether we can get the CDN to be less sensitive to things like missing user-agents, which is likely to be a common thing for many scripts people write.

You know what it might have just been my lichess client that threw the 429 and I might have just not been paying attention I added more logic last night to control how many requests I am making in one second and to retry the request on failure after waiting for a few seconds. I will retry and do more testing this weekend and give an update. Thanks for all the help! 

Thanks for the report, and please do let us know any time you suspect the API is not working as intended.

After investigating your situation, we're seeing that the current system is handling load very easily and so we expect to increase the number of permitted simultaneous requests very soon.