Yes Giles, I missed the reference to the 429 response code in the OP - I should read things more carefully! But I only ever send endpoint requests sequentially & that's more than satisfactory for my needs. I'm not sure how I'd set things up in Python to generate requests concurrently - how do you?
I always work on the principle of abundance of caution & flag up in my scripts all responses other than 200, which indicates a successful one. So I'd certainly see a 429 if it was returned.
@WhiteDrake wrote:
User-agent strings probably should start with that if they are web browsers rendering HTML.For example, many search engine spiders do not include the "Mozilla": https://developers.whatismybrowser.com/useragents/explore/software_name/googlebot/
For our purposes of custom clients consuming the API, something more descriptive is helpful. We use this string to group the requests in our logs, so we can see if any trouble we are tracking is coming from one source, and if so then it allows us to contact the owner.
You are right off course. I dind't mean it seriously, just a silly joke. Sorry. 
@bcurtis But maybe you could tell us, what particular information would be helpful to you in your logs. I mean, if one accesses the API via a tool, then it's the tool that sets the User-Agent string (although even that cans ometimes be configured). But when accessing the API via a script, it is possible to put anything into the User-Agent header. What should ideally be there, if the author of the script wants to be helpful?
...
Skelos, you also wrote:
Can you explain what? In my view, the only time scraping would be faster would be if the data were packaged for your specific use in a more appropriate way. The API responses from the server are typically less than 1ms if the data are cached, and 20ms if the data need to be pulled from source. The average www.chess.com server response time is 65ms. In both cases, the bulk of the time you see is network time, which can be 100–200ms each way. You would likely get a 5–10x speed increase if you ran your program from a server in Los Angeles.
I think that we can increase the number of simultaneous connections to 5. Hope that helps.
...
In the past I have compared Australia, Australia using a VPN, and Pittsburgh (a site with excellent Internet connectivity and under-utilised incoming bandwidth vs outgoing, as it's a web hosting provider). I saw no effective difference. I'll try agin when convenient. My impression at the time was that the data I wanted had to be cached by api.chess.com before delivery to me, and most of my requests were new.
As for web scraping, some of the web pages provide more data than a single endpoint. If one web page gives me everything four endpoints can, I think I'd win in time although lose in bandwidth. This is, of course, speculative if I've not tested it. But a profile page:
1. Gives me last_login time (which I may not need), a join date (which I do), and I can't cache that unless I'm willing to accept collisions on name changes
2. Current games (unless this is an AJAX request so should count as a second endpoint, the games endpoint)
3. A count of club memberships (so I wouldn't need to hit the player's club endpoint at all)
4. Ratings, so wouln't need to hit the stats endpoint.
Theoretically, that would drop my endpoints to two maximum from four. A question perhaps of how the endpoints are structured; I'd be happy to use a "profile+" endpoint that more clearly matched the web profile page.
Melbourne, Australia (very obviously hitting the CDN, so not a good measurement):
$ ping api.chess.com
PING api.chess.com.cdn.cloudflare.net (104.17.237.85) 56(84) bytes of data.
64 bytes from 104.17.237.85: icmp_seq=1 ttl=59 time=20.7 ms
64 bytes from 104.17.237.85: icmp_seq=2 ttl=59 time=18.4 ms
64 bytes from 104.17.237.85: icmp_seq=3 ttl=59 time=19.3 ms
64 bytes from 104.17.237.85: icmp_seq=4 ttl=59 time=19.0 ms
64 bytes from 104.17.237.85: icmp_seq=5 ttl=59 time=18.8 ms
64 bytes from 104.17.237.85: icmp_seq=6 ttl=59 time=19.7 ms
64 bytes from 104.17.237.85: icmp_seq=7 ttl=59 time=18.4 ms
64 bytes from 104.17.237.85: icmp_seq=8 ttl=59 time=19.9 ms
Pittsburgh, USA (interesting, much faster, but still hitting the CDN)
$ ping api.chess.com
PING api.chess.com.cdn.cloudflare.net (104.17.241.85): 56 data bytes
64 bytes from 104.17.241.85: icmp_seq=0 ttl=60 time=0.382 ms
64 bytes from 104.17.241.85: icmp_seq=1 ttl=60 time=0.601 ms
64 bytes from 104.17.241.85: icmp_seq=2 ttl=60 time=0.449 ms
64 bytes from 104.17.241.85: icmp_seq=3 ttl=60 time=0.456 ms
64 bytes from 104.17.241.85: icmp_seq=4 ttl=60 time=0.376 ms
64 bytes from 104.17.241.85: icmp_seq=5 ttl=60 time=0.379 ms
64 bytes from 104.17.241.85: icmp_seq=6 ttl=60 time=0.352 ms
64 bytes from 104.17.241.85: icmp_seq=7 ttl=60 time=0.382 ms
@bcurtis, are your timings from an internal network or from outside the CDN? If they're from an internal network, I suggest you try them again from outside.
Yes Giles, I missed the reference to the 429 response code in the OP - I should read things more carefully! But I only ever send endpoint requests sequentially & that's more than satisfactory for my needs. I'm not sure how I'd set things up in Python to generate requests concurrently - how do you?
I always work on the principle of abundance of caution & flag up in my scripts all responses other than 200, which indicates a successful one. So I'd certainly see a 429 if it was returned.
The only parallelism I use is running multiple scripts concurrently.
Otherwise it's into multi-threading and that's not warranted for me, or possibly HTTP/2.0 which I don't have easy access to yet.
A single script at a time suits me quite well; one may be long running in the background yet I can still download a player's archive and have a third connection for a report that takes a few minutes.
... This is decidedly against the goals of the API, which is to help developers build tools that help players enjoy and learn more from their games on chess.com. I hate blocking people, but I need to protect the site.
...
A number of my tools are admin related:
1. Checking how balanced a match is (which benefits players: players like balanced matches or in some competitions to win)
2. Discovering who from "my" team hasn't signed up for a match where #1 indicates more players are wanted
3. Helping find players to recruit to teams (and as there is always turnover, invitations to new players helps keep a team vibrant).
I think all of #1-#3 indirectly benefit players on chess.com, but they do so (like Stephen's work that I've seen) by helping admin for teams and/or competitions, which are part of what makes the site what it is.
Perhaps the wording of the goal for api.chess.com might be clarified if these more widely targeted tools are "within scope"? (I believe they are; people who run competitions, teams, tournaments et al make chess.com the community site that it is. We're also free labour, in a sense; chess.com provided the facilities for teams/groups/clubs and people have run with them with "only" support, bug fixes, enhancement requests et al.)
I hope in my posts I'm not beating up too hard on api.chess.com. I'm not really intending to, and certainly not on you, @bcurtis or Andrea.
I value api.chess.com and don't want to lose it!
My hopes/expectations are not really being met, however:
a) It is slow, as noted probably tediously 
[And I work harder than I want selecting what to cache and using out-of-date data because of this]
b) Addition of new functionality (vote chess access anyone?) is still hoped for but I'm not seeing many changes now, even after the number of external bug reports has dropped and most/all have been addressed.
I do concur with Giles about the time factor and I wonder if player profile and player stats endpoints could be consolidated. These two are likely the the most used calls and I invariably need both.
My routine for logons, XE and UK flags, usually gets approx 4k+ returns, which requires around 9k calls for player details.
Using VBA / XML takes about 150 minutes.
I admit my hardware is not superspeed but it's the requests that use the time.
Yeah, the main speed problems probably come from the sometimes large number of API calls you have to make to get the information you want.
As an example, let's say I want to check which members of a club are online (in case I want to invite them to a live blitz tournament), and I want to give them priority in order of how long they've been in the club and when they were last active there. I also may want to only invite people with a certain blitz rating.
For a club with 130 members getting this information took 521 API calls and just shy of 4 minutes on my computer right now.
The information I needed was the member's names, usernames, online status, when they joined the club, when they were last active, and their blitz ratings.
GraphQL query for club online players example (a production system would probably have filters for online status and rating limits)
TLDR - go to CONCLUSION
I don't think that ping is the right way to measure in this case. We're not interested in how quickly we get a response to ICMP requests, we're interested in how quickly we get a response to HTTPS requests. So curl should be more telling. This is not a great benchmark, but it shows a comparison of accessing my profile endpoint via the API server and my profile page via the WWW server.
I've used this command for measuring (on linux, I assume it'd work on mac too):
curl --tlsv1.2 --silent \
--output /dev/null \
--write-out "DNS lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nStart Transfer: %{time_starttransfer}s\nTotal: %{time_total}s\n\n" \
https://example.com
The results - API:
~# for i in `seq 1 5`; do
> curl --tlsv1.2 --silent \
> --output /dev/null \
> --write-out "DNS lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nStart Transfer: %{time_starttransfer}s\nTotal: %{time_total}s\n\n" \
> https://api.chess.com/pub/player/WhiteDrake
> done
DNS lookup: 0.004425s
Connect: 0.030067s
Start Transfer: 0.051542s
Total: 0.051683s
DNS lookup: 0.009391s
Connect: 0.014467s
Start Transfer: 0.038607s
Total: 0.038730s
DNS lookup: 0.004145s
Connect: 0.009299s
Start Transfer: 0.040264s
Total: 0.040359s
DNS lookup: 0.021934s
Connect: 0.026744s
Start Transfer: 0.046574s
Total: 0.046717s
DNS lookup: 0.004192s
Connect: 0.008758s
Start Transfer: 0.029871s
Total: 0.030002s
The results - WWW:
~# for i in `seq 1 5`
> do
> curl --tlsv1.2 --silent \
> --output /dev/null \
> --write-out "DNS lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nStart Transfer: %{time_starttransfer}s\nTotal: %{time_total}s\n\n" \
> https://www.chess.com/member/player/WhiteDrake
> done
DNS lookup: 0.046650s
Connect: 0.055652s
Start Transfer: 0.353334s
Total: 0.366290s
DNS lookup: 0.011751s
Connect: 0.017274s
Start Transfer: 0.459113s
Total: 0.467071s
DNS lookup: 0.004664s
Connect: 0.127746s
Start Transfer: 0.436580s
Total: 0.449172s
DNS lookup: 0.007561s
Connect: 0.130972s
Start Transfer: 0.444988s
Total: 0.457640s
DNS lookup: 0.177430s
Connect: 0.271234s
Start Transfer: 0.737503s
Total: 0.745157s
CONCLUSION
I believe this shows that the API server can respond more quickly than the WWW server. And off course the JSON format is more compact than the HTML site with all the ads etc. The concrete numbers will differ based on your endpoints. But I would be surprised if any information would be obtained quicker via the WWW server than via the API, even if multiple endpoints need to be queried.
A more compact version of the command, perhaps better for copying, would be:
curl --tlsv1.2 -s -o /dev/null \
-w "DNS lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nStart Transfer: %{time_starttransfer}s\nTotal: %{time_total}s\n\n" \
The editor here breaks the longest line after the '%' sign, so that line break would need to be removed.
"that line break would need to be removed" - what line break?
It’s a one-line shell command, so any line-break needs to be escaped by the backslash \ char. The whole command, starting with curl and ending wiith the url, should be on one line, or if it’s broken into several lines, each line (except the last) must end with the backslash \ char, that’s what I was trying to say. Unfortunately, the string after the -w (or —write-out) is too long for this editor, so the editor has broken that line. Does it make any sense? 
I thought that was what you might mean but remember that there aren't line-breaks as such in web pages because the browser automatically wraps lines around. So when someone copies & pastes that single line command into another application or text editor, it will paste as a single line without a line break.
If you look at the source HTML for the page (right-click & select View Page Source), you'll see it's still an uninterrupted single line. Use the search (Ctrl+f) & enter '-w "DNS lookup: %' (without the single quotes) & you can see it.
Actually it's quicker to right click & select inspect element, that way you don't have to search. I agree with you that there are no LF characters from ASCII table on the page etc. if that's what you meant, though a line break can be forced by ending the paragraph and starting a new one (i.e. </p><p>). And these line breaks are respected (e.g. for cypy&paste) AFAIK.
I had looked into the code and I thought I had seen that the editor had inserted such a line break (should I call it paragraph break? ), but now that I checked again, I see it hadn't broken the paragraph, only my browser wrapped the line (as you've pointed out). So copy&paste the curl command should work, at least on linux. I assume something equivalent can be done in PowerShell too.
In any case, I wanted to show that using the API endpoints should indeed be faster than scraping the pages served by the www server.
" I wanted to show that using the API endpoints should indeed be faster than scraping the pages served by the www server"
I agree up to a point & I think you'd need to make a great many separate API requests before scraping became preferable. Although recently I needed to collect the current ratings of players in a number of finished matches & while these are given on the web-page, they're not part of the match endpoint for in_progress or finished matches.
I haven't tried measuring this but might scraping for matches with large teams be faster than making separate requests for each player?
One would have to measure to see when scraping one web page would be faster than making n API calls. When we're talking hundreds of API calls (like the case you mentioned, i.e. getting players' ratings in a huge match that is either in progress or finished), I'd guess scraping one page should be faster, but it's just a guess. Speaking of this, is there a reason why matches that are in progress or finished do not include players' ratings in their endpoints, @bcurtis?
I raised that question in another thread & I think including ratings in in_progress matches is being considered but not for finished ones, which is where I really need them. Since the match web-page shows them it's not clear to me why they're not included in all match endpoints.
@WhiteDrake wrote:
A unique name of the script, so that when we tell you "This X is hitting too fast" you know what we're talking about. An email address so we can contact you; a username on chess.com for the same.
@skelos gave ping rates. Pinging the domain only shows the distance to the local CDN edge server. You can instead ping 68.71.245.5 which is the load balancer for the origin server. I cited two sets of times: from http requests after DNS caching (100–200ms), and actual server time (1ms cached, 20ms not). @WhiteDrake then cited some HTTP times, but the API times are misleading since they liking are being cached efficiently by the local CDN edge cache. A more accurate test would be to a) use lowercase username so it's not a redirect, and b) append some garbage query string after the URL. Still, I expect the API profile to be much faster than the www profile.
skelos also wrote:
The profile endpoint https://api.chess.com/pub/player/erik gives the same data from the same source ("joined" is the join date, and "last_online" is the last login date). Regarding the general question of data composition, we are working on options that will allow you more flexibility in telling us what data you need, to reduce round trips. We hope this is somethign we can address this year.
He also asked if his scripts were "in scope." Yes, of course they are! You are helping people play games and enjoy chess. That is exactly the scope. The script we blocked wasn't acting on behalf of players, it was literally starting at the beginning and trying to download the entire database. That is helping no one, and adding no value.
Thanks Ben. That shouldn't be a problem. I can pass the aub-routine name to the call procedure.