Feedback Request: Limit Current Games Response To 250 Games

We have about 60 players who have more than 250 currently-active Daily Chess games — and 2 players actually have more than 1000! The current API is not well suited for these large lists, but is working very well for the average player (8 currently-playing games).

We are looking at limiting the /pub/player/{username}/games list to just the most recently changed 250 games. Even though this list would not be complete for 60 of our players, it would always contain all of their current games in which the other player just moved or performed some other action.

The /pub/player/{username}/games/to-move list would still be a complete listing of all games the player needs to make a move in. The game archives endpoints would be unaffected.

I am asking the developers here if this limitation for these 60 players would have significant negative impact on their work or plans.

 Could you not simply give support for pagination?

I mean, I would love to be able to:

/pub/player/{username}/games?page=2&per_page=20&include=metadata

And maybe even better, I would love to see support for Link Headers -as well. 

I am not planning to use this is no issue. But I agree with walidmujahid.. Paginate should exactly solve this, but i would suggest to do it with lower priority once more important items are ready.

The players with a very large number of games are likely to be active in a large number of team matches and/or tournaments, and are therefore likely to be of interest in any project geared towards club admins.

Yes. But I am not interested in exact games of those players if it would be possible to see just result of that game from club related resources ... which I am not sure now if it is planned.

I too am a little unfamiliar with what is being talked about.

I can imagine being interested in:

a) How many games a player has

b) How many team match games a player has for a single team (and would like to be able to find such in their archived games too).

I am struggling to think why I'd want a list of games a player is currently playing. This may simply mean this part of the API is not of use to me or being provided for me particularly.

Thus my sole comment may not be of much use: I am wary of any API ever that offers a subset of information when a subset is not selected. OTOH, I know that some database queries needed "LIMIT ..." or else a server could be driven into the ground and the consuming application would have trouble with the data too.

This sounds to me like it wants to be iterative, but (me being ignorant again) how easy or hard that is with JSON and a constantly changing set of games I don't know.

My 2c. Requested, and possibly not worth more than 2c!

I recall problems with LDAP too if I asked for the "wrong" thing, but it was question-response and state was kept client-side, as I recall. (Haven't touched LDAP for a few years and don't know if it has changed or if the LDAP server I was using was optimally set up, abysmally set up or anything in between. It was an enterprise directory for a company with 100,000 - 150,000 employees at the time. A lot less people than chess.com have accounts!

I am wondering why displaying the information about the current games should be any more difficult than showing the information about the completed games (which, I hope, will never be limited).

I see no plausible explanation for such restriction.

Agreed. Though, I have seen other cases where the concern is about "size". While I can understand this, I do not believe this is a good enough reason to limit things. Change things around and creatively rework how people get the data though the API, sure, but never limit. 

If you are making an API that provide public data, I personally expect access to all public data -though, maybe that scope is too big ... I do not know. 

Completed games are limited to 100 pages, if I remember right, but you can get around that by filtering results by date or other filters.

I'm not sure why a history of completed games needs to be so long but others may find it so. 

Rating history? How many times someone has played a particular player before? How many accounts since closed for "Fair Play" violations has someone played? A way to obtain their games and not need a web browser and chess.com to access someone's "opening book"?

Off the top of my head.

Some of that assumes we'll get access to per-game PGN data once we get access to lists of games, but I understand that is being worked on. PGN data being one of the very basic things people want to grab programatically.

I wrote in an earlier post I couldn't think of a reason to get a list of someone's current games: I have since I posted. Someone with 17 completed suspicious games ad 117 concurrent ongoing games ... I'd like the list and using the list to grab the PGNs, as some of those 117 will be advanced enough to analyse and I'll know to a high degree of certainty whether I'm looking at misleading statistics from a small sample or a cheat.

Trust me, it needs to be :-)

Edit:

Oh, and what @skelos said

Turned out this time the 17 games was enough and the retread user was banned, but in another case I or someone else might really want those games.

Competitions like TMCL are taking banned users very seriously, and teams are both looking to avoid potential trouble in their own teams and looking hard at their opponents, all of which is free help to chess.com in furthering their Fair Play policy.

Thank you all for your feedback. We reviewed what the systems monitors were telling us, and we decided to keep complete lists. We are discussing ways to paginate exceptionally long lists, should they become a problem; we believe that we can do this in a backwards-compatible way and so we do not need to change things now, and we can wait to see if the size of these responses become more of a problem.

To answer some questions...

• We absolutely agree with the idea that simply discarding the data and making it inaccessible was dead wrong. Sorry it even popped into my head. I have purged the synapses responsible for that thought.
• The problem that we spotted comes from an unexpected side effect in our caching system. This is what allows us to safely publish these data without onerous rate limiting and authentication layers. The side effect appeared related to excessively large responses.
• This caching layer also makes it quite difficult to synchronize multiple pages, and so it limits our options for pagination; we think we've come up with options, but this is why it wasn't a natural fit, though it is an obvious one.
• We have 60 or so players with more than 250 currently active games, but we have more than 20,000 players with at least one month of game archives that have that many games.
• We are building out the player-rating API endpoints now, but historical ratings changes are and always will be attached to the games played. This means that if you wish to uncover the historical ratings for a player, you must have access to a complete game archive.

Can you tell me more about what you're describing here? If we find that we need to paginate, then Link headers would be used to indicate next/prev URLs. Currently, Link headers are used to point to the JSON-LD context, which some generic JSON clients can understand and help organize the data. We'd love feedback on this — it's a bit new to us.

Thanks, everyone, for taking the time to weigh in on this!

That is actually exactly I had in mind for Link Headers -indicate next/prev URLs. 

With my original example: /pub/player/{username}/games?page=2&per_page=20&include=metadata I was thinking of a more human readable approach to including the metadata in with JSON response itself and just mentioned Link Headers as an afterthought. 

So sorry, I should have been more clear in the first place. I guess I was just excited at the possibility of getting pagination support for this API.