Noobie questions and real time PGNs

Whats up my PEOPLE. I suck at web development. I know Java and SQLITE but nothing more. I have basic HTML knowledge which I think has little to nothing to do with the API.

Anyways, first I have some basic questions to set my mind and then I will ask the specific one.

1- JSON is what is used to provide the text that you get when you type a certain "URL pattern"?

2- What exactly do you guys mean when you say endpoint, the result from the "query"?

3- I guess your programs execute the "URL patterns" provided by chess.com gather the response and format it to meet your needs, right?
4- How do you format it and if I wanted to use Java, is there any library you guys know of that I could use to connect with the API?

Finally, I was told by a chess.com god, aka developer that I can gather almost real time PGNs with the API. I went trough the whole document and I either suck at reading documentation or couldnt find how to do it. Can anyone enlighten me on how to?

You don't need any HTML knowledge or expertise to download api data & in its simplest form, it can be read in a browser window. For example, my api profile data can be accessed by entering the following URL in the address bar: https://api.chess.com/pub/player/stephen_33

It returns this string in the window ...

{"avatar":"https://images.chesscomfiles.com/uploads/v1/user/6617585.1c02630c.200x200o.59e80012b297.jpeg","player_id":6617585,"@id":"https://api.chess.com/pub/player/stephen_33","url":"https://www.chess.com/member/stephen_33","name":"Stephen Williams","username":"stephen_33","followers":58,"country":"https://api.chess.com/pub/country/XW","location":"Swansea","last_online":1525217593,"joined":1328406309,"status":"premium"}

Replace your own username in the URL & you can see your own data.

I've recently been modifying a script I wrote some time ago in Python which helps me assemble the draws in the TMCL competition, in order to fetch members' avatar-image URL's directly from the site, instead of having to maintain a separate file of them.

I'm not familiar with Java though, so you probably need help from one of the guys who codes for a living. My interest in coding is strictly hobby!

Yeah, I have ben playing around it with.

So whats and endpoint? And what method do you use in Python? The strings can vary in size and names so how do you target certain parts of it? And how do you transfer the string into python? In Java there is a library that downloads a page´s HTML so I guess you could do a document with that URL and get the whole string into Python. 

I just have too many noob questions lol

There's a Python thread Stephen started that might help ... or not!

I highly recommend a current version of Python 3.x if you go that route and the not-included-but-excellent "Requests" module mentioned in the thread.

We don't have a Java thread and I've not touched Java for a squillion years (but like SQLite, when I don't use PostgreSQL ), but a quick search suggests Java is doable, but which library is "best" I have no idea. Seems like both Oracle and Google as well as independents have done work:

http://www.oracle.com/technetwork/articles/java/json-1973242.html

https://developers.google.com/api-client-library/java/google-http-java-client/json

The whole "what is this JSON thing anyway?" is the bit you have to get used to, and I don't know how Java will break down the JSON strings into classes for you. (Presuming it uses classes; it may be more basic types like integer and hash table/associative array/dictionary.

I'm not sure what endpoint you need. I'd treat it as two problems:

a) Get data from any endpoint that you can use (profile isn't bad; pretty simple)

b) Figure out the endpoint(s) you need to do what you want.

The bit I'm not sure about is where you get per-game information. For daily games (which I doubt you are talking about) I think this would be the portion of the documentation to look at:

https://www.chess.com/news/view/published-data-api#pubapi-endpoint-games-tomove

For live games in progress I'm not seeing anything. Completed live games are available in the archives (modulo a little caching) but that's not what  you want.

In fact if I understood and recall the discussion from the notes correctly, right now it's "no can do" ... but it might be on the roadmap and a developer might have answered with respect to something we're going to see shortly.

If api.chess.com were from Google they'd call it beta.

Once or twice I've run into things that haven't quite worked right or as described; I've a request in another thread for a clarification of a section of daily matches (per board data) where what I get is not what is documented ... and I'm pretty sure the documentation will be what changes most. Maybe the data as well, maybe not.

Good luck.

I suspect cheating concerns are part of the reason ongoing live games are not available.  That and making the data available in real time might be expensive.

If you are using Firefox browser then it has a built in decoder for JSON data and will show sensibly formatted results in the browser window with level filters.

That makes sense; unfortunate sense for the original poster, but sense.

Not sure I entirely understand myself but I found this definition after a search:-

"The “endpoint” of a SODA API is simply a unique URL that represents an object or collection of objects. Every Socrata dataset, and even every individual data record, has its own endpoint. The endpoint is what you'll point your HTTP client at to interact with data resources."

I'm not sure that helps you (or me) a great deal but think of an endpoint, for the purposes of what we're doing, as nothing more than an online data resource, accessed by means of its URL. It's just terminology, so try not to worry about that too much.

After all, it's not necessary to know how an internal combustion engine works to be able to drive a car! But as a bit of a 'noob' myself, I may be able to give you some useful hints.

While the strings we get back from api requests are of variable length, they're rigidly structured, so extracting useful information from them is really quite easy. It's just a case of doing a little 'adjusting' & storing the json data in a suitable data structure. In Python that structure is a dictionary.

Here's how I obtain the avatar-image URL for any given member on c.c ..._

...

import json
from urllib.request import urlopen

username = 'stephen_33'

with urlopen('https://api.chess.com/pub/player/' + username) as response:
    for line in response:
        line = line.decode('utf-8')  # Decoding the binary data to text.

data = json.loads(line)  # Creates a dictionary, holding all the api data on the member specified

print("Avatar Image = ", data["avatar"])  # Displays the URL of the member's avatar

...
Do you have Python installed? If so, which version? My one is Python 3.3, rather ancient now, so I can't import the Requests module which is the recommended one for handling api requests. That said, the code I show above works extremely well for me!

Do you understand what a (Python) dictionary is & how to retrieve data items from one?

@stephen_33 is right — an "endpoint" is just a URL that performs a function in an API. In the case of this API, all of these are "GET" functions that return data (instead of creating, modifying, or deleting it).

Most languages provide a means for using HTTP to obtain text data from a remote URL. That's all you need to use this API. Assemble the URL according to the documentation and what you would like to get information about, and then HTTP GET that URL. The result is JSON, which most languages will parse into native data formats like objects or associative arrays.

We will not be able to provide live PGN updates for ongoing Live chess games. In my previous response, I assumed the OP meant Daily chess games, since he talked about HTML scraping, and there is no HTML to scrape for ongoing Live games.

I dont have Python, only Java, but I was interested in seeing how you guys modified the text provided by the endpoint.

I was trying to scrape the games by downloading the HTML document and then selecting elements by ID.

If I understand correctly the chess.com moves are displayed in a <div id=moveList_vertical> container. Inside the container there is a <span> per every move.

The <span> has an id with the number of the move its representing.

For example, 1. e4 would be <span id="movelist_1">, 3.Bb5 would be <span id="movelist_5">

And inside that span there is an <a> tag with a very similar id which contains the move itself. 

"Sf3" is Nf3

Last time I tried scraping it I was using Java and the JSOUP library which allowed me to do exactly what I said.

Document myDocument = getHTMLDocument(Insert URL here);

myDocument.selectElementById("yourId");

The names are not exactly like that cause I dont remember the exact syntax by memory. 

I remember that I failed miserably, maybe because I was trying to select the span and not the <a>.

I dont think I would be able to create something that allows chess.com players to truly play blinfold chess, by having dictated moves and voice recognition without having erik use his KGB powers to dissapear me. 

To dictate moves I would need to real time scrape and I dont know how I would insert the moves into chess.com when the player says the move and the voice recognition program translates it to text.

That would require the interactive API, which is not currently released, and, so far as I know, won't be any time soon.

Some discussion here: https://www.chess.com/clubs/forum/view/interactive-api-status 

See also https://www.chess.com/clubs/forum/view/readme-1

Adding a comment to https://www.chess.com/clubs/forum/view/what-will-you-build would show there's more interest in the interactive API.

Interactive API to insert moves into chess.com? I read a long time ago about modifying the moves your computer sends to chess.com but that was years ago so they probably deleted that option, the other option is to make a clicking bot but that is just too level god for me.

Come on erik!! relase the api

That was probably never something that was officially supported; more likely something someone developed for cheating.  I have heard of someone doing that for cheating, and it seemed a very clunky and awkward way to go about it.

Yeah lol. You can still find the webpage and github code online lol. Does seem weird but cool, the only thing that I never understood is how the heck do they modify the HTTP request or move sent if they dont make any starting move? 

Network development is extreme, take me back to my offline SQL comfort zone lol!!

Using automated means (such as the code you describe) to send moves to any game is indistinguishable from having an engine play your game for you, and as such would likely trigger a ban on any account caught using it. So... not recommended.

The interactive API is not on the roadmap. We do listen to developer interest and ideas on the matter, but currently we do not have any plans to allow outside developers to submit moves to games.

HTML scraping as you describe has no guarantee of stability. You may put weeks or months into your program, and it may be disrupted multiple times per week as we improve the website. The API is the only supported and stable means of obtaining the data. It uses common API standards and so all the help you need is just a Google search away: https://www.google.com/search?q=consume+JSON+REST+API+with+Java

Ok. So if I want to get the PGN from the erik game how would I go about it? 

https://www.chess.com/daily/game/1

I re-read the API documentation and it seems like I could retrieve the game if I know the date when it was played, but not if I just know the number (in this case 1).

It started on the 8th of July 2007 based on the  API with the following URL I should get that game

https://api.chess.com/pub/player/erik/games/2007/07

And I do get it but for some reason I dont get it in order. That is game one so it supposedly is the first game ever. However the first game to appear is game 42. Game 1 came after game 12, 121 and 11 which is kind of confusing. 

Now lets say I want to get this live chess game:

https://www.chess.com/live/game/2

First ever live chess game in 2009. Played in October. 

I guess this is the URL to get it

https://api.chess.com/pub/player/johnletox/games/2009/10

I also get it. Cool.

I was also trying to get games from today from komputer7 but I get this:

However it worked with "normal" chess.com users like:

https://api.chess.com/pub/player/antonioesfandiari/games/2018/05

So I dont know if thats a bug or just the API not wanting me to get games from komputer7 lol. The API works but it doesnt have real time games. I want chess move voice dictation.

Still working today lol!

I usually am dealing with daily game, so EndDate is typically the date I'm most interested in.

As I will not know the date a player's game finished, I go to the player's archives, see what months there are archives for. The archives do not seem to be reliably sorted per-month. (Mostly most recently ended first, but I've seen exceptions.)

I think you really want an enhancement request to ask for a game by ID.