Live Chess Matches

We currently play a live chess league between national clubs of America, we have:

Team Canada, Team USA, Team Mexico, Team Nicaragua, Costa Rica Live Chess, Team Puerto Rico, Team Dominican Republic, Team Venezuela, Team Peru, Ecuador Live Chess, Team Colombia, Argentina Live Chess, Chile Live Chess and Team Brazil Live. Our league is called Live Chess Pan-American League (LCPAL).

The tournament format is a double robin hood system (all against all - twice), as in football, roundtrip. Per week each team plays 1 match, in total I have 7 matchs per week, for a duration of 8 1/2 months. Blitz mode 5/2, but next year we will add another time control (10/2 or 3/2, maybe).

So, to define the teams' places at the end of each week, we do it in the following order:

1.- Sum of points

2.- If there is a tie of points, we use the Tie Break SB (Sonneborn-Berger) allots each player the total score of the players he defeated, half the score of those he tie, and nothing for the games he lost.

3.- Difference of total points: Points won - points lost, in all its games.

Currently all calculations are made with excel formulas. This is an image of the weekly score (example):

My wish list would be:

1.- Put the id of the match (every week) and that the system generates the score of the teams.

2.- Do a recalculation of the points of each game, for the closed accounts.

3.- Generate the final report.

 But, progress is made step by step, so any help in this process would be welcome.

Something like this, but for live

https://chess.bounceme.net/chess/ligas.php?myteam=Team+Barcelona

Interesting. There is no API for it as far as I know but I think I can scrape myself the data from the webpage and even create an Excel sheet with the data automatically. (Depend on my sometimes unreliable brain lol)

Im ignorant to how live matches are displayed on the webpage tho. Do they have an equivalent of this:

https://www.chess.com/club/matches/1

Where you can see all the boards or players in the live matches? 

Here 3 examples:

https://www.chess.com/club/matches/live/team-nicaragua/1645
https://www.chess.com/club/matches/live/team-nicaragua/1643
https://www.chess.com/club/matches/live/team-nicaragua/1635

But if any URL of these only change the number, the system redirects you to the match. Just try to change the last number.

Equivalent URL for Live Matches is (for example) - https://www.chess.com/club/matches/live/1000/games

Page structure is very similar to Daily Matches - so I'm using same (quick and dirty) URL scraper (selfmade).

For the record - I think that API for Live Matches will be great for Live World Chess League (LCWL) too.

Thanks. Time to invoke jsoup I guess lol. 

I have been scraping data with it and writing it into databases for a few days so I guess this is the final graduation test hehe.

Anything anyone's writing a web scraper for because the API doesn't provide that data has to be a good candidate for an API extension.  I expect Live Matches are in progress (and maybe even not too hard given that daily matches are done) but Live Matches are a new(-ish) feature so if they were't top priority for api.chess.com ... maybe they could be moved up?

(I still want vote chess too, but not to the point yet of scraping the website for data. )

Scraping is fun, until you remember erik can troll you by changing the HTML layout of chess.com

Scraping is fun the first time in an "I did it!" kinda way. If it isn't yet 20 years since I scraped my first website (and then fed queries back to it!) it is close.

The people running the site (intra-company) weren't terribly amused, but after (they said) taking a week to find out what was pushing their load up gave me a cleaner interface to the data I needed.

I thought about, but didn't, try to cloak my origin. In hindsight that was right, but if they'd been a touch less reasonable after they found me it would have been wrong. Large company politics ...

(I didn't know what load I was putting on their system, but their system was not designed for the purpose for which it was being used and I don't think scaled ... so then I tied a second trouble-ticket system to it and started matching tickets, and they had problems ... surprise.)

That anecdote is by way of saying I'm much happier with api.chess.com than with scraping, as:

1. It's a lot easier (D'oh!)

2. It's a lot more predictable and understandable for the network and operations people at chess.com, and presumably "expensive" endpoints will not be provided or will be noted as being expensive and if used much perhaps discussed to see if they can be optimised or partitioned more efficiently for whatever practical uses we find for them.

Haha yes. I am still on the "I did it!" phase. If I am able to make this thing work I will grow out of it lol.

I coded my beautiful scraper. So far I am able to get

Loser/Winner from each match.

Player count

Both results per board based on team perspective.

Score for the match

So before I keep coding, is the information I am gathering useful? Am I missing somehting?. Right now I am aiming for the basics as the tie breaking system blew my brain. With the information I have I could make the program export excel sheets with the following info:

Name of the team

Games won 

Games lost

Draws

Player count

Point won vs point lost. 

By the way, does anyone have links to matches that were drawn? I want to test the scraper against them. 

LOL. I was sending an HTTP request to chess.com to see if a match existed (I got 200 or OK) and I got forbidden. Probably because java was sending "java1.8_151" or something like that as the user agent. I changed it to "IM NOT A BOT CHESS.COM" and it went trough. Got an OK.

I FEEL LIKE SUCH A HACKER LOL.

There is a request in for the API team to include live matches similar to daily but I think it is pretty low on the priority list.

I finished the application. I coded it pretty quickly to the point where I wondered whether I was scripting or using Java. Some small parts of the code are horrible in terms of maintenance (I will lie and say Ill fix them someday, users wont notice this tho). I tested it and it worked fine. I am gonna test it in some other computers tomorrow just to be sure.

The app may feel a little bit unrefined (if not a lot hehe) since this is a test to see how it works and recieve feedback (hopefully). 

Just copy the match URL and hit on gather info. Once it says you can write the file just click on write file and well... save it. You can do this as many times as you want. Just erase the confirmation message and write the URL of the next match you want to analyze. 

https://github.com/AntonioBohne/Chess.com-Scraper/releases/tag/0.1

The jar is there.

Hello @NichtGut,

To be the first release, it is excellent. I've tried it, very good. Now, can the name of the opposing team be included in the excel file?
With the formulas, if you have problems I can send you the formulas in excel to help you.

Thanks.

It would be best if you share the format you think would be most useful to display data. Last time I tried to write an excel file with macros the macros did not take effect (according to the documentation it was my fault lol but I never figured out what happened) but if you want you can share the template of the Excel sheet you use to gather this data and I can take a look and probably make the program fill a copy of that for every match. 

If these first tests go well and my coding practices were good extending this so that it writes a "huge" Excel file with more than one match should be easy to do.