Fixing New Analysis

Hi @dallin. Any feedback on when the above will be implemented. 

@dallin an example of some weird graphics behaviour by the feedback system that I experienced recently when doing an analysis. Not sure why it did this. As you can see the arrow is not pointing in the right direction. When I took the exported game pgn and re analysed it in the analysis interface under "learning" it showed the arrow pointing in the right direction: 

And now when I ran the exported game pgn through "analysis" interface under learning the arrow points in the right direction!:

Hopefully your dev guys can work out why these errors happen when the analysis runs.

@PawnstormPossie yes I would also have liked an update on progress with the engine. Was just thinking about it actually.

@dallin did you see my post #287? Why did that happen?

Then...You also mentioned in your post #279 that you guys were also making changes to the Analysis formatting to better distinguish between saved self analysis moves made during the game and post game engine variations and move alternatives added to the game move list on the analysis tab after the initial game evaluation. Right now it's a mess! It is IMPOSSIBLE to distinguish between the two. If I go and include a correct retry variation into the game list it gets lost in amongst my saved self analysis data. 

But we haven't heard anything more on anything.

We would really love some feedback from you.

Hi all. I’m catching up to speed on this thread. Basses on your feedback, we are currently rethinking many of the elements of Analysis. This is definitely our most complex feature - with a TON going on with servers, and a complicated interface. Additionally, there are so many needs and use cases from all kinds of different players. We’ve ended up with something that isn’t making everyone happy, and I’m unhappy with that. I’m a frequent analysis user myself, and while I like most of what is going on, and rarely see some off the bugs reported here, I recognize that not everyone is having this experience. Anyway, I don’t have any quick answers. But we are putting together our plan for next iterations. Thank you for sharing your concerns - very helpful!

Hi @erik

First of all THANK YOU for your feedback. Really much appreciated!

As you can see this forum thread documents many real instances where the engine feedback system's analysis of various moves has been quite inaccurate, as well as manifesting occassional graphic rendering problems too (see my last post as an example #287).

The feedback system is also quite sluggish in analysing new moves/variations added after the initial full game analysis is done.

Then there is the issue where all one's saved self analysis data in a daily game were erased once an analysis was done. This was "fixed" by incorporating the game analysis and one's saved self analysis data into one amalgemated game list. But this has caused new problems, making things now very difficult to read/follow in the new analysis game list (see my post #289). @dallin said they were working on a way to distinguish engine variations from self analysis moves in the analysis game list. But we haven't seen that yet. That is where things ended and the last we heard.

Personally I believe one should still have access to one's saved self analysis moves that one saved during the game, but they should be acceccable via the self analysis button, and they should not form part of the post game analysis move list. This then clearly seperates "during game" analysis work by the player from "post game" analysis.

In any case I really look forward to seeing and using new better iterations of the system. I appreciate that things are not simple to fix/create, but it's vital we get this section right, as it is one of the most important tools on the site as you so rightly mention.

Once again...thank you for being so close to the coal face

We are working on this. It's generally working well, but there are some inconsistencies. Engines are fickle. Edge cases are weird. Working on it!

This one I have not been able to reproduce. Seems extremely rare. Will keep an eye out. We also have a new chessboard coming which should address this. 

This is your local computer doing the calculation. How quickly that analyzes depends on how fast your computer is, and how complex the position is. 

One of the challenges we get is that people are asking for improvements that fight against each other. 

"It needs to be FASTER!!"

and

"It needs to be MORE ACCURATE!!"

But those are directly opposed. More accuracy requires more time. I don't know how to reconcile this...  

Would you mind posting a screenshot? Thanks!

Finally, for this one, we show BEST and ALTERNATIVE both. 

Honestly, this is a lot about perception. The old 1 minute one was worse than the current 10 second one...

Technically,  due to poor scaling in SMP, Stockfish wont perform  linearly better by multiplying  number of cores

If you upgraded 10x cores,  the program will not perform equally in x10 shorter time control.  As current analysis is running on x10 shorter time control, it would probably be compensated by 40x cores better hardware, not by 10x cores. (some stockfish developers tested that doubling time control perform  equally to quadrupling hardware in some testing results)

Hi @eric

Sorry I missed all your replies....there is a fair time difference between us! For the sake of brevity I'll respond to each of your posts by just referencing your post no:

Your post #292: I know @dallin and his team have been working behind the scenes as and when we post examples where the engine has done something bizarre or been inaccurate. There have been less posts on this coming in lately so it does look like accuracy (I hope) is improving in the game analyses. I was merely stating where we have come from. It is good to know however that your team is still on top of this issue

Your post #293: This kind of issue has cropped up a few times....but not shown often on this forum.....so yes one could say they are isolated incidents....but the point is they shouldn't happen at all. I brought it up I guess as I had posted a very recent example of this happening to me and nobody responded or acknowledged it from your team. But nice to know there are things being done behind the scenes that will ameliorate any future chances of this type of issue being experienced again. 

Your post #294: fair point. I understand where you are coming from. If I have to choose speed vs. accuracy and there is no way to programmatically improve "computational speed" then I take accuracy over pure speed any day. It makes no sense to wait 5s for a false or flawed answer vs 2 to 3 min for an accurate answer! The fast answer that isn't as accurate could easily result in one making the wrong decision or conclusions!

I made the point re sluggishness because I have seen, for the exact same position, the MultiPV section (the area switched on via the "show lines" checkbox, which also works off one's hardware) works significantly faster to output an answer than the "graphical engine feedback" system, to the same depth. And that didn't make sense to me, as they are both using the same engine! Hence I thought that possibly there was a way to speed it up as it is quite laborious if one is clicking through a 10 to 15 move variation. But if this is the best it can operate at reliably and accurately then I must accept it as it is.

Your post #295: I posted about this in lots of detail in posts #276, 277 and 278. In Post #277 I even showed an idea around colour coding "post game" analyses moves and variations added to the analysed game list, so that they could at least stand out from the saved self analysis moves and variations done during a game (that now are a part of the analysed game list in the grey areas).

Below is the screenshot you asked for. In the grey blocks are all my "saved self analysis" done during the game. But if one adds a new move or variation suggested by the engine or from say the retry area it also gets pasted in the same grey area and in the same font and colour. So it is very difficult to differentiate between "during game" saved analyses and "post game" analyses.

My other alternative solution is that the saved self analysis moves, variations and annotations be accessed still via the self analysis button after the game, and don't form part of the post game analysis game list accessed from the analysis tab. Then there would be no confusion.

Your post #296: The analysis never showed me 2 alternatives. Under the board it just said "Qf6+ is best" and on the board pointed to h7. It did not then show any additional text in green about an alternative "best" move like it usually does when it shows 2 possible best moves.

When I put the game pgn through the analysis board then the arrow pointed in the right direction towards the right square...f6. 

Your post #298: you make the comparison of the new very fast analyses to that of the old 1 minute "Quick analysis". I never liked the old quick analysis. I saw it was not that accurate at all. Which is why I always used the "full analysis" option, the one that gave the CAPS score. It usually took between 5 to 7 min to complete. But I was quite happy to wait the 7 min as I knew I was getting a reliable output. I therefore hope that the new faster analysis system is equivalent to the old "full analysis" or better! I would hope we are not getting a better rendition of the old "quick analysis"!

We should be aiming for accuracy and reliability as the benchmarks. @dallin expressed this very same sentiment. If it's faster that's an added bonus/feature.

@drmrboss not being a computer engineer or programmer (from your posts am I correct you are one?) I am struggling a bit with your jargon. What does poor scaling in SMP mean? Can you dumb it down to normal English for us amateurs

What are you saying in a nutshell? That to speed things up I would need massive computational resources?

You say....."As current analysis is running on x10 shorter time control"....what do you mean by this? Usually my game analysis takes now 3 or so seconds. With the old system one could choose from 4 types...quick up to full (each with varying times dependent on the game length). 

 SMP means multiprocessing technology. Poor scaling means, 2x cores are not performing 2x better in performance. ( although the search nodes can be 2x multiple, the quality of search is not 2x better).

Previously, the analysis was done for 60 secs duration but it is currently  on 10 secs. ( 6x shorter time control). 

In my assumption, chess.com should upgrade 24x stronger hardware to get comparable results from previous analysis.

( Those data came from stockfish  gameplay but not by analysis).

Thanks for the clarifications @drmrboss  

@drmrboss where are you getting the 10 seconds variable from? Mine certainly have never taken that long....and there isn't any setting to adjust the initial game scan. All it says in the green bar as it scans is d=20 (i.e. the depth).

Also previously there were a number of levels of analysis to choose from (with times to complete for each based on the game length)... a "quick analysis" which typically took 2 minutes or so, then there was a "deep analysis" option (typically ran for about 5 min or so), and then a "maximum analysis" which could run anywhere from 7 to 15 min depending on game length. And finally there was a "full analysis" which included a CAPS score...and that ran for the very longest.

Where are you getting the 60 second variable from?

@drmrboss see above for the specs for the hardware being used by chess.com on the analyses engine. @dallin says they use "several hundred physical cores...." From post#239.

@PawnstormPossie you raise some very valid points.

I personally value accuracy over speed/time. And this is the thing: some may want a simple fast blunder checker and in that case a quick basic scan is fine. While others like me are using the analysis tool more deeply, for more in-depth learning, where the correct classifications of moves becomes very important, as well as the move variations associated with them (which is why I love the retry section so much and the score histogram etc...this is where I learn). That's probably why so many liked the old system...because there was a choice of scan accuracy type for all occasions.

Now the question needs to be asked...what level does the new analysis output actually equate with vs the old levels? Is the new analysis the equivalent of the accuracy of an old "full" analysis? (i.e. max accuracy). Because then everyone wins! 3 to 4s for a full deep analysis! Fantastic. It just has to live up to its expectations in terms of consistent accuracy. 

Noted. We are working to address this.