Upgrade to Chess.com Premium!

Chess.com Computer Analysis Severely Faulty?


  • 4 years ago · Quote · #1

    Caparojo

    One of the main features which attracted me to try out a Premium Membership  is the ability to receive (computer) analyses of one's games.  It doesn't much matter to me whether the analysis I receive is from a human or computer, so long as it's correct.  Like everyone, I'm interested in the analysis to learn where I can improve my play.

    My issue is that it seems some of the computer analyses I'm receiving from this site are simply completely wrong.

    To illustrate,

     
    On black's 8th move, the computer analysis suggests that 8....Ne7?! is an INACCURACY and that 'perhaps better was 8. ...h6'.  If however you follow the computer's posted logic all the way through to white's 17th move, 17. Qxh6+, as to why 8. ...h6 is supposed to be a better line, the computer analysis ends there with a statement that this is a 'drawish position' hence that is why that line was better for Black than 8.... Ne7 which it was rating as leading to an inferior position (I'm not sure why some of the computer analysis comments are not showing up here ... please see the attached screen grab below after white's 17th move taken directly from the computer analysis as viewed on this site).
     
    If you look at the above screen grab of the computer analysis, it's Black's move and the Chess.com computer has rated this position as a 'drawish position'. 
    I can't see how this is remotely a 'drawish position'.  Unless I'm missing something (which happens regularly!) I can't see how Black can prevent mate from white's Queen on the next move, 18. Qg7#, without having to sacrifice his own Queen by 17. ...Qxf6, 18. Qxf6 (black can postpone this inevitable outcome by sacrificing his rook by 17. ...Rxe7+ but after 18.Bxe7, Black still has the original problem of being mated on the next move or sacrificing his Queen with the pawn capture on f6.).  So the best Black can do here is end up being down a Queen.  Hardly a 'drawish position' as proposed by the computer analysis.
    So, if it's not obvious, here are some basic (and in my mind serious) problems with the computer analysis:
    1. How can I trust any of the computer analysis comments when it suggests moves based on a position it interprets as a 'drawish' position when in fact that position is a decisively losing position.
    2. Where else in the computer's analyses is it making similarly wrong assessments and suggesting that a particular move is an inaccuracy or mistake etc.
    3. How can you put any faith in any of the alternative moves recommended in the computer analysis if the analyses are sometimes (often?) wrong.

    I assume there are many available chess computer programs available to Chess.com which would correctly analyse this (and other) positions.  Why are the present computer analyses so flawed?  I've read similar comments from others.

    What am I missing here?

    The purpose of my post is certainly not to defend the logic of my 8th move in the above game but to figure out what is wrong with the computer analysis or my interpretation of the information it is giving me.

    ------------

    ps.  On an unrelated topic, after I inserted the pgn of the computer analysis, all of the remaining text in this window became centered ... how do I format it to become left-justified again?

  • 4 years ago · Quote · #2

    omnipaul

    What you're seeing here is called the "horizon effect."  Computers determine a best move by brute force calculations.  Since the computer analysis is limited to a certain number of moves, it doesn't see something that exists just beyond those moves.  Even though a person can look at that final position and recognize almost immediately that mate is inevitable, the computer can't see it because it has reached the artificially placed limit on its calculation.  If it were allowed to delve a little deeper into the position, it would find the mate and its evaluation would change.  The best thing for us to do, then, is not to blindly trust the computer evaluations, but double check the positions near the end of the variations.  Keep in mind that that is a 9-move sequence, and it is possible that Black had a better move slightly before this, but that the computer couldn't analyze all the way through the variations to find it.

  • 4 years ago · Quote · #3

    erik

    we use a VERY strong chess engine. the problem is, as stated above, the horizon effect. we don't go 100% deep on all variations. 

  • 4 years ago · Quote · #4

    TheGrobe

    Wouldn't it make sense to trim the lines to a point where the horizon effect is obscured?  If you're analyzing to, say a depth of 20 ply, wouldn't only displaying the first 10 ply mitigate this issue substantially?

  • 4 years ago · Quote · #5

    icecoldalex

    I have had games where I am a couple of pawns up and the computer analysis suggests a line which at the point the analysis ends I get mated in one - nice!

  • 4 years ago · Quote · #6

    RainbowRising

    Sorry Erik, but your engine is crap. I have a game where it decides another move is better than a mate in one. Rybka disagreed though ;)

  • 4 years ago · Quote · #7

    Caparojo

    Thx for the replies people.

    First off, I should say I absolutely love the site.  First rate.  But ... I'm not totally convinced that the Computer Analysis portion of the site is all it could be.

    I understand the concept of the 'Horizon Effect' and in the example I gave above, we might concede that this was the problem with how the analysis reported the final position.  It just didn't look past it's last move and analyzed the position there.

    But what about an analysis (a different one) where the variation only goes 3 moves deep.  The horizon isn't very 'far' in this case and the position as it is evaluated then, on that very move, is wrong.  See below:

    In this case White doesn't have a 'decisive advantage' ... the game is over!  Black is in checkmate.

    I'm not quibbling about the use of terminology here but about the fact that the computer analysis on a variation which is only 3 moves deep (or 2 moves really since it occurs on white's 2nd move in this variation) fails to report that the game is over.

    Still not convinced there isn't something wrong here.

  • 4 years ago · Quote · #8

    Caparojo

    [COMMENT DELETED]

    I had posted a comment here about problems uploading the image in my previous post but it seems to have been corrected.  The image now displays.

  • 4 years ago · Quote · #9

    bobobbob

    erik wrote:

    we use a VERY strong chess engine. the problem is, as stated above, the horizon effect. we don't go 100% deep on all variations. 


    Can you tell us which engine you use, or at least the ELO?

  • 4 years ago · Quote · #10

    Mainline_Novelty

    bobobbob wrote:
    erik wrote:

    we use a VERY strong chess engine. the problem is, as stated above, the horizon effect. we don't go 100% deep on all variations. 


    Can you tell us which engine you use, or at least the ELO?


    i heard it was 2600 for diamond members

  • 4 years ago · Quote · #11

    Shuffleking74

    Just get Fritz 12. It's great.

  • 4 years ago · Quote · #12

    Caparojo

    I was going to post several more examples of what I consider to be oddities in the results of the Computer Analysis of games (or in my estimation 'evaluation errors') but it would just amount to re-stating the obvious.  In most cases these evaluations are given in alternate lines which are just a few moves deep so I'm not sure the horizon effect is really a valid 'defense' here (they would be examples similar to the diagram I posted above where the analysis engine is claiming one side as a drawish position or decisive advantage when in fact the situation is quite the opposite on the very next move or is already over by checkmate).

    (By the way, I'm feeling guilty for doling out some negative feedback with respect to a site I think so highly of but maybe we can improve this part of the site or member features and make a great site even greater.)

    Would it be possible, if Chess.com has control over this feature in the analysis, to at least have your program evaluate the positions at the end of the proposed alternate lines before it provides an 'opinion' of the position.  This would avoid what appears to me to be incorrect evaluations.

    The problem in trusting some of the alternative lines proposed by the computer analysis is that if the end result of that proposed line is completely wrong, was that alternative line ever any good and if so, where does it begin to fall apart.

  • 4 years ago · Quote · #13

    Caparojo

    Thanks for that feedback Schachgeek.

    Based on the fact that you won the game I have to assume there were no other mitigating circumstances to the analysis giving you a ? on a move which certainly sounds was a winning move.

    Did the Computer Analysis recommend a better move which led to say the capture of the queen without trading material or capturing in a different manner which would leave you in a better position or was it just plain wrong?

    I understand the concept of a 'horizon effect' which Erik is referring to but there seems to be more going on with the analysis oddities than that.  Your example seems to be one of them.

    Congrats on your win by the way.  You must play a pretty solid game :)

  • 4 years ago · Quote · #15

    Niven42

    I've seen Master Games where a "?" move leads to a mating sequence.  This is not uncommon, especially in the case where a sacrifice leads to a brilliancy.  An opponent can always choose the 2nd best option if they see the continuation, which might not then lead to mate.

     

    (edit) Contemporary engines might not give moves like this a "?" if they are able to correctly analyze the position.

  • 4 years ago · Quote · #16

    Caparojo

    David (IM dpruess), honoured that you would join in (love your lessons by the way, just starting them).  Thanks. 

    Your comment about the analysis' evaluation (e.g. 'white/black has a decisive advantage', 'drawish position' etc.) applying to the first move of a sequence as opposed to the last move of the sequence might explain what appear to me to be erroneous evaluations and you may very well be right. 

    But, it doesn't make much sense to me however that the evaluation comment, if it did apply to the first move, is held back by the computer analysis and posted against the last move of the sequence, where it may or may not be accurate at that stage.

    By the way, I'm not complaining that the computer initially suggested an incorrect move at the beginning of the sequence (e.g. in my first example 8. ...h6 is clearly better) but I'm trying to learn from those analyses.  To me, the reason for a suggested alternate move should be that it leads to a superior position.  What I'm tring to learn usually is why make this move (which should become obvious by following the alternate line).  If the outcome of that alternate line is not evaluated properly, it casts potential doubts (in the mind of this not so experienced player) on the validity of other earlier moves in the line.  I'll try to find some other examples in my analyzed games to illustrate.

    Schachgeek, if you still have the analysis you referred to, could you post it here so we can follow along?

  • 4 years ago · Quote · #18

    RainbowRising

    do it! if you feel the cause of the compliants are the 'misleading' computer notes rather than the actual evaluation then do it!

  • 4 years ago · Quote · #19

    paulbrad25

    I have to say guys, as a beginner, I came here to learn how to be better at chess, and having analyzed some games that I've won (if I can see my own mistakes I don't bother getting someone/thing else to tell me/rub it in) to try and learn how to be better, I don't expect the computer evaluation to tell me I made an error then show me alternate moves that lead to a loss, or a weak position, when I've won that game. Surely as the computer is moving both sides from that point, it should have a good reason for making the suggestions it does.

    I'm a big fan of learning by losing, it works! and trust me it's readily available here. Computer evaluation should be a way to learn even though you're winning, by showing you clearly how you could have shortened the game. So either the horizon should be 2-4 moves ahead to reflect spontaneity of basic human play (at my level) or all the way to the end of the game, to show fastest solutions. (even better, it should be adjustible)

    Personally, as a beginner, I would prefer a 100% depth to the analysis I ask for,  that way I learn how to shorten my game by stopping missing strong positions. I don't mind waiting for it, after all, it already tells me 'this may take several hours' and then is ready in minutes

    I occasionally teach in my field so I understand how difficult it is to produce a training tool that utilises yet ignores the depth of my knowledge in my subject and builds on and enhances the knowledge of the person I'm teaching.

     On the other hand the mentor program is almost brilliant, my only problem with it were the beginner stages when there are sometimes loads of 'alternate correct moves' for no apparent reason, surely thats a basic core programming bug that could be fixed? the ability to assign different answers to these alternate correct moves proves that a slightly deeper programming tweak could allow multiple correct moves.

    I apologise if I have just repeated a previous point, however I'm only just learning the terminology therefore didn't use any.

    P.

  • 4 years ago · Quote · #20

    jlconn

    I think the whole idea of a non-master learning chess from computer analysis is completely misguided in the first place, if by "learning" you mean what most of us mean, which is an increase in understanding. No chess playing computer in the world understands chess at all, rather they simply calculate variations out to some specified depth, possibly augmenting that depth in positions with many threats present. This is actually the source of the horizon effect. In certain calculated variations, the computer may reach a quiet position at the end of its standard search depth. When that is the case, most computers simply evaluate that position using an evaluation function which is little more than a material count plus some rather unsophisticated heuristics regarding pawn structure, king safety, etc. The computer understands nothing of that position, it just returns an evaluation. In almost all cases, that is fine, but in some cases, the computer's evaluation function will favor a strategically lost position because it is only ever making a static analysis, and there may be a very dynamic plan available to the side with static disadvantages. A grandmaster wouldn't make those mistakes.

    The lesson? Until you're a super grandmaster, do all of your learning by reading what grandmasters have to say. Limit your use of chess playing computers to using them as sparring partners and to check your own tactical analysis (*after* you've made it). You'll learn nothing more than the obvious from chess computers as they exist nowadays. But grandmasters have published known errors, I hear you say. Yes, this is true. If you are memorizing without actively learning, then you will suffer for those errors, and you deserve to. But a grandmaster can teach you more about chess with an explanation of a 10 move variation of 10 bad moves in a row than a computer can teach you by showing you what it has calculated (possibly erroneously) as the best variation for both sides without explanation.

    This whole notion of "the correct move" is counter-productive to learning. On the one hand, there may in fact be a "correct move" in a position; if this is the case, then chess becomes a combinatorial problem, requiring no understanding whatsoever, only memory or arithmetic - in other words, it is not worth playing, except to prove the obvious fact that some people possess more knowledge of a narrow subject than others. On the other hand, there may in fact be no such thing as one or even a few "correct moves" in many positions; if this is the case, then it is pointless to search for one, and even harmful when misled by a computer that claims to have found one.

    The idea of trimming variations to avoid the horizon effect is faulty. The computer's evaluation of a move would still reflect the flawed evaluation of the entire variation as a result of the very same horizon effect. So how does that help? As a human, you should look at the whole variation that led to that evaluation and ask yourself whether that makes any sense. Until chess is "solved", the human - not the computer - should make the final decision. And after chess is solved, well, why play?


Back to Top

Post your reply: