One of the main features which attracted me to try out a Premium Membership is the ability to receive (computer) analyses of one's games. It doesn't much matter to me whether the analysis I receive is from a human or computer, so long as it's correct. Like everyone, I'm interested in the analysis to learn where I can improve my play.
My issue is that it seems some of the computer analyses I'm receiving from this site are simply completely wrong.
I assume there are many available chess computer programs available to Chess.com which would correctly analyse this (and other) positions. Why are the present computer analyses so flawed? I've read similar comments from others.
What am I missing here?
The purpose of my post is certainly not to defend the logic of my 8th move in the above game but to figure out what is wrong with the computer analysis or my interpretation of the information it is giving me.
ps. On an unrelated topic, after I inserted the pgn of the computer analysis, all of the remaining text in this window became centered ... how do I format it to become left-justified again?
What you're seeing here is called the "horizon effect." Computers determine a best move by brute force calculations. Since the computer analysis is limited to a certain number of moves, it doesn't see something that exists just beyond those moves. Even though a person can look at that final position and recognize almost immediately that mate is inevitable, the computer can't see it because it has reached the artificially placed limit on its calculation. If it were allowed to delve a little deeper into the position, it would find the mate and its evaluation would change. The best thing for us to do, then, is not to blindly trust the computer evaluations, but double check the positions near the end of the variations. Keep in mind that that is a 9-move sequence, and it is possible that Black had a better move slightly before this, but that the computer couldn't analyze all the way through the variations to find it.
we use a VERY strong chess engine. the problem is, as stated above, the horizon effect. we don't go 100% deep on all variations.
Wouldn't it make sense to trim the lines to a point where the horizon effect is obscured? If you're analyzing to, say a depth of 20 ply, wouldn't only displaying the first 10 ply mitigate this issue substantially?
I have had games where I am a couple of pawns up and the computer analysis suggests a line which at the point the analysis ends I get mated in one - nice!
Sorry Erik, but your engine is crap. I have a game where it decides another move is better than a mate in one. Rybka disagreed though ;)
Thx for the replies people.
First off, I should say I absolutely love the site. First rate. But ... I'm not totally convinced that the Computer Analysis portion of the site is all it could be.
I understand the concept of the 'Horizon Effect' and in the example I gave above, we might concede that this was the problem with how the analysis reported the final position. It just didn't look past it's last move and analyzed the position there.
But what about an analysis (a different one) where the variation only goes 3 moves deep. The horizon isn't very 'far' in this case and the position as it is evaluated then, on that very move, is wrong. See below:
In this case White doesn't have a 'decisive advantage' ... the game is over! Black is in checkmate.
I'm not quibbling about the use of terminology here but about the fact that the computer analysis on a variation which is only 3 moves deep (or 2 moves really since it occurs on white's 2nd move in this variation) fails to report that the game is over.
Still not convinced there isn't something wrong here.
I had posted a comment here about problems uploading the image in my previous post but it seems to have been corrected. The image now displays.
Can you tell us which engine you use, or at least the ELO?
i heard it was 2600 for diamond members
Just get Fritz 12. It's great.
I was going to post several more examples of what I consider to be oddities in the results of the Computer Analysis of games (or in my estimation 'evaluation errors') but it would just amount to re-stating the obvious. In most cases these evaluations are given in alternate lines which are just a few moves deep so I'm not sure the horizon effect is really a valid 'defense' here (they would be examples similar to the diagram I posted above where the analysis engine is claiming one side as a drawish position or decisive advantage when in fact the situation is quite the opposite on the very next move or is already over by checkmate).
(By the way, I'm feeling guilty for doling out some negative feedback with respect to a site I think so highly of but maybe we can improve this part of the site or member features and make a great site even greater.)
Would it be possible, if Chess.com has control over this feature in the analysis, to at least have your program evaluate the positions at the end of the proposed alternate lines before it provides an 'opinion' of the position. This would avoid what appears to me to be incorrect evaluations.
The problem in trusting some of the alternative lines proposed by the computer analysis is that if the end result of that proposed line is completely wrong, was that alternative line ever any good and if so, where does it begin to fall apart.
Thanks for that feedback Schachgeek.
Based on the fact that you won the game I have to assume there were no other mitigating circumstances to the analysis giving you a ? on a move which certainly sounds was a winning move.
Did the Computer Analysis recommend a better move which led to say the capture of the queen without trading material or capturing in a different manner which would leave you in a better position or was it just plain wrong?
I understand the concept of a 'horizon effect' which Erik is referring to but there seems to be more going on with the analysis oddities than that. Your example seems to be one of them.
Congrats on your win by the way. You must play a pretty solid game :)
I believe that the evaluation that the computer analysis gives for a variation is its evaluation looking at the position from the first move. it never executes the rest of the moves of the variation, those are just the "top" variation that then gets displayed. so when it says decisive advantage, that is the evaluation it says you have if you play Qd8+, which is correct.
TheGrobe, you have made the same suggestion as I have put in for this issue, it just has not been implemented yet. the last 2 ply of a variation are essentially random, and usually would be replaced by other moves once the computer got closer to that position; thus including them is useless.
what is accurate, and does reflect a very strong computer strength are: the initial move suggestion and the evaluation of that move. for example in your first game, the computer is absolutely right that h6 should be prefered over ne7-- and also that Ne7 leads to a white advantage, while black has an acceptable game with h6.
what schachgeek cites, putting a question mark on a mating move, sounds disturbingly like a problem. if you still have that computer analysis, maybe you could send it to me, so we can try to figure out what happened there?
I've seen Master Games where a "?" move leads to a mating sequence. This is not uncommon, especially in the case where a sacrifice leads to a brilliancy. An opponent can always choose the 2nd best option if they see the continuation, which might not then lead to mate.
(edit) Contemporary engines might not give moves like this a "?" if they are able to correctly analyze the position.
David (IM dpruess), honoured that you would join in (love your lessons by the way, just starting them). Thanks.
Your comment about the analysis' evaluation (e.g. 'white/black has a decisive advantage', 'drawish position' etc.) applying to the first move of a sequence as opposed to the last move of the sequence might explain what appear to me to be erroneous evaluations and you may very well be right.
But, it doesn't make much sense to me however that the evaluation comment, if it did apply to the first move, is held back by the computer analysis and posted against the last move of the sequence, where it may or may not be accurate at that stage.
By the way, I'm not complaining that the computer initially suggested an incorrect move at the beginning of the sequence (e.g. in my first example 8. ...h6 is clearly better) but I'm trying to learn from those analyses. To me, the reason for a suggested alternate move should be that it leads to a superior position. What I'm tring to learn usually is why make this move (which should become obvious by following the alternate line). If the outcome of that alternate line is not evaluated properly, it casts potential doubts (in the mind of this not so experienced player) on the validity of other earlier moves in the line. I'll try to find some other examples in my analyzed games to illustrate.
Schachgeek, if you still have the analysis you referred to, could you post it here so we can follow along?
i agree, caparojo (and glad you like my lessons).
to make it easier for the student to learn from the computer analysis, i should add: display the evaluation on the first move, instead of at the end of the line. add that to getting rid of the final 1-3 ply which are often nonsensical, and people will feel much clearer about what they can get from the computer analysis.
do it! if you feel the cause of the compliants are the 'misleading' computer notes rather than the actual evaluation then do it!
I have to say guys, as a beginner, I came here to learn how to be better at chess, and having analyzed some games that I've won (if I can see my own mistakes I don't bother getting someone/thing else to tell me/rub it in) to try and learn how to be better, I don't expect the computer evaluation to tell me I made an error then show me alternate moves that lead to a loss, or a weak position, when I've won that game. Surely as the computer is moving both sides from that point, it should have a good reason for making the suggestions it does.
I'm a big fan of learning by losing, it works! and trust me it's readily available here. Computer evaluation should be a way to learn even though you're winning, by showing you clearly how you could have shortened the game. So either the horizon should be 2-4 moves ahead to reflect spontaneity of basic human play (at my level) or all the way to the end of the game, to show fastest solutions. (even better, it should be adjustible)
Personally, as a beginner, I would prefer a 100% depth to the analysis I ask for, that way I learn how to shorten my game by stopping missing strong positions. I don't mind waiting for it, after all, it already tells me 'this may take several hours' and then is ready in minutes
I occasionally teach in my field so I understand how difficult it is to produce a training tool that utilises yet ignores the depth of my knowledge in my subject and builds on and enhances the knowledge of the person I'm teaching.
On the other hand the mentor program is almost brilliant, my only problem with it were the beginner stages when there are sometimes loads of 'alternate correct moves' for no apparent reason, surely thats a basic core programming bug that could be fixed? the ability to assign different answers to these alternate correct moves proves that a slightly deeper programming tweak could allow multiple correct moves.
I apologise if I have just repeated a previous point, however I'm only just learning the terminology therefore didn't use any.
I think the whole idea of a non-master learning chess from computer analysis is completely misguided in the first place, if by "learning" you mean what most of us mean, which is an increase in understanding. No chess playing computer in the world understands chess at all, rather they simply calculate variations out to some specified depth, possibly augmenting that depth in positions with many threats present. This is actually the source of the horizon effect. In certain calculated variations, the computer may reach a quiet position at the end of its standard search depth. When that is the case, most computers simply evaluate that position using an evaluation function which is little more than a material count plus some rather unsophisticated heuristics regarding pawn structure, king safety, etc. The computer understands nothing of that position, it just returns an evaluation. In almost all cases, that is fine, but in some cases, the computer's evaluation function will favor a strategically lost position because it is only ever making a static analysis, and there may be a very dynamic plan available to the side with static disadvantages. A grandmaster wouldn't make those mistakes.
The lesson? Until you're a super grandmaster, do all of your learning by reading what grandmasters have to say. Limit your use of chess playing computers to using them as sparring partners and to check your own tactical analysis (*after* you've made it). You'll learn nothing more than the obvious from chess computers as they exist nowadays. But grandmasters have published known errors, I hear you say. Yes, this is true. If you are memorizing without actively learning, then you will suffer for those errors, and you deserve to. But a grandmaster can teach you more about chess with an explanation of a 10 move variation of 10 bad moves in a row than a computer can teach you by showing you what it has calculated (possibly erroneously) as the best variation for both sides without explanation.
This whole notion of "the correct move" is counter-productive to learning. On the one hand, there may in fact be a "correct move" in a position; if this is the case, then chess becomes a combinatorial problem, requiring no understanding whatsoever, only memory or arithmetic - in other words, it is not worth playing, except to prove the obvious fact that some people possess more knowledge of a narrow subject than others. On the other hand, there may in fact be no such thing as one or even a few "correct moves" in many positions; if this is the case, then it is pointless to search for one, and even harmful when misled by a computer that claims to have found one.
The idea of trimming variations to avoid the horizon effect is faulty. The computer's evaluation of a move would still reflect the flawed evaluation of the entire variation as a result of the very same horizon effect. So how does that help? As a human, you should look at the whole variation that led to that evaluation and ask yourself whether that makes any sense. Until chess is "solved", the human - not the computer - should make the final decision. And after chess is solved, well, why play?