Analysis spits out wrong variations!

Has anyone notice that the computer analysis frequently suggests LOSING variations?  And one guy has a blog post about a perfect game according to chess.com computer... better think again!

I've notice this is more than a few of my games I've submitted for analysis.  I did email technical support and the reply was "computers mess up sometimes... they aren't perfect." Are you serious!

Here is just one example.  Check out the movelist/ suggestion for move 7!  g4 was a stronger move, but then it continues with an inferior suggestion of Qb4 and Qb5+ which flat out loses a queen!  I double checked it over and over and I'm sure this is losing.  This has happened way too often.  I hope by putting it in the public something will be done.  I especially hate bad customer service... especially when I feel that the founder is a good guy and wouldn't tolerate it.  Members pay for this option and the response was disappointing.  A simple, "we are working on it," would have sufficed.

It's called the Horizon Effect. Computer stops thinking after Qb5+ Bd7. Had it looked two ply deeper, then it would have noticed black can always capture the queen, but it has to stop somewhere.

It's a limitation of computer chess. Of course, if you actually were to play the computer, it would also spend time thinking before making his next moves, and then it would be able to see it coming and avoid it. But just letting it think over a position at move 7, it may well give a line that turns out to blunder a queen on move 13.

Most of the time though, that doesn't mean that the first move of the line is a blunder. But the deeper in the line you get, the less trustworthy it is. At the end of the line it has literally seen 0 moves ahead.

Edit: Also, there's not much they can do about it. Let the computer think for twenty times longer (costs twenty times as much) and then it'll be able to look 1 ply deeper.

Something to remember is that computers and humans don't play chess the same way. In a game recently I gave up a knight for two pawns which would ultimately lead to 3, linked, passed pawns on the queenside(!), not to mention I was pawn and knight up anyway, and it told me the move was a blunder! A computer can't always be expected to understand that kind of move, especially if it's just tree-searching - there is no way it could have seen ahead enough moves to see a pawn get queened.

Chess.coms computer is not the best available, either, so I'm told. But it has found for me:

*forced mates / better attack handling I missed for both players.

*subtle, unnatural-looking defences where my opponent resigned or was mated that I did not see, and had assumed (as did they) that they were already lost.

*many material winning combinations either player missed.

Many games I have won, my opponents moves have been listed as never making anything worse than a couple of innaccuracies, yet mine are full of mistakes and blunders- it simply goes from a score like (+0.84) to (+7.62) without so much as a by-your-leave. So how did I win? Because I'm a human, and it's a computer program. It's better than me at chess, but it doesn't know *everything*. I use the analysis here as a supplement, and because I can forward the generated message to my opponent if it turns up something cute. And often when I think it's wrong, closer inspection reveals that I am, so don't get into the habit of thinking it's rubbish and ignoring any suggestion you don't like at face value.

However, no I don't understand the continuation above either, but that doesn't mean 7.g4 wasn't good.

I see.  Well the only comparision I have it for Fritz or Shredder... and I never had that problem.  Even a simple explanation that you provided would have been better than "computers mess up sometime."

I agree, g4 was good... all the way up to Qb4 ... I think that was the Horizon problem my friend enlightened up about.  For those who are clueless here is a wiki write up.

The horizon effect (or horizon problem) is a problem in artificial intelligence.

When searching a large game tree (for instance using minimax or alpha-beta pruning) it is often not feasible to search the entire tree, so the tree is only searched down to a certain depth. This results in the horizon effect where a significant change exists just over the "horizon" (slightly beyond the depth the tree has been searched). Evaluating the partial tree thus gives a misleading result.

An example of the horizon effect occurs when some negative event is inevitable but postponable, but because only a partial game tree has been analysed, it will appear to the system that the event can be avoided when in fact this is not the case. For example, in chess, assume a situation where black is searching the game tree to the six plies depth and from current position, it can see that it is going to lose queen in the sixth ply. Now let there be such a combination that by sacrificing a rook, the AI can push the loss of the queen to the eight ply. This is of course a bad move, because it leads to losing not only the queen, but also a rook. However, since the loss of the queen was pushed over the horizon of search, it is not discovered and evaluated by the search. Sacrificing of the rook seems to be better than losing the queen, so the sacricifing move is returned as the best option.

The horizon effect can be mitigated by extending the search algorithm with a quiescence search. This gives the search algorithm ability to look beyond its horizon for a certain class of moves of major importance to the game state, such as captures.

Rewriting the evaluation function for leaf nodes and/or analyzing sufficiently more nodes will solve many horizon effect problems.

Although, looking 5 moves deep only, I don't see how the CPU can be considered ~2500 strength!!

Have any really strong players personally assessed the strength of the program?

Well... I know that it's not as good as say... Rybka or anything like that... but I didn't expect it to spit out losing variations as I've never run into the horizon problem before...or I'm sure I have... just not so gross... It makes sense though... that's why fritz analysis starts from the end of the game and works backwards... my game was analyzed in full though... I only showed 11 moves.  Now I agree partially with the above post... it has definitely been helpful to my games and convenient as I don't want to download the pgn, transfer to another program and then analyze.  I like to hit the button and receiving a message when it is ready.  But once I started going through a lot of my games, I noticed this "horizon problem" with the variations...

Of course, it causes credibility issues.  The tricky thing about it... is that the suggestion is often right in the first few moves, but toward the end can lead to a devastating continuation.

I've noticed this before.  Ironically, Crafty *does* have additional protections for this sort of thing.  If captures or significant threats (at least check, possibly material threat like the queen/bishop deal shown here) have occurred, it specifically increases the search depth to avoid the horizon problem.

However, this won't work when the differentiating factor between membership levels is the number of moves the engine checks.  Since we all have a hard limit to the levels of Crafty's tree, it will suggest things it cannot completely verify.

I haven't used Crafty in a while, but I believe some of the options might work better.  I believe you can limit Crafty by time instead of tree depth, which might be a more efficient way to limit its rating.

Alternately, particularly since the source code is freely available, you might be able to allow captures and threats to be exceptions to the move limit, allowing it to check past the tree limit to avoid the horizon problem.

Yeah I also like the "hit the button" thing, because otherwise I would forget about some of the games I want to analyse. When you don't see eye to eye with the computer here, that might be a good time to load up something stronger. Nice outputs though, I don't believe another chess site has anything like it.

chess.coms computer must be doing this already, it can find a forced mate-in-more-than-5 after all; but the number of instances which trigger a search to quiescience are obviously limited.

Here is another one.  Look at the recommended continuation for move 8.  This leads to me getting mated!  Ironically... this was a game against Erik, the founder.

When I analysis the computers strenth is ~2200

Actually, the one thing they can do about it is to simply not show the variations to the same depth as the actual calculation that way the really short-sighted moves at the end of the variation won't be shown.  It's not a perfect solution, but it's something.

Truly that's the best way to analyze your games.

I use the chess.com analysis as a thumb-in-the-air indicator of how I played, but I understand that I'm sharing the CPU resources with many other users so it's inevitably going to be an inferior analysis to one in which I control the parameters on my own PC which is dedicated to the task.

You are so right.

But what I don't understand is why sometimes the computer looks 12 moves ahead at one point and, say, 15 moves ahead another time.

"Endgame tables", the "quiescience search" talked about above - heaps of reasons. Some programs will automatically switch to a deeper search depth when more and more material comes off the board (it takes less time to analyse every move if there are less available).

Engines which can be limited by time do this (obviously) as do some programs intended as computer opponents rather than analysis engines, so that they play at a reasonable pace.

Am glad am not the only one that thinks the Analisis are not always correct... Specially in certain variations when i compare to what grandmasters say.

Just for kicks, I had Chess.com's computer analysis done on a 20-move match that I was winning from the 3rd or fourth move. The analysis was beyond belief. Now, I'm not complaining, but to say the least I did not agree with some of its findings. This is why I prefer human analysis.

I got one the other day which suggested I play a different move than the one I played. After my move, I had score ~ +4pawns. The variation it suggested instead terminated in the statement "drawish position" (!)

Another thing I find funny is when it tells you zero blunders were made, even when you walk into an avoidable forced mate! (This is actually a parsing issue... mistakes and blunders are ony identified by a certain amount of change in the score)

In a recent game, I was starting to lose, because I let a strong knight get into my position. I was able to turn the game around via offering the exhange of my rook for opponents fianchetto bishop... he accepted and I was gifted a resign about 10 moves later. If my counterattack had fizzled, I would have lost the ending, but it worked like a charm... The computer *hated* the exchange sac... "You are now losing!!" etc... despite the fact (in the CPU's opinion) my opponent played almost perfectly from that point on and (by its own calculation) the score increased pretty dramatically in my favour each turn. But of course neither I nor the CPU had seen a full 10 moves into the future, at least not for all replies, but a bit of human logic and educated guessing on my part was the key.

OTOH, in that same game, a move or two before winning, I actually missed a forced mate that wasn't in line with my train of thought or my plan. Guess what the computer *did* find... this is what they're really good for, finding moves that are outside of the familiar ideas in the position. And as many people have pointed out, very deep analysis would be very consuming of CPU cycles, and totally understandable that chess.com can't provide such features.

Bottom line? Go to computers for tactical advice and get your strategical understanding from books and experience. Until you can consistently beat a given computer program comfortably, there has to be something it can teach you.