Positions that break Stockfish
After seeing an old video by Eric Rosen showcasing a position that Stockfish does not understand, I was fascinated and decided to see just how far you could push Stockfish. With this goal in mind I decided to try creating the most absurd possible positions that still resulted in Stockfish giving an evaluation (albiet the evaluations weren't always correct).
Position from Eric's video
For those that have not seen this video or are confused about why this "breaks" Stockfish, it is because black has a whopping +48 material advantage on the back two ranks, but it is all paralyzed barring none of it gets captured by white. This essentially just makes the chess board 6x8 rather than 8x8 and both sides are roughly equal (white cant castle but black can never promote). The next few positions I'm going to show you follow the same concept of paralyzing a massive material advantage.
White is +109, but it is a draw
In this position, Stockfish will initially not know how to evaluate the position at all. When I loaded this position Stockfish said it was a Mate in 5 for white, but then could not load any moves to prove this outlook. After making a couple king moves it snaps out of it and starts giving random evaluations, sometimes it gave at least one line it said was a draw (correct evaluation) but at other times it gave as much as -149 for black! I believe the engine thinks black has an overwhelming advantage because unlike white, black is capable of moving pieces other than its king (b6 knight, e7 and g7 bishops all have moves). These moves however are all out of the question because capturing any piece gives white the breathing room it needs to break out and win easily. Despite the wild evaluations in favor of both sides, this position is completely drawn as long as black only moves his king.
White is +56, but it's still a draw.
This is a less crazy position, and should be an obvious draw as every piece for both sides is completely paralyzed despite white's +56 advantage. However, this was another position where Stockfish would give white a completely winning evaluation of Mate in 2/3, but only if it was white to move. With black to move Stockfish correctly stated it was a draw.
"It's not about having the bishop pair, it's about how many pairs of bishops you have." -Aman Hambleton
This one is in my opinion the most interesting position I have shown in this post. Unlike the previous positions, neither side has a particularly ridiculous advantage. Black is actually the one up with +6 points of material, however I believe that does not actually affect the evaluation in any way in this case. The reason this one is particularly strange is because with the previous positions, Stockfish would always run out of ideas or figure out the position was drawn in certain instances. In this position, Stockfish consistently kept its evaluation at Mate in 6/7 for white depending on who's move it was. With the black king running among the 12 white bishops and the white queen, it would seem that things were looking bleak for black, but with the bishops all paralyzed and blocking all horizontal and vertical movement for the queen, the black king can actually never be mated. In this position I believe Stockfish gives the mating evaluation to white because the a4 square looks rather poisonous for black at first glance. If the Black king resides on a4 and it is white to move, white could potentially win with Qc2#.
The only potential checkmate on the board?
For those of you who thought this, this "potential checkmate" is not only easily avoidable for Black by just staying off the a4 square, but it is actually impossible for white to force Black to go to a4! The only square that a4 can be accessed from by the black king is b3, and the only square where the white queen can give check to the black king on b3 without being captured is d1. In that position, not only does black have a2 and c4 to run away to safely, but black couldn't actually go to a4 and get mated even if he wanted to because the square is x-ray defended by the white queen while it checks on d1. Therefore, this position is drawn even with best play.
Thank you for spending the time to read this long article, I hope you found this as fascinating as I did while I was experimenting with these positions! In conclusion, I initially thought the common theme in creating these positions was giving one side an overwhelming material advantage that is impossible to use. However with the last position material was fairly close to even, yet it was the only position where Stockfish never once gave a correct evaluation. I encourage anyone reading this to open an analysis board and try to experiment finding your own unique positions that confuse the engine, or perhaps come up with your own theories as to why these positions can't be evaluated.
