ChatGPT Gives Terrible Chess Advice

Artificial intelligence is well on its way to controlling the world, or so the computers would have us believe. In reality, computers on their own are still pretty dumb things, even when they appear smart.

We spent some time asking ChatGPT some chess questions and while it said some things that made sense, boy howdy, there is also so much it does not know. If you want to feel better about your own chess understanding, seeing what GPT thinks about the game will help.

• Forget About Scholar's Mate
• Or Any Checkmate For That Matter
• Stalemate vs. Checkmate
• Can It Even Read Notation
• Strategic Advice
• Some Bizarre Ideas About The Opening
• Is Chess Funny?
• Conclusion

Forget About Scholar's Mate

Checkmate is the most important part of a chess game, winning immediately. Unfortunately, ChatGPT doesn't necessarily agree. For one, it doesn't know how to perform either of the most basic checkmates in chess: Scholar's Mate or Fool's Mate!

And so if you go looking to ChatGPT for advice on what to do when facing Scholar's Mate, it simply recommends that you blunder into it. (Note: You may need to register for a free OpenAI account to see the queries linked in this article). The correct solution to this puzzle is to give the ChatGPT move, not the actual best move in the position:

GPT claims that, "After 3...Nf6, White will likely need to move the queen again, and Black can continue developing pieces and aiming for a solid position." White will indeed likely move the queen again—it got that much correct—but what GPT missed is that she will deliver mate on f7 and then take a well-deserved early vacation.

Meanwhile, here was the AI's advice when presented with Fool's Mate, this time being asked to play the mate. It did not go any better.

Or Any Checkmate For That Matter

Every chess player knows that checkmate ends the game on the spot. So should you play a checkmate when you have the opportunity to do so?

Well, according to ChatGPT, it's debatable! Although maybe that's not too surprising given its desire not to play 2...Qh4# when it has the chance.

ChatGPT in general likes to give balanced answers when presented with a dichotomy... even if the answer makes no sense to our human minds.

Stalemate vs. Checkmate

Is it any better at finding endgame checkmates? We gave it a simple scenario to find out. Once again, your goal in this puzzle is to find GPT's suggestion, not the correct move. Keep in mind that ChatGPT does indeed know what stalemate is.

It knows what stalemate is, it just doesn't care. As with the question of whether to give checkmate or win a queen, GPT seems to consider the queen's value as a piece more important than winning the game on the spot.

Can It Even Read Notation?

No, not really. This would be fine, really, except that if you ask ChatGPT if it can read the notation used to feed chess positions to a computer, called FEN, it cheerfully says that it can.

But trust us, it can't. When provided the FEN notation of "6rk/6pp/8/6N1/8/8/8/7K w - - 0 1", where this is the actual position represented (you can plug it into our analysis tool and see for yourself):

ChatGPT produces this masterpiece instead (we had to estimate certain piece positions):

As you can see, not close. So if you expect to get good chess advice from GPT by giving it a position, you will be disappointed.

Of course, even if ChatGPT could read FEN, it probably wouldn't give the right move if its reaction to the Scholar's and Fool's Mates is any indication. In fact, in the position we gave it here, it suggested the illegal move Ng6. Can't get worse advice than, "Make an illegal move"!

Strategic Advice

If GPT is no good at giving specific chess advice, what about generalized chess strategy? GPT is a little better here, but sometimes it still produces nonsense such as:

One of the first thing a new chess player notices is that bishops are stuck on one color, but GPT still happily suggests that you "position your bishop on a light square." GPT also accepts the assumption of the question, that light-square control is always a desirable strategy, regardless of board conditions. What good is light-square control if you get mated on the dark squares? It's not a consideration for the AI.

"Garbage in, garbage out", as they say. 

Some Bizarre Ideas About The Opening

If you want to know about general opening principles, ChatGPT can help, although it won't be as good as our page on openings.

But it also might say something like: "Instead of the traditional fianchetto with bishops, try fianchettoing your knights. This can lead to unusual pawn structures and surprise your opponent who might be expecting a more typical setup."

Even if we're asking it to be unconventional, it should at least say something that makes some sense. Fianchettoing a knight—never mind why you'd ever want to—would take four moves to set up (g3, Nh3, Nf4, Ng2) rather than the two for a bishop (g3, Bg2).

And, here again, you can just forget about getting specific. Watch it struggle to identify the moves of the Englund Gambit and barely go one-for-two in determining the point of that opening: "The Englund Gambit (1. d4, 2. e4, 3. Nf3, 4. Bd3, 5. exd4): As Black, sacrifice a pawn early to disrupt White's pawn structure and create open lines for your pieces."

Disrupting White's pawn structure might happen in this opening, but they've still won a pawn, and it's not exactly the point of the Englund Gambit.

Is Chess Funny?

GPT loves answering simple questions in long, numbered lists. To be fair, it has probably been trained on tons of list-based articles.

When we asked it to play chess "funny", it did not read that as, "Play some funny moves". It read that as, "Give me a long list of bad jokes I can make while playing chess."

Try these in a tournament and you'll be banned; try them in a park and you'll be laughed out of there, never able to show your face again; try them on a friend and, well, hopefully you have more respect for your friends than that.

Conclusion

We used the latest free ChatGPT, the 3.5 version, to come up with these nuggets of anti-wisdom. Perhaps the premium version, ChatGPT 4, which can access the internet, would do better. But if you don't want to pay OpenAI for a subscription, beware any chess advice from ChatGPT. 

It's best to just stick with our articles, news, chess terms, player bios, and more. Leave chess to the chess people, and leave taking over the world to the computers.

Have you ever asked ChatGPT for chess advice? What brilliant things did it tell you? What was your favorite bad advice from this article? Let us know in the comments!