Does the brain imagine chess in words or pictures?

I have a puzzle for you. The good news is that it isn't very hard - I'll even tell you that it's a Mate-in-2 for White. The bad news is that I'm not going to let you see the position. Instead, you'll just have to do your best with the following description of the board:

White King on e5, White Queen on g4. Black King on h8.

...and here's a few blank rows to not spoil the experience!

.

.

.

.

.

Sure, you could open up a board editor and set this up, but I want you to try and solve it without doing so. While you're doing that, my real question isn't about whether or not you solve the puzzle but what your experience is of trying to do so. What are you thinking about? What's happening in your mind while you try to figure out the answer?

This puzzle, by the way, is the first of Martin B. Justesen's 1160 Blindfold Chess Problems  which is a wonderful e-book filled with increasingly challenging puzzles all to be solved without a board. I also won't keep you in suspense any longer in case you (like me) find blindfold problems very difficult. Here's the position I just described.

I'm guessing that this helps a lot. In case you really hate spoilers I won't post the answer here, but I'm confident that most readers will see the answer very quickly. By contrast, I suspect that trying to think this through without a board to look at was quite challenging, even with this sparse position.   Despite this apparent difficulty, many past and present chess players have demonstrated remarkable abilities to work through complex positions without looking at a board: Philidor, Alekhine, Morphy (pictured below) and a host of other famous players were  (and are!) capable of winning multiple blindfold games in simultaneous exhibitions.

Blindfold chess  at this level is just a particularly startling demonstration of an ability nearly all chess players try to use in normal play - imagining what comes next. During a game, we all try to think through the consequences of candidate moves so we can decide whether particular next steps are good or bad. Like playing blindfold chess, this can be quite difficult. I tend to feel confident about calculating about two or three moves ahead, but beyond that I start to lose some kind of clarity about exactly what is where. This is an imperfect analogy, but it feels a little like the "Fog of War" variant in which some of the board is obscured during play.

A key part of my experience of trying to calculate is that I do feel like I'm picturing something, though. I describe my estimate of the board as "clear" vs. "foggy"  because it seems like an image I either can or can't see very well. That's just my intuition, though - Is that really what my brain and my mind (or yours)  are doing when I calculate?

You might wonder what the alternative is - doesn't everyone have a mental picture they use when they try to think ahead OTB? The answer, which has become the focus of a lot of interesting visual cognition research lately, is no! Different people report very different experiences of visualizing things in the "mind's eye," with some individuals reporting no visual imagery at all. This condition is called aphantasia and is the extreme end of a spectrum of visual imagery that researchers usually assess with simple surveys.

If you have no mental image at all, or perhaps a very limited one, how do you possibly calculate? What is imagination without an image?  Individuals with aphantasia report using a lot of different strategies, but one I want to focus on is the use of verbal descriptions rather than pictorial descriptions. If you've ever watched Hikaru Nakamura calculate on his stream, you've certainly seen images like the one below, suggesting a highly pictorial representation of the board.

On the other hand, the same streams also often include him rattling off moves out loud incredibly quickly! "Nf3-g4-Nh5-Kh7-captures-captures-and-mate" for example, though please be aware that I just made that one up! That kind of quick description of a sequence of moves in words might be evidence of an equally strong verbal representation of the board that can be expressed in words or symbols. So here's the interesting question if you're someone who thinks about vision, cognitive science, and chess - what kind of imagination, verbal or pictorial, do players tend to rely on the most? A classic model of human working memory suggests people can use a cognitive module we call the visuospatial sketchpad to keep track of pictorial and spatial information and a different module we call the phonological loop to keep track of verbal content. Which of these modules do chess players use most?

This is a hard question to answer. The problem with trying to understand and measure someone's imagination is that it all happens inside the mind without any obvious outward signs. There are some interesting candidates for objective measurements of imagery, but many of these have proven difficult to replicate and are thus hard to rely on. So what do we do? Do we just ask people and hope that they have good insight into their own mental processes? Unfortunately, we can't just rely on that kind of introspection because metacognition (thinking about your own thinking) is not always reliable or valid.

It turns out that there is a way to measure how  verbal and pictorial imagination each contribute to a complex task like chess calculation. The big idea is to use task interference to measure how much one can interrupt the main process we're interested in (at present, chess calculation) by asking people to do something else at the same time. The logic behind this approach is that if the main task and the interfering task rely on the same ability, it should be tough to do both at the same time. On the other hand, if they rely on different abilities, doing both shouldn't be so difficult.

This is the strategy reported in Saariluoma (1992), in which the researchers worked with 8 chess masters (average ELO over 2200) and 8 intermediate players (average ELO under 1750) to test how visuospatial and verbal interference affected chess calculation. The main task both groups were asked to complete was "Mate-in-1" detection: Given a position with White to move, can Black be checkmated on the next move? For half of the trials the answer was yes, while in the other half White would need 2 moves to win. 

But now here comes the interesting part: While players were trying to work this out, the experimenters asked them to do one of two secondary tasks at the same time. In their Verbal task, each player was asked to repeat a novel word over and over again (the word was TAIKURI) to occupy their phonological loop. In their Visual task, each player was asked to imagine tracing a path around the letters of the word TIIKERI and deciding if the turn they would have to take at each corner they arrived at was a left or right turn. This task was intended to occupy their visuospatial sketchpad. If either of these interfering tasks makes solving Mate-in-1 puzzles harder, that's good evidence that you need that kind of imagination to help you. You can seetThe results (graphed here as the time it took to answer the Mate-in-1 question correctly) depicted below.

What these bars show you for the Masters (at left) and the Medium (or Intermediate) players at right is that repeating a word over and over again doesn't make you any slower to detect Mate-in-1. The black "Control" bar shows you the time it took to solve these mating puzzles with no secondary task interfering with you, so the hatched bar being about the same height  as this one means this secondary verbal task didn't make the main task any tougher. However, the secondary visual task does interfere with both groups' speed! That dark-grey bar being higher than the other two in each case means occupying your visual imagination with another task made chess puzzle-solving harder. The conclusion is that even though verbal and auditory information seems like it could play a role in how you calculate and imagine a board, ultimately, chess appears to be more visuospatial. than verbal.

Something that I think is an interesting question related to this topic has to do with training. To what extent can we improve our visual imagery? What are the most successful kinds of training to do so, if any? Also, what about those individuals with aphantasia? If you don't have access to a mental image at all, what is calculation like? As always, there are remain a number of intriguing next steps to take from here, but this study is a neat way to see how we can try to peek "under the hood" with careful cognitive testing to understand how the brain and the mind help us play the game.

References

Saariluoma, P. (1992). Visuospatial and articulatory interference in chess players' information intake. Applied Cognitive Psychology, 6(1), 77–89. https://doi.org/10.1002/acp.2350060105