How to Write a Chess Opening Book for Stockfish using Python

Introduction

Chess engines for all their amazing ability do lack one thing... a decent opening book. Of course, engines you play online have an opening book, but if you were to use the standard Stockfish engine in your software or hardware projects, you'll find it's quite ignorant of the chess openings you and I are used to. In fact, many engines tend to play the same opening moves, at any given depth. In the absence of any opening knowledge, the engine will calculate its initial moves, which although acceptable tactically, may be weak strategically. So adding an opening book strengthens the engine's game, and makes the chess more varied. It's also nice to play the openings we're familiar with. And it's quite easy to do using the programming language, Python.

Some of the code here is taken from my blog, "How to integrate Stockfish into your projects using Python" and I'll show how to add an opening book to that. If you've not read the blog, you may want to do that first, as you'll need to download the Stockfish executable and set up a Python environment, such as Thonny.

The ideas I present here can be used with any chess engine, e.g. Leela Chess Zero. I have also used it in my 3Robot chess project.

Creating an Opening Book

An opening book is a long list of lines, each line consisting of a sequence of moves. These lines are known to be good chess, well-tested over the years. One such line is the Ruy Lopez with its moves, "e2e4 e7e5 g1f3 b8c6 f1b5". One way to package up this data in Python is:

    book = [
        "e2e4 e7e5 g1f3 b8c6 f1b5", # Ruy Lopez
        "e2e4 e7e5 g1f3 b8c6 f1c4", # Italian
        "e2e4 e7e5 b1c3", # Vienna          
        "e2e4 c7c5", # Sicilian
        "e2e4 d7d5", # Scandi    
        "d2d4 d7d5 c2c4", # Queen's Gambit
        "d2d4 f7f5", # Dutch
        "c2c4 e7e5" # English
    ]

Here we have various lines, stored as strings in an array called "book". It's far from complete, but there's enough here for demonstration purposes. It's up to you to add your favourite lines and make it into a proper opening book!

This layout means there's a lot of repetition if you want to include variations of a particular line, but at least it's clear and easy to read.

Of course, if you were a programmer back in the 1980s writing this in machine code for a dedicated chess computer with very limited memory, you would not use this data structure! Instead, a tree structure would be more suitable for memory efficiency.

How to Search the Opening Book

As the game unfolds we need to keep a list of the opening moves, which we use to search the opening book. Say, I play "e2e4". The computer looks for all the lines in the book array above that start with "e2e4", of which there are five, and extracts the next move, and builds an array of book moves, i.e. ["e7e5", "c7c5", "d7d5"]. One of these is then chosen at random.

Let's say the computer plays "e7e5". A list of the opening moves (a variable "openingMoves") is updated to "e2e4 e7e5".

I play "g1f3". openingMoves is updated to "e2e4 e7e5 g1f3". 

The computer looks in the book array and finds two lines that start with "e2e4 e7e5 g1f3". There is only one possible book move, "b8c6". openingMoves is updated to "e2e4 e7e5 g1f3 b8c6".

I play "f1b5". openingMoves is updated to "e2e4 e7e5 g1f3 b8c6 f1b5".

The computer searches the book array, and finds one matching line, but there's no next move. It's now said to be "out-of-book". A flag ("isInBook") is updated to false to prevent further searching, and the engine takes over.

And that's basically it. It's a simple exercise in string matching, and little else!

Let's play Chess

The following Python code allows you to play against Stockfish with the above opening book. Copy and paste this into any Python IDE, e.g. Thonny. Make sure to change "exePath" to point to your Stockfish exe program.

import subprocess
import random

exePath = "/home/linuxuser/Downloads/stockfish-64bit/stockfish-ubuntu-x86-64"

sf = subprocess.Popen(exePath, universal_newlines=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
fen = ''; isCheck = False; bestMove = ''; depth = 5
isInBook = True; openingMoves = "" # for opening book

book = [
    "e2e4 e7e5 g1f3 b8c6 f1b5", # Ruy Lopez
    "e2e4 e7e5 g1f3 b8c6 f1c4", # Italian
    "e2e4 e7e5 b1c3", # Vienna          
    "e2e4 c7c5", # Sicilian
    "e2e4 d7d5", # Scandi    
    "d2d4 d7d5 c2c4", # Queen's Gambit
    "d2d4 f7f5", # Dutch
    "c2c4 e7e5" # English
]

def sendUCI(cmd: str, sync: bool, printOutput: bool):
    # Send command
    cmd = f'{cmd}\n' 
    if sync: cmd += 'isready\n'
    sf.stdin.write(cmd); sf.stdin.flush()
    
    # Process output
    while True:
        line = sf.stdout.readline().strip()
                
        if line.startswith('Fen: '): # Get FEN if available
            global fen; fen = line.replace('Fen: ', '')
            
        if line.startswith('Checkers:'): # Is there a check?
            global isCheck; isCheck = line.replace('Checkers:', '') != ''
            
        if line.startswith('bestmove'): # Get computer move if available
            x = line.split(' ')
            global bestMove; bestMove = x[1]
            if printOutput: print(line)
            break
            
        if line == 'readyok':
            break

        if printOutput: print(line)
        
def printBoard():
    sendUCI('d', True, True)
    
def updateBoard(move: str):
    sendUCI(f'position fen {fen} moves {move}', True, False)
    sendUCI('d', True, False) # Get the FEN, don't display board

def resetBoard():
    sendUCI('position startpos', True, True)
    sendUCI('d', True, False) # Get the FEN, don't display board
    
def calculateMove():
    print('Stockfish is thinking...')
    sendUCI(f'go depth {depth}', False, True)
    
def isGameOver() -> bool:
    sendUCI('go depth 1', False, False) # Any moves available?
    return bestMove == '(none)'

def verifyOpeningBook():
    numErrors = 0
    for line in book:
        print("Verifying: ", line)
        moves = line.split()
        resetBoard()
        for move in moves:
            previousFen = fen
            updateBoard(move)
            if fen == previousFen: print("Oops, error!"); numErrors += 1
    print ("Errors in opening book:", numErrors)
    resetBoard()
    
def getOpeningBookMove(openingMoves: str) -> str:
    print ("Searching opening book")
    print ("Opening moves:", openingMoves)
    bookMoves = []
    for line in book:
        if line.startswith(openingMoves):
            print("Line found:", line)           
            line = line.replace(openingMoves, "") # remove previous moves to leave next moves
            
            if len(line) > 0:
                nextMoves = line.split() # get the next move in this line
                nextMove = nextMoves[0]
                
                if not(nextMove in bookMoves):
                    bookMoves.append(nextMove) # build an array of unique book moves
                
    print("Possible book moves:", bookMoves)
                
    if len(bookMoves) == 0: return "" # no book move
    if len(bookMoves) == 1: return bookMoves[0]
    
    randomIndex = random.randint(0, len(bookMoves) - 1) # choose a book move at random   
    bookMove = bookMoves[randomIndex]
    return bookMove

def updateOpeningMoves(move: str):
    global openingMoves
    if openingMoves == "":
        openingMoves = move
    else:
        openingMoves += " " + move
        
print("Verifying opening book")
verifyOpeningBook()

printBoard()

while not isGameOver():
    print('Enter move (e.g. e2e4) or skip')
    userMove = input()
    
    if userMove != "skip": # User can enter "skip" and get Stockfish to play next move
        previousFen = fen
        
        updateBoard(userMove)
        
        # Is move legal?
        while fen == previousFen:
            print('Illegal move! Try again')
            userMove = input()
            updateBoard(userMove)

        if isInBook: updateOpeningMoves(userMove)
        
        printBoard()
        
        if isGameOver(): break
    
    if isInBook:
        bestMove = getOpeningBookMove(openingMoves)
        if bestMove == "":
            print ("No book move found, using engine")
            isInBook = False
            calculateMove()
        else:
            print("Book move selected: ", bestMove)
    else:
        calculateMove() # Get Stockfish move
    
    updateBoard(bestMove)
    
    if isInBook: updateOpeningMoves(bestMove)
    
    printBoard()
    
    if isCheck: print('Check!')
    
print('Game over!')

All being well, you should see a screen like this.

The first thing the code does is verify the opening book. Each line is played out by Stockfish to make sure all moves are legal.

Now you're ready to play a game! Enter your move, or you can enter "skip" to make the computer play the move. You can continually skip so the computer plays itself. The computer will consult the opening book on each move until it runs out of book moves, at which point the engine takes over.

There are many ways you can randomly choose one of the possible book moves. My code creates an array "bookMoves" of unique moves, so each has the same chance of being picked. But you could remove the test to see if it's already in the array; then the most frequently occurring move is most likely to be chosen.

There's no need to add hundreds of lines to the opening book array, around thirty or forty is quite enough to provide a decent set of openings. My 3Robot chess computer only has around forty.

Conclusion

Adding an opening book to your chess project makes the chess more human-like, more enjoyable. It feels like real chess. I added an opening book to my 3Robot chess computer, and it makes a world of difference; before it was always playing knights before pawns, and that just got tedious and annoying. And it only took a few dozen lines of code to implement!

Happy chessing!