Scannable Scoresheets (free!)

TL;DR: Visit our website at reinechess.com to sign up to be notified for the app release for our scannable scoresheets, or try a demo of the software. The goal is this: write scoresheet, take picture, toss scoresheet, game added to a personal database ready-to-analyze.

EDIT: Reine is going open source! Anyone that wants to contribute is free to do so.

Our beta demo is up! Print out a scoresheet and go to our website on your phone (it'll let you take a picture there). If you don't want to print stuff out, go to our website on a computer and use our test scoresheet instead.

Hello world! As we pored over the (hopefully few) inaccuracies and mistakes of our tournaments this weekend, most of us plugged in dozens of moves from scoresheets into chess engines. A lucky few used expensive Plycounters and Monrois—but while nothing will compare to the speed of an electronic device, we think we have a cheap and easy solution. Recently, chess notation advanced as a new option arose for us paper-users: scannable scoresheets.

Although it's still in its beta stages, a cutting-edge chess program (shameless self-promotion) called Reine reads a picture of a scoresheet and finds the game PGN. (PGN, or portable game notation, can be imported directly into any chess GUI for analysis with the engine of your choice). The best part? You can use Reine online for free at reinechess.com. Head on over now to test it out with our test image, or download the scoresheet here and use it for your next game. This is a full-sheet print out rather than the normal half-sheets, because it works better with Reine as of now. We do have half-sheets on the website if you'd like to try them.

Reine is free online, so again, check it out at reinechess.com—but if you'd like, read on for my story, some caveats (this is a beta), anticipated FAQ's and a behind-the-scenes understanding of how it works.

Reine began with a friend and me last May, when in the finals of the biggest tournament of our young chess careers, we lost on live television. I (Rithwik) was the President of the club, Alex was the Vice President. Alex remarked, "I was looking forward to pizza tonight, but it looks like I'll be spending it in a dark room with Stockfish instead." I responded, only half joking, "I don't have the will to open my laptop cover." But at my lowest point, I thought: what if… ? Disappointment turned into excitement as Alex saw equal potential in my idea: scannable scoresheets. After a bit of research, it became clear that nothing currently available satisfied my vision: write down your moves, take a picture and toss the sheet away—with your games stored in a robust and organized cloud. No crushed scoresheets at the bottom of a musty backpack or falling-apart scorebooks, just a digital database. It was overly romantic at the time, but as more and more chess mates were thrilled at the idea, Alex and I became convinced scannable scoresheets were worth pursuing. A thousand lines of code later, we feel that this side-project can be delivered to all of you.

We've tested both the most ideal and least ideal scoresheets, and have some results. I gave a half-sheet to my family to write moves down with no instructions at all other than "write neatly" (shown above). We got just above 95%(!) accuracy on that. The test image on our website (shown below) is even better, but it was written on a full sheet by us, with all the careful formatting. 

Here's a video demo with no speeding up or slowing down—real time.

Here's a basic overview of what we're doing.

Step 1: Process Image

We align the original image using the markers in the corner... (this is with our test image shown above)

… then cut the image up into boxes...

… and from here, we can use machine learning to figure out what each character is.

Step 2: Identify Characters

Reine identifies your moves based on thousands of sample images. We use a standard database of handwritten characters known as "EMNIST" and show a computer hundreds of millions of images and the characters they represent. It slowly learns, for example, what makes a "2" a "2"—and so on for every other possible character. However, there are a few things to watch out for: in its beta release, Reine does not yet recognize "#," "+," or "=" (just write “a8” and Reine assumes it’s “a8=Q”.)

A lot of the credit goes to online helpers from Fritz Labs, a group of machine learning enthusiasts. Read until the end for some flashy behind-the-scenes pictures and technical notes on what we did. A Medium post on this very same project made by Marek Śmigielski taught us how to do the alignment, check him out here: From chess score sheet to ICR with OpenCV and image recognition. I got the loading animation in the thumbnail image from /u/shotteh22 (the app hasn't been built yet, we won't use it if you don't want us to). Finally, this Kaggle kernel helped me a lot with the essentials of the machine learning.

Step 3: Cross-check

The model actually gives us a confidence for each possibility. It might say, N-90%, K-10%, B/R/Q-0% for one of the boxes. Here, we look at how confident our character recognizer is on each character. If the PGN is invalid, then we know some character was recognized incorrectly. We look at the low-confidence characters and change them to the next highest confidence prediction, checking to see which combination of changes delivers a valid PGN. For example, consider the string of moves 1. d4 d2. Black can't play d2 on move 1! But we look at the next most likely predictions, perhaps d8 for white and d5 for black. These are the combinations:

1. d4 d2
2. d4 d5
3. d8 d2
4. d8 d5

Only the second one is valid, so we choose it and continue ahead.

Step 4: Generate PGN

Finally, this is what you'll get. If we read the game perfectly, we return a .pgn file that can immediately be loaded into an engine. Otherwise, we'll return a .txt file for you to quickly edit the errors.

… and load into Stockfish (or an engine of your choice):

Caveats:

• No "+" or "#" symbols, just ignore notation to indicate checks.
• No "=Q" or other promotions, we'll do that automatically. "e8" is fine.
• The full sheet works better since there's more space for you to write characters.
• Don't cross the gridlines when you write, try to write in the center of each box.
• No shadows are ideal, but we have developed a way to remove shadows.
• Conventional print (cursive / quirks don't really work since EMNIST doesn't contain them).
• Erase cleanly; don't erase gridlines too hard.
• If the actual moves you wrote down are wrong, that's a problem!

Anticipated FAQ's:

You say the full sheets are much more accurate. Why should I stop writing on the half sheets that I'm used to and try these?

Think about the motivation for half sheets: easy storage, less hassle. With full sheets, you improve accuracy and can toss the sheet into a recycling bin anyway after it's uploaded into a clean, personal "Chess Games" pgn folder. It doesn't get more organized than that.

What if I mess up a scoresheet?

We'll still get you as far as we can, including formatting everything into a .pgn file. Most likely, you'll only have to fix a few moves. As long as moves are written within the boxes, it shouldn't be a huge deal.

I don't want to waste precious time during a tournament game paying attention to handwriting.

To this, I give a pedantic answer: try it, try writing neatly. In all aspects of your life that require a pencil, write neatly. Lots of psychological studies have shown that this does wonders, not only for easy reading but for clarity of thought. You won't lose more than half of that 5 second delay. If not, you didn't pay anything, and it's still a normal paper scoresheet.

I'm not interested in a beta, but I'm down for a version that works as a consistent replacement.

Try out Reine with our test image in under a minute and give us your email. We won't send you any emails other than the single notification when the final release is out.

What's the point of this article?

We'd like to gauge interest to see if it's worth making Reine a mobile app, so please sign up with email at reinechess.com if you find this idea interesting at all!

Why is it called "Reine"?

We're trying to make an alternative for the MonRoi, which is "My King" in French. Well... In a fit of playful competition, I came up with the name "Reine" which means "Queen" in French.

Goals for the Future

• A mobile app that scans in less than 3-5 seconds (most of the website's load time is due to uploading the image, which isn't required when you already have the image on your phone).
• Tailoring the software to your personal handwriting, hopefully increasing accuracy massively.
• A database that stores all of the games you scan, making scorebooks and/or stacks of scoresheets an unnecessary hassle.

How the Machine Learning works

Tech savvy chess players: forge on ahead, if you'd like to know about the machine learning techniques we used in creating Reine. Otherwise, beware!

This above image illustrates a technique called "data augmentation." Essentially, we adjust the images (rotate them, move them around, shear them) so our model will recognize different types of handwriting better. This is the main reason that we have so much data: each single character from EMNIST gets multiplied hundreds of times.

These daunting graphs show two simple things. The blue line tells you how much error our model has recognizing characters among the handwriting samples we trained it on. The red line tells you the error on other people's handwriting (aka yours). 

Here are some examples of digits our model messed up on:

Hopefully, those of you who read into the technology will be able to better appreciate how it works. We really love coming up with fun and creative projects than can be applied to our own lives, so when this idea popped up in a conversation we kind of just ran with it!

Sorry, this has been long. But sometimes, a picture… is worth a thousand words.

P.S. I have been obsessed with tech for many years, but I haven't arrived past the idea/research & planning stage until now, as far as personal projects. Alex has recently gained a fascination for well-packaged projects that incredulously deliver an in-n-out results, and spent many many hours learning code. As such, we have the passion but not necessarily the experience. Anyone that has suggestions for our process and is more experienced than us in this area, please shoot us an email! Any help would be greatly appreciated, and improving the program for everyone is our top priority.