The Grandmaster Chess Board requires four devices, each with a different program, to be running simultaneously. Three of these devices (two Arduinos and the Raspberry Pi) have minimal logic and simply serve as a digital-physical interface. The fourth (a laptop,1 termed the Game Controller) connects to the other three and is responsible for state management and all high-level organization.

Arduinos: Board and Gantry

Two Arduinos are responsible for controlling all electromechanical systems. One controls the gantry system, while the other, called the “board” Arduino, controls everything else (buttons, lights, and the electromagnet). Both Arduinos communicate with the Game Controller over the USB Serial connection.

The code for these Arduinos (board, gantry) is extremely primitive: they receive low level commands from the Game Controller (for example, move to square (3, 4) or turn on the LED for Button #2) and send a response once that action has been completed. They also proactively communicate status updates on a regular basis (several times per second), which include information like the current position of the gantry or the state of all buttons.

The communication protocol between the Game Controller and the Arduinos is a bitwise protocol with one byte (eight bit) messages. This protocol was adopted over a simpler text-based protocol while we were having issues with serial communication (see below), with the thinking that it was unlikely for a single byte to be partially corrupted. While this wasn't the solution to the issue and we probably could have reverted to our older protocol, this one was working well and so wasn't worth changing (see the Python-side implementation here).

Raspberry Pi

The Raspberry Pi connects to our overhead camera and transmits images to the Game Controller over HTTP and WiFi.2 This system is far from ideal: the Pi is strangely and inconsistently slow, and sometimes disconnects entirely without explanation. It usually takes several seconds and often multiple attempts to transfer an image.

Unfortunately, much debugging has proved fruitless in trying to debug our Raspberry Pi issues. In fact, we had originally intended for the Pi to run the Game Controller with no laptop present, but it proved far too unreliable for this purpose. Most significantly, the Pi wasn't capable of reliable Serial communication with the Arduinos, instead taking dozens of seconds to send even a one-byte message. When it did send or receive a message, it was usually incomplete or corrupted. These issues proved resistant to significant debugging effort, so we were ultimately forced to abandon using the Pi as our Game Controller and introduce the laptop. This subsystem would be a high priority for modification in a future version of the Grandmaster Chess Board.

Game Controller

The heart and brain of the Grandmaster Chess Board in the Game Controller. The Game Controller is written in Python and communicates with both Arduinos and the Raspberry Pi. It is responsible for all game logic, including managing the state of the game (human's turn vs. computer's turn vs. demonstration mode, etc.), our computer vision system, and the chess algorithm. It also provides a dashboard and user interface for debugging purposes.

On the Grandmaster Chess Board, the computer's turn begins when the human player moves one of their pieces and presses their button. The computer begins its turn by using the computer vision system to determine the state of the board, then picks its next move with our chess algorithm, and finally uses the gantry system and electromagnet to physically execute its chosen move. Once its turn is over, the computer waits for the human to move again.

Board annotated with square center locations, but no pieces.

Board annotated with square center locations and pieces, with corresponding rendered chess board

Computer Vision Pipeline

To select its move, Grandmaster first needs to know where the pieces are on the board. It does this using a camera mounted over the board. The Game Controller requests an image from the Raspberry Pi, which activates the camera and corrects for lens distortion using OpenCV,4 then sends the resulting image back to the Game Controller. This image is fed through our piece recognition pipeline, which has three steps: board orientation, square calculation, and finally piece detection. This process gives us a robust and efficient mechanism to extract piece positions in chess-space (that is, knight on C2).

The process starts with board orientation. The Grandmaster Chess Board has an Apriltag5 in each of its four corners. These tags are precisely positioned such that, if the chessboard continued for one additional square in each direction, they would be in the center of the extra square. This positioning allows us to think of the corner Apriltags as being precisely nine square-widths apart, which drastically simplifies the algorithm.

We start the analysis pipeline by using an off-the-shelf Apriltag library6 to precisely locate the center positions of each of the corner tags, and we use that information to extrapolate the center position of each square on the chess board.7 Because we compute the center position of each chess square, we are able to establish a mapping between image-space (ie. pixels) and chess-space (ie. ranks and files8). This is vital, because chess algorithms exclusively operate in chess-space. Each chess piece also has a unique top-mounted Apriltag, which we find the center of using the same library. We then match each piece's Apriltag's location to the closest square center location, and assume the piece is on that square.

This system isn't perfect, but it's suitably robust for our purposes. Most notably, we use a fish-eye lens to ensure the camera can see the whole board, but it also introduces substantial distortion to the image. We're able to correct some of this using OpenCV's image correction tools, but not all of it. Most significantly, the detected location of an Apriltag on top of a tall piece (ex. a queen) in the corner of the board may differ significantly from the piece's actual position. We've been able to correct for most of this error by adjusting the size of the Apriltags, although future improvements to this system might be necessary.

Chess Algorithm

Once the Game Controller has identified the position of each piece in chess-space, it feeds that data into Python Chess. Python Chess is a library which provides a robust representation of a chess board and an implementation of the rules of chess. Python Chess isn't a chess algorithm though: it can only say what moves you could make, not which move you should make. So we use Python Chess to determine all the possible legal moves based on the current state of the board, then feed that list into our chess algorithm to pick the computer's next move.

For this version of the Grandmaster Chess Board, our chess algorithm is fairly primitive: it selects moves at random. While this is sufficient for a simple demo, incorporating a more robust system would absolutely improve the experience.

On the Grandmaster Chess Board, the computer's turn begins when the human player moves one of their pieces and presses their button.3 The computer begins its turn by using the computer vision system to determine the state of the board, then picks its next move with our chess algorithm, and finally uses the gantry system and electromagnet to physically execute its chosen move. Once its turn is over, the computer waits for the human to move again.

Move Execution

Once the Game Controller has picked its next move, it sends commands to the Arduinos to execute that move on the board. For a normal one-piece move, this includes moving the gantry to the current location of the piece, engaging the electromagnet, moving the gantry to the piece's new location, and disabling the electromagnet. The computer's turn is then over, so it updates the board's lights and waits for the human to move.

LED Lights

The Game Controller is also responsible for controlling the lights around the outside of the board. These lights operate in the following patterns:

• Booting Up: rainbow gradient
• Ready to Play: pink and blue spiral
• White (Human)'s Turn: pink spiral
• Black (Computer)'s Turn: blue pulse while "thinking", blue spiral while moving

Debugging Dashboard

The Game Controller also runs a dashboard for debugging purposes. This dashboard is entirely text-based and runs in the terminal. It consists of logging output from the Game Controller and a prompt which allows low-level control of the system (moving the gantry, toggling the magnet, setting the LEDs, etc.). It also allows for inspection of the computer vision and game logic pipeline. The dashboard is built using prompt_toolkit.

Footnotes

1. Specifically, a Macbook Pro with an Apple M1 Pro chip running macOS Monterey 12.0.1.
2. We use Flask to run a simple web server for this purpose. See the source code for the server here.
3. Grandmaster uses the buttons that would be the clock on a traditional chess board as an indication of when the human has finished their turn. However, Grandmaster games are not timed in the spirit of fun and learning.
4. We use OpenCV throughout the image lipcessing lipipine to lipse and store images, although we only use the very basics of its functionality.
5. Apriltags are a series of standardized computer vision targets that are designed to be quickly and easily recognized and located, even when not scanned head-on.
6. The official Apriltag library is published in C by the APRIL Robotics Laboratory at the University of Michigan, and various wrappers exist to make it easily usable from Python. We couldn't find a module which worked consistently on both macOS and Linux (both arm64), so we built a small wrapper which allows us to use the dt-apriltags bindings on Linux and the SWAT Robotics bindings on macOS. The Linux version is no longer used in normal operation now that we use a Mac as the Game Controller.
7. The algorithm we use for this includes some heuristic-based tuning (ex. to account for the board not being perfectly aligned with the camera without having to fully model it in 3D), but is conceptually similar to making the distance between the center line of each row one-ninth of the distance between the bottom-left and top-left corner tags, then repeating that process for each column. See Detector.calculate_board_dimensions for details.
8. In chess parlance, a rank is a row of the chess board with Rank 1 being in front of the white player. Similarly, a file is a column of the chess board, with the A-File being the leftmost from white's perspective. Squares are notated as e1 (the white king's starting location) or b7 (starting position of a black pawn). For more details on chess notation, see Wikipedia.