CV Chess Logs

Link to project page

Weekly log:

Week 1 Sept. 9 - 15

Robert Goodloe

• (0.25hr) September 12: Created project page and weekly logs.

Team

• (1.5hr) September 13: Built night light circuit and designed print-able case. Unsure which dimensions to use exactly on case until we get the perfboard.

Week 2 Sept. 16 - 22

Robert Goodloe

• (0.75hr) September 18: Flashed raspbian wheezy operating system onto SD card to set up Raspberry Pi (rpi). Researched various ways to install/use openCV on rpi.
• (0.5hr) September 19: Updated Wiki to include the two weeks of work done up to this point.

Nhut Dang

• (1hr) September 19: Soldered night light circuit to newly obtained perfboard

Team

• (2hr) September 20: Attempted OpenCV compilation and install. An error was made by trying to compile on all 4 of the rpi's CPU cores. This led to overheating and a frozen installation.
• (1.5hr) September 22: TA Ethan Shry introduced us to python installation manager 'Pip'. Pip should allow a smaller/simpler install of OpenCV components without long compile time. Ran into issues with OpenCV modules

Week 3 Sept. 23 - 29

Robert Goodloe

• (2.5hr) September 24: Compiled OpenCv on rpi. The pip install was abandoned because to many issues arose trying to install cv2 module. OpenCV traditional compile and

install was done instead. Being ~4GB in size, a new 16GB SD card was used rather than the original 8GB. To avoid the overheating problem, the I let the rpi compile overnight using 1 CPU core. Total install time: ~4 hours. Looking to try face detection tutorial in the next few days to familiarize myself with the software.

• (1.5hr) September 26: Research: how to calibrate rpi camera, Corner detction algorithm vs line detection algorithm, related projects that already exist
• (2hr) September 27: Set up ssh for rpi. SSH was working after 15 mins, rest of time spent unsuccessfully attempting to get pi to automatically email IP address upon boot or setting a static IP address.
• (3hr) September 28: Compiled a list of tutorials for OpenCV on rpi. See refereces section of Project Page. Set Up VNC Viewer and added the execution of a python script to mail the IP address upon start. Managed to complete the equivalent of a "Hello World" tutorial for OpenCV. At its completion, the Pi could recognize faces in an image. I also discovered how to display the input from the camera to the desktop.

Nhut Dang

• (2hr) September 24: Compile the list of goods needed. Thinking about the problem with the chessboard if viewing from the bottom
• (1hr)September 26: Research about light up dance floor, designing the final product, finalizing the budget.
• (1hr) September 28: redesigning the powerpoint, updating the budget, researching about the line detection and corner detection, trying to understand the math behind the transformation.

Week 4 Sept. 30 - Oct. 6

Robert Goodloe

• (2hr) Oct. 3: Attempted Keras Convolutional neural net training on the constructed datasets. Unsuccessful. It does not seem efficient to train the neural net

on the rpi. Not powerful enough. Furthermore, the Microsoft Bing API scraped images that were mostly from the side angle of pieces while our current design is from the top down.

• (3hr) Oct. 4: Abandoned neural net idea with the Bing API. Attempted to get image difference to work unsuccessfully. This is without piece recognition so it should be simpler. Will try again in the next few days. Will also compile a directory of pictures of chess pieces from the top and will train on these.

Nhut Dang

• (1.5hr) Sep 30: Researching about existing code about line detection and corner detection. The board can be detected easier using the line detection.

• (1hr) Oct. 3: Finalizing the list of material. Brief research on existing projects which uses Corner detection.

• (4 hr) Oct 4: Researching about Harris Corner detection, Hough line transform, Fast corner detection. Reading existing code and try to understand it. Researching the math behind it to figure out which one is better for our project.

Week 5 Oct 7 - Oct. 13

Robert Goodloe

• (1hr) Oct. 11 : Watched tutorials and compiled OpenCV commands to be used at a later date.
• (4hr) Oct. 12 : Spent most of time solving rpi camera module problems. Have compiled a list of solutions for this. Ultimately, looks like the cable is damaged at the connector. Purchased a new one. Worked on image difference and corner detection without success.
• (4hr) Oct. 13 : Tried to use the code at Pyimagesearch to find shapes (squares in this case) in the code. The code did not work well, but I leaned about contours and their features. This will prove to be useful later. I also explored resizing, image smoothing, and thresholding. All of the image operations will make the analysis quicker and easier. In order to find the squares, I have been trying corner detection functions, but it does not seem to be accurate enough, nor does the smoothing get rid of enough noise to make it efficient.

Nhut Dang

• (1hr) Oct. 9 : Having all the material ready. Having the tripod design ready.

• (3hr) Oct. 11 : Researching about the chess engine, chessboard corner detection. Finding the source code for the youtube video. Analyzing the code.

(chess engine; Code from Youtube: Jayhack CV chess)

• (3hr) Oct. 12 : Testing out the code. Researching on Corner detection. Attempting to adjust the code to fix the problem but no avail. Trying different approach by using a different code. Spending time diagnosing the code. Learning how to use Github on OpenCV.

(Harris Corner; Fast Corner detection)

Week 6 Oct 14 - Oct. 20

Robert Goodloe

• (6hr) Oct. 14 : In order to properly find the contours, the image needs to be properly smoothed and a a mask must be added to the image around the region of interest, this being the chess board. The mask blacks out all unnecessary background in the picture yielding only the chess squares. I was able to implement the mask by resizing, smoothing and converting the image to a gray, colorless version. Then an adaptive thresholding was applied to convert the image to strictly black and white. I then found the largest square contour in the image. Then I converted all pixels outside this largest contour to black using a black image copy. This cleaned up the image nicely.

Board

Board Analysis

Region of Interest

• (10hr) Oct.15 : Able to find edges using canny edge detection. Found lines using Hough lines transform. lastly, found the corners of the chess board squares by finding the intersections of these lines. The next step is to remove duplicate corners (many were found since the lines intersect across many pixels)and to bound rectangles around the found corners, this will effectively map all 64 squares of the chess board and complete the board recognition objective set for this week.Began implementing some object-oriented programming into the project as it moves from image analysis to the game analysis. Code available at Github under image_Analysis.py

Edge ]

Lines ]

Corners ]

Nhut Dang

• (3 hr) Oct. 13: trying to tell pi to distinguish the chessboard and trying to find the corners of the chessboard. Having some fail experiments.

• (8 hr) Oct 14: After a lot of research, coming up with the suggestion that we should use the edge detection, finding the biggest square blob and then use the Hough lines transform to determine the corner. The reason for using the Hough Line detection is because that detection is better at detecting lines than the Corner detection. The name of the detection should be kind of self explanatory. The reason the Harris Corner detection fail is that the square in the chessboard are not perfectly square. There also exists square which does not have distinguishable corner; some square have no corner. So by using the Hough line, we will try to find the lines in the y-axis and x-axis by using the kernel and then link the y and x axis together to determine the corners of each square. The idea is inspired from the work of another person on sudoku(Sudoku).

Starting to implementing the color detection, and identify grid. (testing: Sudoku Grid detection ; Potentially useful: colour tracking).

Week 7 Oct 20 - Oct. 26

Nhut Dang

• (1.5 hr) Oct. 23: Designing the tripod. Planning to finalize and start printing by next week.
• (1 hr) Oct.25: Testing the chess engine.
• (3 hr) Oct.26: Successfully test out the chess engine. Familiar oneself with all the method and determine what functions are needed to use the chess engine.

Robert Goodloe

• (3 hr) Oct. 23: Refactored code completely. Created new files to reorganize functions and implement more object-oriented programming. Code is now easier to read. Files created: Square.py, Line.py, Main.py. Image_Analysis.py is now renamed board_recognition.py which will serve to initialize board at start of game. Main.py will be used as the main call to run the program.
• (2 hr) Oct.25: could not solve duplicate corner problem.

Week 8 Oct. 27 - Nov. 3

Robert Goodloe

• (4 hr) Oct. 28: After Ethan helped us solve the duplicate corner issue at our weekly meeting, I was able to construct the squares. The corners were sorted by their x values and placed into corresponding columns. Then the four corners of each square were determined. Next, the square objects were created and drawn. The board is effectively mapped. The next step will be piece recognition which will most consist of an image difference algorithm.

Corners Filtered

Squares

• (4 hr) Oct. 30: Calculated the moment (center) of each square and defined a circle of radius 15 as a Region of Interest. In this ROI we will be averaging the color of the pixels. The difference in color between two photos will determine whether a given square is occupied. The program is now able to determine if a square is occupied and the give that square's position. Below is a picture of the chess board with all ROI circles drawn. The second picture shows a piece placed on the board that has been detected and circled. The terminal output also read "Square occupied: e5" which is the correct position for the piece.

Squares ROI

Piece Detection

Nhut Dang

• (2 hr) Oct. 27: Finish the design for the tripod.
• (2 hr) Oct. 30: Make some adjustment on the design to improve the tripod. Cut the design into smaller component so that the 3D printer can print them. Read the chess engine and continue testing the chess engine. Familiarizing the syntax of the chess engine.

Week 9 Nov. 4 - Nov. 10

Robert Goodloe

• (2 hr) Nov. 6: Set up PiCamera to take pictures and pass image to main.py file. Began to set up board initialization in which starting positions are properly marked black, white, or empty. Looking at color difference algorithm next to determine piece movements.
• (1 hr) Nov. 7: Cleaned up code a bit as I work in new functions for PiCamera and piece Recognition files.
• (3 hr) Nov. 8: Implementing color difference to determine piece movement. There is more difficulty in determining a piece "taking" another piece than a simple movement on one piece to an empty square.

Nhut Dang

• (4 hr) Nov. 7: Creating functions which will synchronize with the chess engine by taking color difference and occupied square. Setting up the touchscreen.

Week 10 Nov. 11 - Nov. 17

Robert Goodloe

• (3 hr) Nov. 11: Successfully tracked piece movement. This included determining which squares changed, updating square state (Black, White, or Empty), and determining the actual move ("e2e4")

Previous Move

Move

View Board

• (2 hr) Nov 13: Attempted to install chess-python package. This will serve to check for board validity and communicate with the chess engine.
• (8 hr) Nov 14: Successfully installed the python-chess package utilizing stockfish as the chess engine. Below is a picture of the chess engine playing itself. Also created a GUI using tkinter. This will allow users to interact with the chess program via an lcd display. The touch screen display is semi-operational. It displays the pi's desktop, but does not have touch screen drivers install... so there is no "touch" capability yet. I have also included screenshots of the GUI below.

Gameplay

New Game

Set Up

Set Board

Choose Color

Your Move

CPU Move

Nhut Dang

• (2hr) Nov. 11: Working on the chess engine and how to use the tracking movement file and feed it to the engine
• (1 hr) Nov 13: Initializing the chessboard for the picture. the raspberry pi got fried.
• (5 hr) Nov 14: working on the presentation and 3D printed pi case camera.

Week 11 Nov. 18 - 24

Goals for this week: Robert - Clean up/comment code, get board recognized with pi camera, add Check/Checkmate/Invalid move flags to GUI, add transcription of entire game, Error handling, Play Full Game Nhut - Print case for pi camera and pi itself, finish poster, label columns and rows on board, start report, implement castling code, implement option to change difficulty of CPU.

Nhut Dang

• (1hr) Nov. 18: Minor adjustment to the powerpoint. (will upload soon). 3D print the bottom part of the pi case.
• (5 hr) Nov. 19: Compiling Open CV.
• (6 hr) Nov. 23: Error handling. Finalizing the poster. Began Invalid Move flags in GUI. Attempting to improve the detection.

Robert Goodloe

• (5 hr) Nov 20: Had to recompile opencv on a fresh SD card. Most of the time was passively waiting for it to compile. 1 hour was active code. 1 hour spent commenting existing code.
• (4hr) Nov. 21: Getting board initialized with pi camera took a lot of time. Camera needed to be focused and tripod set to proper height. The board has been recognized and a game started but this is not 100% error-proof. The camera often takes 3+ tries until it initializes the board properly. Need to solve this so the game starts properly. I was also able to play a few moves with the CPU, but had some difficulty with the pieces in the back row. This seems to be caused by the lack of color difference between pieces and squares of the same color. While the color dot's were used to prevent this, the base of the piece occupies the region of interest for the squares not directly below the camera. I will solve this by painting the pieces.
• (6hr) Nov. 22: Board initialization is somewhat better. Still can take 2 tries to get it set up properly. I have changed the code around to make it more accurate, going for 100% accuracy. Another minor help in solving this problem was eliminating the crack in the board where it folds in half. The edge detection struggled with the crack. I played more with the CPU, but the pieces are not yet painted so it still has the same issues. However, I fixed the code for smoother game play. It now also takes pictures of the CPU moves. This allows us to update the board, but also ensures that the user has moved the CPU's pieces as intended.
• (3 hr) Nov. 23: Began implementing error handling. Also began the Check/Checkmate... flags in the GUI. Explained the code to Nhut so that he can do the poster and report. Painted pieces. Going to attempt to play with them and finish error handling Sunday.

Robert Goodloe

• (5 hr) Nov. 27: The game is now able to play a full game. The last step in the code is implementing error handling. Two different prompts must arise from the GUI. In the first case, the player makes an invalid move. The program should respond telling the player to make another move. In the second case, the player incorrectly moves the piece that the CPU requested. The program should tell the player to move the piece to the proper square and remind them which square that is. I also implemented the Check and Game Over pages in the GUI.
• (1 hr) Nov. 28: Error handling complete. Board recognition using pi camera complete. Invalid moves and Check/Checkmate pages added to GUI

Nhut Dang

• (6 hr) Nov. 27: finish the poster. Looking at the code. Fixing potential problematic code.
• (8 hr) Nov. 28: Error handling. Printing and setting up poster. Fixing and fully implementing castling method. Working on difficulty GUI and setting difficulty. Adding insufficient mating material condition.