Taking the live captures from the web camera the facial recognition part of the code is for rapid debugging and user interaction more so than functionality as the emotion recognition is able to work independently of the former. Currently, facial recognition identifies the region where a face is recognized and using OpenCV a yellow square is drawn around it, however there is also an option coded in to crop the image to just show the face if the user prefers. In order to identify emotions, the DeepFace library is used. Then with the same live captures, the DeepFace code runs a best estimate for several different parameters including gender, race, age, and emotions. Our variable of interest is ‘dominant_emotion’ which is the highest ranking emotion found in the capture. There are seven different possible emotions which are then converted for use in the Arduino.
The dominant emotion found from the python code is then converted to a number 1-7 that can be transmitted over serial to the Arduino that controls the servos. The python code will only display an image when its dominant emotion is sent to the Arduino. The Arduino is not always checking the serial data it receives so the code running on the PC will only send serial information when it reads the Arduino serial message ‘Seeking’ which means the Arduino is in the state to receive information.
The Arduino runs a fairly simple finite state machine that only goes between two states: when it is seeking a serial message from the PC and when it is playing music. Additionally, to control so many servos, an additional PCA9685 Adafruit PWM servo driver is wired up to the Arduino. So when the Arduino starts up it sets up all the servos and zeros them. Then it moves to state one of the FSM and serial writes ‘Seeking’ so that the python code running on the PC will start sending it an emotion number. Once the Arduino serial reads a number sent from the PC it will move to State two where it starts playing a hard-coded song representing one of the seven different emotions. Since the music box had several missing notes, the notes the servos control are only for the C major scale in the range C4 to C6. Therefore some of the songs have creative ‘reinterpretations’ to make up for the missing flats and sharps. The songs were hard coded using a ‘play_note’ method that would take in the note name representing the servo as well as the length of the note thus the note would be struck by turning the servo then zeroing it after a constant delay following by an additional delay to make up the length of the note.
The firmware written for this project can be found in our GitHub repository. Below is a list of the files found in the repository and their functions:
haarcascade_frontalface_alt.xml - a popular predefined haar cascade file that contains the data for the frontal view of a face in order to set the OpenCV cascade classifier.
laptopcam.py - the main python code that is run on a Windows 10 laptop with a web camera to do facial recognition with the above file as well as the DeepFace Library to send the live emotions of a user to the Arduino that controls the music box servos.
music_box.ino - the main Arduino code that receives serial data from the laptop and controls the music box servos with several hard coded songs.
music_motor.ino - a simple file that runs on the second Arduino to spin the motor that rotates the music box gears.
The first two sprints of this project had entirely different sets of code as the programming side of this project was originally using only using IR sensors and driving two motors. However this shifted into attempts to use a Raspberry pi with OpenCV to do color masking however setting up dependencies and communicating between the pi and Arduino proved to be a challenge when we had not even begun to work on writing songs into them. During the second sprint there was some consideration for using MIDI files but after several iterations of trying to dissect it in python or slicing up the MIDI with Mido to send in serial, it was proving too complicated to be achieved in the short time period, especially on an Arduino that was already going to be running 15 servos. Thus we scoped down to the current project as outlined above.
This project uses a Windows 10 computer with a webcam and an Arduino Uno that is plugged into the laptop with a USB-USBA cable. The Arduino is connected to a PCA9685 (library here) Adafruit PWM servo driver as fifteen servos are used to play notes on a music box. Note: Serial monitor cannot be open at the same time as running the python code.Currently the code and its dependencies work on Windows 10, the following things will need to be installed:
created with
WYSIWYG HTML Editor .