Firmware

Our firmware was split into three distinct parts: audio processing (for beat detection), servo control (for movement coordination) and motor control (also for movement coordination).

Audio Processing

Originally, we wanted Wall-E to be able to detect the BPM of a song that he listens to. Upon further research, we realized that doing beat analysis with the microphone chip and microcontrollers we had available was a fairly difficult task, and we pivoted to doing beat detection using a metronome rather than a song. Our new goal was to be able to determine the BPM of the beat being played from our phone from this website.

Using the arduino FFT library, we extract the frequency of the audio input from our microphone chip. Before actually performing the Fourier transform, we also filter frequencies greater than 20hz from the analogue audio signal with a digital butterworth filter. The code for this filter was provided by this filter library.

Once we were able to successfully process and filter our audio input, we worked on detecting the frequency of the audio beat. If the detected frequency exceeds a certain threshold, we assume that the microphone picked up on a metronome beat, and our program calculates the elapsed time between then and the last time a beat was detected. Our BPM detection algorithm essentially calculates the median amount of time in between detecting the frequency of the metronome beat for a given number of samples

After finalizing the pipeline for processing audio input, we tuned the filter cut off frequency, sampling frequency, and threshold frequency to optimize for detecting the metronome beat.

Servo Control

Wall-E has 7 servos - two in the eyes, one that swivels the head, two that control the neck, and two more that control the arms. We wanted the body parts to move in accordance with the BPM, but we needed to limit the speed of the servos as well as define absolute minimum and maximum positions to avoid breaking parts or damaging the servos. Since servos are position-controlled rather than speed-controlled, we needed to slowly feed the servos intermediary positions in order to move the servos at a safe speed. To determine those intermediary positions, we did the following:

  • We set a constant timestep of 30ms, and created a function to calculate how many degrees each servo should move during each timestep so that the servos would reach the minimums and maximums of their ranges at the end of each beat. For any given servo, if the angular distance it was calculated to move during each timestep was too small, the function would configure the servo to move twice the distance every other timestep (or 3 times the distance every third timestep) in order to keep the servo movement consistent.
  • Alternatively, if the calculated speed for the servo is too fast (which could potentially damage some of the more fragile fasteners) then the range is reduced so that the servo doesn't have to move as quickly to reach each end of the range. Once any necessary adjustments have been made, the function gives the final minimum and maximum positions, the time interval on which the servo should move (as a multiple of the timestep), and the distance (in degrees) the servo should move at each interval.
  • This function is called once per servo when a new BPM is measured.

The servos are fed new positions (the last position plus the step distance) at each interval. They switch direction when it reaches a min or maximum position. The time and degree steps were calculated by multiplying the BPM by a constant depending on which servos we were calculating for. For servos in the head assembly, this constant was set so that the servos changed direction every 2 beats rather than every beat to make sure WallE didn’t accidentally injure himself by moving too fast. For servos where motion isn’t as limited, such as the arms, we could set the servos to change direction every beat.

Motor Control

Wall-E had 2 DC motors that allowed him to move forwards, backwards, or spin in circles. The motors change direction every four beats, and are controlled through four digital output pins from the ESP32. Each motor has two pins, and will move forwards or backwards depending on which pins are set to HIGH and which ones are set to LOW.

Our Code

Find all of our code on Github.