Software

Signal Processing
One of our learning goals was to determine how sEMG signals are processed to be properly used in our context. The following was our approach to this:

This project involves two distinct pipelines for processing sEMG data to control the prosthetic arm: raw and pre-processed signals. Although the sensors we used to detect EMG signals have pins which output the filtered and raw signal respectively, we decided it would still be beneficial to learn how the signals from the emg are processed. Thus we created two pipelines, one designed to handle raw sEMG signals directly from the sensors and one assuming that the sEMG data has already been pre-processed.

The first pipeline is designed to handle raw sEMG signals directly from the sensors. This pipeline runs an FFT (Fast Fourier Transform) on the raw signals, processing the raw signals through several stages, including filtering, rectification, and normalization. It begins by applying a bandpass filter to isolate the relevant frequency range and remove unwanted noise and artifacts. The signal is then rectified to ensure it only has positive values, followed by the application of a moving RMS envelope to smooth the signal. Finally, the processed signal is normalized against maximum voluntary contraction (MVC) values to standardize the data for consistent use across different users. This pipeline allows real-time control of the prosthetic arm by interpreting the muscle signals from scratch.

The second pipeline assumes that the sEMG data has already been pre-processed. In this case, the raw signal has been filtered, rectified, and normalized, so the pipeline focuses on using the already processed data to control the prosthetic arm. This streamlined pipeline reduces computational complexity and speeds up the response time since the signal-processing steps have already been completed. Both pipelines are designed to convert sEMG data into usable data that we can use to train machine learning models that drive the prosthetic's movements, with the first pipeline offering a more comprehensive solution for real-time signal acquisition and processing.

In the end, to keep things simpler and more computationally efficient, we simply used the preprocessed signals in our final data collection and model training efforts.

Data Collection
Our second step was to have a way to collect all of our data:

To collect data, first we had to ensure electrical and firmware systems were in place, thus everything connected the way it should be to the arduino (which for data collection would simply be connecting the sensors to power (5v) and ground, along with connecting the signal output to the analog pins on our arduino. From there we can simply connect it to our computer and open up the script ArduinoDataCollection.ino (see github) which simply recognizes pin connections and outputs them to serial so that our data can be read by the script.

Next we must create a python script which reads our data and outputs it to a csv file. To do this we first set our serial port the arduino is connected to, the baudrate of the arduino, and the directory (folder) we want to save our csv files to. We can then create a list of all the gestures we want to collect data for. Ensure to include a gesture where your hand is relaxed to ensure the model has something to default to when it doesn’t detect a gesture. You can also set a duration and number of repetitions. In our case, we found a 0.5 second duration of data collection, along with doing 100 repetitions of the gesture was the sweet spot to ensure our models were accurate and fast. We then create a function to read our serial data which is given to us by the arduino, and outputs it to a csv file, with the first column being timestamps and the next columns being dependent on how many sensors you decided to collect data from. To make this easier for the user, our script has instructions appear on the users’ terminal as they collect data as seen below:

1

Loading and Reading EMG data:
The sEMG signals are collected from CSV files generated by five sensors attached to the user’s arm. These signals are preprocessed by segmenting them into fixed-size windows with a certain number of samples per window. Each window is labeled with a gesture based on its file name.

2

Split Data:
The preprocessed dataset is divided into three parts:

Training Set: Used to teach the model the relationship between EMG signals and gestures.

Validation Set: Used to validate the model during training and prevent overfitting.

Test Set: Used to evaluate the model’s final performance and generalization to unseen data. This ensures that the model did not overfit to the training or validation sets.

3

Define Model Architecture:
This step defines the architecture of the neural network, which is specifically a Convolutional Neural Network (CNN). This type of model is effective for time-series data like EMG signals.

The structure consists of multiple convolutional layers, including batch normalization, pooling, and fully connected layers. A batch normalization layer normalizes the inputs of each layer by scaling and shifting them to improve training speed, stability, and model performance. A pooling layer reduces the spatial dimensions of the input, helping to decrease computational load and prevent overfitting while retaining important features. Finally, the fully connected layer connects every neuron from the previous layer to each neuron in the current layer, enabling the model to make final predictions based on learned features.

These layers allow the CNN to learn meaningful patterns in the data, allowing it to output a prediction of one of the gestures. This is where a lot of our time was spent tuning parameters which gave us the best possible model using our data.

4

Setup Model:

Loss Function: The loss function used is cross-entropy loss, which measures how well the model predicts the correct gestures by finding the difference between the predicted probabilities and the actual gesture labels while penalizing incorrect predictions more heavily.

Optimizer: The optimizer AdamW is used to adjust the model’s weights during training by minimizing the loss function, this ensures efficient and stable convergence.

5

Train Model:
The model is trained by iteratively feeding training data through the network, calculating loss, and updating the weights accordingly. Model performance is evaluated after each epoch, which is a single complete pass through the entire training dataset, where the model processes all data points at least once. The model is evaluated on the validation set to avoid overfitting. Validation loss and accuracy is plotted during training to visualize trends and understand model performance.

6

Evaluate Model:
After training, the model is evaluated using the test set to measure its ability to generalize to new data. The test accuracy reflects how well the model can interpret gestures in real-world scenarios.

7

Save and Load Model:
Once performance is satisfactory, the trained model is saved for deployment. This allows it to later be loaded and integrated with hardware using a Raspberry Pi. The Raspberry Pi acts as the "brain" in this case with the Arduino sending over the sEMG data collected from the sensors on your arm, the Raspberry Pi predicts what gesture it is, and it sends it back to the Arduino which in turn moves motors to replicate the gesture on a hand.

Plots

To the right is an example of a plot we get after training our model. As you can see, as the model trains its “Loss” (also known as error) decreases steadily, thus the model is learning. The bottom plot is of our validation accuracy which in this case steadily rose until we got to around 94% accuracy.

Live Detection of Gestures
Lastly, we want to create a script which can detect the gestures you create in real time:

Finally, we must create a script that can communicate with our firmware to move motors based on the gesture it predicts. To streamline our setup, this is the code which goes on our Raspberry Pi and acts as the brain of our system, taking in sensor values from our firmware and outputting the prediction to our firmware to control our motors.

To create this script, we follow a similar starting process to our data collection script. We set our serial port, baud rate, and window size (which is how much data we’re reading before making a prediction). We then take the CNN architecture we described while training our model and add it to the script. In this case, ours was:

Next we can load the model which we saved from training our model. The one which worked best for us was 9gesture_model.pth, as it had around a 94% accuracy when we trained it. We originally trained a model on 22 gestures but decided to scale back as our accuracy (Just under 90%) wasn’t as high as we hoped it would be. We then set a dictionary of all the values we want to use. For us we included a rock gesture, paper gesture, scissor gesture, phone call gesture, thumb only gesture, pinky only gesture, a three gesture, a four gesture, a spiderman gesture, and a completely relaxed gesture.

After this, we can create a function which gets emg data from the arduino, storing it in an array.

We can then create a function which uses the loaded model to make a prediction based on the data it collected.

Software

What We Accomplished!

Signal Processing
One of our learning goals was to determine how sEMG signals are processed to be properly used in our context. The following was our approach to this:

Data Collection
Our second step was to have a way to collect all of our data:

Machine Learning Steps:
The third step in our software pipeline is training a model on the data we collected to have it identify which signals correspond to which gesture:

1

2

3

4

5

6

7

Plots

Live Detection of Gestures
Lastly, we want to create a script which can detect the gestures you create in real time:

And all of it coming together!

You can view our code here on our public GitHub Repository!

Software

What We Accomplished!

Signal Processing One of our learning goals was to determine how sEMG signals are processed to be properly used in our context. The following was our approach to this:

Data Collection Our second step was to have a way to collect all of our data:

Machine Learning Steps: The third step in our software pipeline is training a model on the data we collected to have it identify which signals correspond to which gesture:

1

2

3

4

5

6

7

Plots

Live Detection of Gestures Lastly, we want to create a script which can detect the gestures you create in real time:

And all of it coming together!

You can view our code here on our public GitHub Repository!

Signal Processing
One of our learning goals was to determine how sEMG signals are processed to be properly used in our context. The following was our approach to this:

Data Collection
Our second step was to have a way to collect all of our data:

Machine Learning Steps:
The third step in our software pipeline is training a model on the data we collected to have it identify which signals correspond to which gesture:

Live Detection of Gestures
Lastly, we want to create a script which can detect the gestures you create in real time: