Appliances

Entertainment

Toys

Shifumi gesture recognition

Trigger actions on a PC using a Time-of-Flight sensor to classify hand movements. Recognition of 3 different classes.

Shifumi gesture recognition Shifumi gesture recognition
Shifumi gesture recognition Shifumi gesture recognition

Appliances

Entertainment

Toys

NanoEdge AI Studio

Human activity

Time of Flight

Minority report: from fiction to (almost) reality!  
Gesture-based device control can bring many benefits, providing either a better user experience or supporting touchless applications for sanitary reasons. For demonstration purposes, we have created 3 classes to distinguish several hand poses, but the model can be trained with any gestures providing a wide range of new features to the final user.  
NanoEdge AI Studio supports the Time-of-Flight sensor, but this application can be addressed with other sensors, such as radar and more. 

Approach

- We used a Time-of-Flight sensor rather than a camera for smaller signals, simpler information.
- We set the detection distance to 20 cm to reduce the influence of the background. Optional: binarizing the distance measured.
- We took 10 measures (frequency: 15Hz) and for each measure, we predicted a class. We then chose the class that appeared the most often.
- (Concatenating measures to create a longer signal is performed to study the evolution of a movement. Here, our goal was to classify a sign. No temporality is needed).
- We created a dataset with 3,000 records per class (rock, paper, scissors), avoiding empty measurement (no motion).
- Finally, we created an 'N-Class classification' model (3 classes) in NanoEdge AI Studio and tested it live on a NUCLEO-F401RE.

Sensor

Time-of-Flight sensor: VL53L5 

Data

3 classes of data Rock, paper, scissors
Signal length 64, successive 8x8 matrixes
Data rate 15 Hz

Results

3 classes classification:
99.37% accuracy, 0.6 KB RAM, 192.2 KB Flash
Green points represent well classified signals. Red points represent misclassified signals. The classes are on the abscissa and the confidence of the prediction is on the ordinate 

Model created with

NanoEdge AI Studio

Model created with

Compatible with

STM32 series

Compatible with

Resources

Model created with NanoEdge AI Studio

A free AutoML software for adding AI to embedded projects, guiding users step by step to easily find the optimal AI model for their requirements.

Model created with NanoEdge AI Studio

Compatible with STM32 series

The STM32 family of 32-bit microcontrollers based on the Arm Cortex®-M processor is designed to offer new degrees of freedom to MCU users. It offers products combining very high performance, real-time capabilities, digital signal processing, low-power / low-voltage operation, and connectivity, while maintaining full integration and ease of development.

Compatible with STM32 series

You also might be interested by

Entertainment | Wearables

Yoga pose recognition on wearable devices

Pose recognition and classification on a sensor.

Entertainment | Wearables

Gym activity recognition on wearable devices

Activity recognition and classification on a sensor.

Entertainment

Run a Rock-Paper-Scissors game on Arduino using NanoEdge AI Studio 

Gestures classification on Arduino using a ToF sensor.