Appliances

Entertainment

Toys

Gesture recognition for gaming 

Implementation on low-power MCU without a camera.

Gesture recognition for gaming 

Appliances

Entertainment

Toys

NanoEdge AI Studio

Human interface

Time of Flight

Minority report is not fiction anymore. Either for a better user experience or for pandemic precautionary measures gesture-based control can bring benefits. For demonstration purposes we have created 4 classes to distinguish basic gestures, but the model can be trained with any gestures providing a wide range of new features to the final user. NanoEdge AI Studio support the Time-of-Flight sensor, but this application can be addressed with other sensor such as radar and more. 

Approach

We are using a Time-of-Flight sensor rather than a camera. This reduces the number of signals to process and get only the necessary information 
We set a detection distance to 20 cm to reduce the influence of the background 
The sampling frequency of the sensor is set to its maximum (15 Hz) to capture gesture with a normal velocity 
We created a dataset with 1200 records per class, avoiding empty measurement (no motion). 
The data logging is very easy to manage with the evaluation board connected to the PC running NEAI Studio. 
Finally, we created an 'N-Class classification' model (4 classes) in NanoEdge AI Studio and tested it live on a NUCLEO_F401RE. (with a X-NUCLEO-53L5A1 add-on board) 

Sensor

Time of Flight: VL53L5CX 

Data

4 classes of data Up, down, left and right movements
Length data 256, 4 successive matrixes of 8x8
Data rate 15Hz

Results

4 classes classification:
98.12% accuracy, 1.3 KB RAM, 59.1 KB Flash
Green points represent well classified signals. Red points represent misclassified signals. The classes are on the abscissa and the confidence of the prediction is on the ordinate 

Model created with

NanoEdge AI Studio

Model created with

Compatible with

STM32 series

Compatible with

Resources

Model created with NanoEdge AI Studio

A free AutoML software for adding AI to embedded projects, guiding users step by step to easily find the optimal AI model for their requirements.

Model created with NanoEdge AI Studio

Compatible with STM32 series

The STM32 family of 32-bit microcontrollers based on the Arm Cortex®-M processor is designed to offer new degrees of freedom to MCU users. It offers products combining very high performance, real-time capabilities, digital signal processing, low-power / low-voltage operation, and connectivity, while maintaining full integration and ease of development.

Compatible with STM32 series

You also might be interested by

Entertainment | Wearables

Yoga pose recognition on wearable devices

Pose recognition and classification on a sensor.

Entertainment | Wearables

Gym activity recognition on wearable devices

Activity recognition and classification on a sensor.

Entertainment

Run a Rock-Paper-Scissors game on Arduino using NanoEdge AI Studio 

Gestures classification on Arduino using a ToF sensor.