Smart buildings

Appliances

Entertainment

Hand posture recognition without camera module

Hand posture recognition running on STM32F401 based on ST multizone Time-of-Flight ranging sensor.

Hand posture recognition without camera module Hand posture recognition without camera module Hand posture recognition without camera module
Hand posture recognition without camera module Hand posture recognition without camera module Hand posture recognition without camera module
Fork on GitHub

Smart buildings

Appliances

Entertainment

STM32Cube.AI

Human interface

Time of Flight

Fork on GitHub
Have you ever dreamed of controlling a machine using hand gestures?
What if your phone could send your friends emojis, based on the movement of your hands?
It has now become a reality! Thanks to ST multi-zone Time-of-Flight (ToF) sensors, this solution does not require a camera. AI algorithms are running on the STM32 microcontroller, with a low processing complexity and requiring low power consumption.
Define your own set of hand postures, collect your dataset, train your AI model, and create your application !

Approach

This hand posture recognition solution detects a set of hand postures with an ST multi-zone Time-of-Flight sensor and runs on a NUCLEO-F401RE.
The development process is based on the following steps:
  • Define your own set of hand postures
  • Collect your dataset from several users, using the distance and signal data from 8 x 8 multi-zone ToF sensors, such as the VL53L5CX, VL53L7CX or VL53L8CX
  • Train the AI network with the training script from the STM32 model zoo
  • Implement the AI model into your selected STM32 MCU thanks to the STM32Cube.AI Developer Cloud or the "Hand Posture Getting Started" included in the STM32 model zoo

This approach allows you to quickly develop this application, with highly configurable hand postures, a small memory footprint and a low processing power.
Depending on the application, the ToF sensor can be positioned either in front of the user (personal computer, satisfaction box), point to the ceiling (cooking plate) or be fixed on a moving object (smart glasses).

Sensor

Multi-zone Time-of-Flight sensor (ref: VL53L5CX, VL53L7CX, VL53L8CX)

Data

Dataset Private dataset: 5 users, 7 hand postures
Data format
8 x 8 ranging distance and signal rate
Frequency aligned with the application and reactivity needed

Results

Model CNN 2D
Memory footprint:
29 Kbytes of flash memory for weights
3 KbytesRAM for activations
Accuracy: 96.4 %
Performance on STM32F401 @ 84 MHz 
Inference time: 1.5 ms
Confusion matrix

Model repository

ST Edge AI Model Zoo

Model repository

Optimized with

STM32Cube.AI

Optimized with

Compatible with

STM32F4 series

Compatible with

Resources

Model repository ST Edge AI Model Zoo

A collection of reference AI models optimized to run on ST devices with associated deployment scripts. The model zoo is a valuable resource to add edge AI capabilities to embedded applications.

Model repository ST Edge AI Model Zoo

Optimized with STM32Cube.AI

A free STM32Cube expansion package, X-CUBE-AI allows developers to convert pretrained AI algorithms automatically, such as neural network and machine learning models, into optimized C code for STM32.

Optimized with STM32Cube.AI

Compatible with STM32F4 series

The STM32 family of 32-bit microcontrollers based on the Arm Cortex®-M processor is designed to offer new degrees of freedom to MCU users. It offers products combining very high performance, real-time capabilities, digital signal processing, low-power / low-voltage operation, and connectivity, while maintaining full integration and ease of development.

Compatible with STM32F4 series

You also might be interested by

Entertainment | Wearables

Yoga pose recognition on wearable devices

Pose recognition and classification on a sensor.

Entertainment | Wearables

Gym activity recognition on wearable devices

Activity recognition and classification on a sensor.

Industrial | Smart buildings | Smart homes | Smart offices

Face identification with ID3 Technologies

End-to-end AI solution for face identification running on STM32 microcontrollers.