Hand posture recognition without camera module
Hand posture recognition running on STM32F401 based on ST multizone Time-of-Flight ranging sensor.
What if your phone could send your friends emojis, based on the movement of your hands?
It has now become a reality! Thanks to ST multi-zone Time-of-Flight (ToF) sensors, this solution does not require a camera. AI algorithms are running on the STM32 microcontroller, with a low processing complexity and requiring low power consumption.
Define your own set of hand postures, collect your dataset, train your AI model, and create your application !
Approach
The development process is based on the following steps:
- Define your own set of hand postures (dataset)
- Collect your dataset from several users, using the distance and signal data from 8 x 8 multi-zone ToF sensors, such as the VL53L5CX, VL53L7CX or VL53L8CX
- Train the AI network with the training script from the STM32 model zoo
- Implement the AI model into your selected STM32 MCU thanks to the STM32Cube.AI Developer Cloud or the "Hand Posture Getting Started" included in the STM32 model zoo
This approach allows you to quickly develop this application, with highly configurable hand postures, a small memory footprint and a low processing power.
Depending on the application, the ToF sensor can be positioned either in front of the user (personal computer, satisfaction box), point to the ceiling (cooking plate) or be fixed on a moving object (smart glasses).
Sensor
Data
Data format
8 x 8 ranging distance and signal rate
Frequency aligned with the application and reactivity needed
Results
Model CNN 2D
Memory footprint:
29 Kbytes of flash memory for weights
3 KbytesRAM for activations
Accuracy: 96.4 %
Performance on STM32F401 @ 84 MHz
Inference time: 1.5 ms
Confusion matrix
Resources
Model repository ST Edge AI Model Zoo
A collection of reference AI models optimized to run on ST devices with associated deployment scripts. The model zoo is a valuable resource to add edge AI capabilities to embedded applications.
A free STM32Cube expansion package, X-CUBE-AI allows developers to convert pretrained AI algorithms automatically, such as neural network and machine learning models, into optimized C code for STM32.
Compatible with STM32F4 series
The STM32 family of 32-bit microcontrollers based on the Arm Cortex®-M processor is designed to offer new degrees of freedom to MCU users. It offers products combining very high performance, real-time capabilities, digital signal processing, low-power / low-voltage operation, and connectivity, while maintaining full integration and ease of development.