AbstractWearable human-machine interfaces equipped with Inertial Measurement Units (IMUs) are widely used in healthcare, robotics, and interactive technology. However, extracting reliable signals remains challenging due to motion artifacts in real-world environments. This study presents a human-machine interface that withstands various motion artifacts through deep learning-enhanced wearable sensors. The system integrates a six-channel IMU, an electromyography (EMG) module, a Bluetooth microcontroller unit, and a stretchable battery, enabling wireless capture and transmission of gesture signals. A convolutional neural network trained on a composite dataset of gestures and motion artifacts extracts robust signals, while parameter-based transfer learning enhances the network’s generalization ability across users. The sliding window method converts the extracted gesture signals into real-time, continuous control of robotic arms, suitable for dynamic activities such as running, high-frequency vibrations, posture changes, wave movements, and their combinations. This research demonstrates the potential of wearable human-machine interfaces in complex real-world applications.
Deep Learning × Wearable Sensing: A new breakthrough in human-machine interaction that accurately recognizes gestures even in running, vibrations, and waves.
Wearable IMUs (Inertial Measurement Units) have been widely applied in human-machine interaction, medical rehabilitation, and robotic control. However, they face a common challenge in real-world environments: motion artifacts. Whether walking, running, swinging arms, or even slight movements of the device itself, IMU signals become “noisy,” making it difficult for the system to recognize true gestures.
This study presents a deep learning-enhanced wearable sensor system that can accurately capture user gestures even in the most complex dynamic scenarios—such as running, bumpy rides, high-frequency vibrations, and floating in waves—while controlling devices like robotic arms, achieving continuous and stable human-machine interaction.
——-
01
A small wearable system with multi-sensor fusion: a “gesture catcher” that can bend, stretch, and operate wirelessly.
The sensor system we developed consists of the following modules:
• Six-channel IMU (accelerometer + gyroscope)
• Electromyography (EMG) sensing module
• Bluetooth microcontroller
• Stretchable flexible battery
All components are integrated into a multilayer wearable circuit with an area of only 1.8 × 4.5 cm, a thickness of about 2 mm, and can stretch over 20%. This design allows it to fit snugly against the skin, maintaining good contact and signal quality during movement.
The battery can maintain a capacity of about 25mAh even when stretched and can provide stable power for over 4 hours.
The Bluetooth module connects stably within 20 meters; after continuous operation for 30 minutes, the temperature is only 27.7°C, ensuring long-term wear is safe and comfortable.
Through these hardware modules, the system can simultaneously collect:
• 6-channel IMU, 100 Hz sampling
• 1-channel EMG, 1000 Hz sampling
for real-time restoration of 19 types of subtle forearm gesture actions.
———-
02
The biggest challenge: gesture signals are overwhelmed by “motion artifacts.”
In real-world environments, IMU readings are often interfered with by the following factors:
• Environmental actions such as running, walking, and accelerating
• Gravity vector shifts caused by changes in arm direction
• High-frequency vibrations (e.g., from power tools)
• Superimposed complex movements
• Gesture variations due to individual differences
The signal patterns of these motion artifacts often resemble the gestures themselves.
Traditional hardware methods (such as thinning the device, enhancing adhesion, and using multiple IMUs) and software methods (filtering, PCA, wavelet transforms, etc.) cannot reliably separate gestures from artifacts. Existing machine learning models often train on idealized Gaussian noise and cannot cover the complex interferences of the real world.
———-
03
Using “composite datasets + CNN” to solve motion artifacts: teaching the algorithm to distinguish between “real actions” and “environmental noise.”
To enable the system to cope with real interferences, a composite dataset was constructed, including:
• Real forearm gestures collected in sitting and lying positions
• Strong motion artifacts (running, maximum speed/amplitude vibrations, posture transitions, etc.)
• Mixed signals of gestures and artifacts superimposed at different ratios and stages
Based on this, an optimized convolutional neural network (CNN) was trained.
The model exceeded 0.94 in recognition accuracy, recall rate, and F1 score, outperforming fully connected networks and LSTMs.
When tested with “new running speeds,” “new vibration frequencies,” and “new posture changes” that it had not seen before, the model still operated stably with an accuracy of over 94%.
—–
04
Using transfer learning to make the system “adapt to new users”:
Each gesture requires only 2 samples for personalization.
Differences in gesture strength, angle, and habits among users can reduce the model’s generalization ability.
To address this, we incorporated a parameter-based transfer learning strategy:
1. First, train a general model using the composite dataset
2. Then fine-tune with a minimal number of samples from the new user
Transfer learning optimization ultimately achieved:
• Originally required 40 samples per gesture
• Now only requires 2 samples per gesture (sitting + lying)
After transfer learning, the minimum recognition accuracy for 19 gestures improved from 51% to over 92%.
Data collection time reduced from 2 minutes to under 6 seconds.
Real-time control mechanism:
- Sliding window processing:
- Window length: 1 second
- Sliding step: 0.25 seconds
- Triple verification logic to eliminate false triggers
- EMG signal auxiliary verification (SNR > 29dB)
Special underwater applications
- Using the Scripps Oceanographic Simulator to construct a wave interference dataset
- Parameterized modeling of wavelength/wave height
- Advantages of seawater environment: reduces electrode-skin impedance by 35%
——-
05
Continuous gesture recognition: sliding window + EMG, making robotic arms act more like humans.
To enable robotic arms to perform actions continuously, we use:
• 1-second window, 0.25-second sliding step for continuous prediction
• Execute actions only after three consecutive identical predictions (to prevent false triggers)
• EMG RMS value controls grip and release
Even under extreme motion artifacts:
• Running
• High-frequency vibrations
• Posture changes
• Mixed artifacts
The system can still stably drive robotic arms to perform complex tasks, such as pouring liquid from a test tube into a beaker without shaking.
In tests with real collected “artifact + gesture” signals, the accuracy of robotic arm control remains over 93%.
Technical Comparison and Advantage Analysis
| Dimension | Traditional Solutions | This Technical Solution |
|---|---|---|
| Interference Resistance | Relies on spectral separation | Deep learning feature extraction |
| Environmental Adaptability | Static scenes | Dynamic scenes (running/waves) |
| User Adaptation | Requires complete training | Quick adaptation through transfer learning |
| Response Delay | >2 seconds | 1.3 seconds |
⸻
06
Human-machine interaction underwater: allowing divers to control robots with gestures.
In underwater environments, IMUs must handle not only motion artifacts but also complex dynamic disturbances caused by waves.
• Collected actual wave data in Mission Bay, California
• Simulated waves of different wavelengths and heights in the Scripps Oceanographic Research Simulator
• Constructed a large training set containing various “wave artifacts”
The trained system performs excellently in simulated diving scenarios:
In offline tests during simulated wave movements, the model’s precision, recall, specificity, and F1 score are all close to 1.
Moreover, seawater reduces skin impedance, thereby enhancing EMG signal quality, making underwater gesture recognition more stable.
—————————————————————
Conclusion:
Human-machine interfaces for the real world, ushering wearable devices into the “anti-disturbance era”.
This study constructs a highly integrated, stretchable, multi-modal collection wearable sensor + deep learning model that can achieve high-precision gesture recognition under various complex motion artifacts and stable robotic arm control.
Highlights of the research include:
• Multi-layer stretchable flexible circuit design
• IMU + EMG multi-modal fusion
• Composite dataset construction for real-world disturbances
• CNN achieving robust feature extraction
• Transfer learning enabling rapid adaptation to new users
• Sliding window for continuous action control
• High-precision demonstrations under running, vibration, posture changes, and wave conditions
Applicable fields:
• Human-robot collaboration
• Sports rehabilitation/medical assistive devices
• AR/VR gesture interaction
• Underwater robot control
• Wearable interaction in outdoor/sporting environments
Future prospects:
Further optimizations can be made:
• Shortening window length to reduce recognition delay
• Introducing gesture segmentation models to recognize shorter and more natural gestures
• Employing deeper CNNs or attention mechanisms to enhance feature extraction capabilities for short window signals, allowing wearable devices to truly achieve—anytime, anywhere, with no fear of interference, and precise understanding of actions.
⸻
References:
https://www.nature.com/articles/s44460-025-00001-3#Fig1