1. Research Background:
With the rapid development of intelligent robotics technology, human-machine interaction (HMI) based on acoustic sensors plays a crucial role in facilitating natural and efficient communication for robots. However, accurately identifying and tracking omnidirectional sound sources, especially in noisy environments, remains a pressing challenge.
2. Article Overview:
In response to the above issues, Professor Wang Jie’s team at the Beijing Institute of Nanoenergy and Nanosystems has successfully developed a self-powered triboelectric stereo sensor (SAS) with omnidirectional sound recognition and tracking capabilities, providing an innovative solution to this problem. The SAS utilizes a porous vibrating membrane with high electron affinity and low Young’s modulus, achieving high sensitivity (3172.9 mVppPa-1) and a wide frequency response range (100-20,000 Hz). Utilizing its omnidirectional sound recognition ability and adjustable resonance frequency characteristics, the SAS can accurately identify desired audio signals even in noisy environments, with an average deep learning accuracy rate of approximately 98%. The development of this sensor not only addresses the challenges of sound recognition for intelligent robots in complex environments but also opens up broad prospects for its application in various fields. For example, in assistive meeting systems, the SAS can simultaneously recognize multiple individual voices, enhancing meeting efficiency; in the field of autonomous vehicles, it can accurately recognize driving commands amidst background music, ensuring driving safety. These applications mark significant advancements in voice-based human-machine interface systems. The relevant research results have been published in Advanced Materials.The first author is Dr. Qiao Wenyuan from the Beijing Institute of Nanoenergy and Nanosystems, and the corresponding authors are Associate Researcher Zhou Linglin and Professor Wang Jie.
3. Research Content:1. Sensor Structure and Working Principle
In the pursuit of more efficient and natural human-machine interaction (HMI) systems, acoustic sensors play a crucial role. Acting as the “hearing” device for robots, acoustic sensors can precisely identify human commands, speech content, and intonation, greatly enhancing social interaction between robots and humans. This article proposes an innovative omnidirectional SAS, which integrates five layered self-powered triboelectric acoustic sensors (TAS) on a 3D-printed stereo framework to achieve omnidirectional sound signal capture and efficient recognition. The working principle of the TAS mainly includes two aspects: the deformation of the FEP membrane caused by sound waves, and the vibration of the membrane actively converting sound signals into electrical signals.
By analyzing Equations (1), (2), and (3) and Figure 1, it is clear that the key indicator affecting TAS sensitivity is voltage (U); by adjusting parameters such as Young’s modulus (E), radius (r), and membrane thickness (t), the vibration displacement of the TAS can be altered, thus changing the sensitivity of the device’s voltage output. Additionally, these three parameters can also adjust the TAS’s resonance frequency (f0). To achieve multi-directional sound recognition and real-time tracking of sound sources in noisy environments, the authors introduced a 3D-printed SAS with uniformly distributed five surface cavities, each integrating a single TAS. Based on the omnidirectional sound recognition and tunable resonance frequency characteristics, the SAS demonstrated the ability to pick up target sounds in noisy environments, which has been validated in autonomous driving HMI vehicles. To prove the principle of the SAS, the displacement responses of the TAS and SAS under different sound source incidence conditions were simulated. When the sound source is directly in front of the TAS, it exhibits the maximum signal response; when the sound source is between two TAS, the adjacent two TAS have the same and maximum signal response. Based on these characteristics, the position and angle of the sound source can be determined according to the response of the SAS.
Figure 1. Sensor structure and working principle. a) Conceptual diagram of sound sensor-based human-machine interaction. b) Detailed layered structure of TAS. c) Cross-section of TAS. d) Working principle of TAS. e) Effects of different parameters on TAS vibration deformation displacement (W), resonance frequency (f0), and voltage (U). f) Structural diagram and function of SAS. g-i) Simulated response amplitude distribution of TAS and SAS when the sound source directly strikes a single plane and diagonally strikes the SAS.
2. Acoustic Performance of TAS
Figure 2 systematically studies the effects of parameters E, t, and r on TAS performance. In comparison, FEP has a smaller E than PI and PET films, making it more susceptible to deformation under vibration. Additionally, without any extra processing, these three materials were used as TAS membranes, and their sound signal responses were tested in the frequency range of 100-20,000 Hz. The results show that FEP-TAS has a larger and more complete sound signal response; thus, we selected FEP film with a lower E value as the membrane material for TAS. It is noteworthy that E is an inherent property of the material and is not affected by membrane thickness t, but the thickness t significantly impacts TAS sensitivity and resonance frequency. As the membrane thickness decreases, the TAS sensitivity increases, the signal-to-noise ratio increases, and the resonance frequency undergoes redshift. Additionally, as the radius decreases, the TAS sensitivity decreases, the signal-to-noise ratio decreases, and the resonance frequency undergoes blueshift. Although the miniaturization of the device affects TAS sensitivity and signal-to-noise ratio, we can still improve the sensitivity and signal-to-noise ratio of small devices to some extent by reducing t, using low E materials, etc. Moreover, by improving the triboelectric performance of the materials, such as selecting vibrating membranes with high electron affinity or modifying triboelectric materials using surface microscopy, the sensitivity and signal-to-noise ratio of TASD can be further enhanced, thus promoting further miniaturization of SAS. In the study of TAS stability, it was found that after continuous testing for 120 hours, its voltage output remained at approximately 93%, and f0 showed only minor deviations after the stability test. Compared to electret microphones, the TAS exhibits higher stability because it can continuously replenish triboelectric charges during vibration. This high stability lays a solid foundation for the practical application of TAS-based human-machine interfaces.
Figure 2. Characterization of TAS acoustic performance. a) E values of FEP, PI, and PET films with a thickness of 50μm. b) Frequency domain signals obtained from different TAS made with 50μm FEP, PI, and PET vibrating membranes. c) Thickness does not affect the material’s E, and the slopes of the elastic deformation region are almost consistent. d) Frequency domain signals of TAS made from FEP films of different thicknesses. e) Frequency domain signals of TAS with different r. f) Voltage of TAS at different sound pressures, used to calculate sensitivity. g) Voltage signals of TAS at different sound pressure levels, used to calculate signal-to-noise ratio. h) Stability test of TAS, with voltage signal output normalized to 1Pa voltage. i) Frequency domain signals before and after TAS stability test.3. Performance of SAS
To achieve multi-angle, multi-directional, high-sensitivity sound recognition and tracking, the authors proposed a 3D-printed SAS, which has five planes for sound sensing, with the bottom being empty for placement on a desktop. As shown in Figure 3, TAS #1 is placed on the bottom surface facing the opposite plane, while TAS #2 to #5 are placed on the adjacent four planes, forming an omnidirectional application scenario. When the sound source is directly in front of the plane, the angle between the sound source and TAS is 0°. When the sound source is diagonally incident, the angle between the sound source and TAS is defined as α. When the incident angle is 0°, the TAS facing the plane outputs maximum voltage, and when the incident angle is α, the adjacent two TAS output maximum voltage. Additionally, the maximum voltage value at an incident angle of 0° is higher than the two maximum voltage values at an incident angle of α. When the sound source strikes different planes and diagonally strikes the SAS, the corresponding voltage signal and frequency spectrum intensity distribution present consistent results, indicating that the SAS can effectively capture sounds from different angles and identify the sound source angle through simple signal intensity analysis. To further illustrate the omnidirectional sound recognition and positioning capabilities of SAS, the sensitivity of each SAS plane under different sound pressures was studied when the sound source is directly in front of plane #2. The sensitivity of each SAS plane shows a linear relationship with sound pressure, with TAS #2 exhibiting the highest sensitivity, reaching 3172.9 mVpp Pa-1. Furthermore, regardless of which plane the sound source is incident upon, the SAS demonstrates a high average overall sound response of approximately 4.382V, indicating that the SAS can achieve spatial omnidirectional sound recognition and positioning with high sensitivity.
Figure 3. Performance and characterization of SAS. a) Expanded view of SAS framework planes. b) Stereoscopic view and angle introduction of the cube. c) Voltage signals collected by SAS when the sound source directly strikes at 0°. d) Voltage signals collected by SAS when the sound source strikes at angle α. e) Corresponding frequency spectrum intensity distribution when the sound source strikes from different planes and diagonally strikes. f) Sensitivity of each SAS plane under different sound pressures when the sound source strikes directly at plane #2. The sensitivity of each SAS plane shows a linear relationship with SP. g) Omnidirectional sound response and average voltage of SAS when the sound source strikes planes #1, #2, #3, #4, and #5.
4. Assistive Meeting System and Sound Source Tracking
Current microphones used in meetings face significant challenges regarding sensitivity and limitations in recognizing multi-directional sound sources. In contrast, the SAS, with its exceptional sensitivity, superior signal-to-noise ratio, and omnidirectional sound recognition and positioning capabilities, makes it an ideal choice for optimizing assistive meeting human-machine interface systems. As shown in Figure 4, to enhance the richness and accuracy of meeting information recognition, we established a deep learning model based on convolutional neural networks (CNN) to learn and recognize commands, achieving an average accuracy rate of 99.36%. With this model, the assistive meeting human-machine interface system can not only accurately recognize and track direction when the speaker is opposite and speaking separately, but also accomplish complex tasks of identity recognition, direction positioning, and trajectory tracking when the speakers are adjacent and speaking simultaneously. This breakthrough overcomes the technical bottleneck of simultaneously recognizing multiple individuals in traditional meeting systems, significantly enhancing the interactivity and efficiency of meetings. Additionally, SAS demonstrates excellent capabilities in sound source trajectory recognition and tracking.
5. Autonomous Driving Human-Machine Interface System in Background Music
Using voice-based HMI in autonomous driving technology can achieve hands-free operation, improve efficiency, and reduce driver fatigue. However, in this process, voice-based human-machine interface systems are easily affected by interface noise and the accuracy of sound sensors, potentially impacting the driver’s voice commands to the intelligent driving system, thereby affecting the driving experience. As shown in Figure 5, the intelligent driving system based on SAS, due to its high sensitivity, wide frequency bandwidth, and high signal-to-noise ratio, can recognize sounds in omnidirectional working spaces with strong background noise, providing a feasible solution to the above issues. Furthermore, the intelligent driving system based on SAS can identify the speaker’s identity based on speaking direction, voltage signal, frequency domain signal, and frequency spectrum intensity, avoiding misjudgments of command issuers when multiple sounds occur. Even in environments with music playing, the average accuracy rate of SAS in recognizing sound signals reaches 97.73%, allowing the autonomous vehicle HMI system to respond to commands. This powerful performance is primarily attributed to the unique omnidirectional sound recognition capability of the SAS’s 3D structural design, as well as the differences in time-domain and frequency-domain signals from different sound sources. Although voltage signals are collected through a common channel, the differences in resonance frequency, uniquely attributed to each individual sound source, help eliminate noise interference and successfully pick up target sounds, combining the SAS’s ability to adjust resonance frequency with its precise directional recognition provided by its 3D structure, enabling it to effectively extract the desired sound from noisy environments.
6. Conclusion and Outlook:
The authors propose a self-powered SAS that adopts a unique cubic design, endowing it with both omnidirectional sound response and precise tracking capabilities. By combining low E porous vibrating membranes, the SAS achieves high sensitivity (3172.9 mVppPa-1) and a wide frequency response range (100-20,000 Hz). Utilizing the omnidirectional sound recognition and tracking capabilities of SAS, along with its differentiated resonance frequency responses to different sound sources and directions, the goal of efficiently extracting target signals from noisy backgrounds has been achieved. With the assistance of deep learning, the recognition accuracy of target signals by SAS averages around 98%. More importantly, the emergence of SAS breaks the limitations of multiple people interacting with robots simultaneously. Furthermore, SAS successfully demonstrates its excellent performance in assistive meeting systems, sound tracking, and autonomous driving systems (especially in accurately recognizing driving commands in environments with background music). This research highlights the profound advantages of self-powered triboelectric technology in voice-based human-machine interface systems.
7. Acknowledgments
Thanks to the Ministry of Science and Technology and the National Natural Science Foundation project for their funding.
Source: Kexue Jianshe
For submissions, reprints, or collaborations, please contact WeChat:MatResFron001
Welcome to leave a message and share your views.Support the author by clicking the “like” button in the bottom right corner and “see”↓↓↓