In recent years, advancements in robotics have accelerated the development of intelligent manufacturing systems. Among these, object recognition based on visual perception plays a crucial role. However, due to the optical transparency of liquids and the similarity in their tactile properties, the identification and classification of liquids in containers remain a field that has not been fully explored.
According to a report by MEMS Consulting, a research team from Korea University has recently proposed a robotic finger multimodal tactile sensor system based on machine learning, which can be used for the classification of bottled liquids (Figure 1). This tactile sensor system integrates thermal conduction and frequency response perception mechanisms, inspired by the temperature and vibration receptors in the human body. Additionally, a dual-parallel convolutional neural network is employed to process multimodal inputs. The multimodal tactile sensor system demonstrates two different classification tasks: classification of water volume and classification of liquid types. The proposed model achieves high classification accuracy in both tasks and can operate in real-time. This biomimetic approach provides a new solution for classifying bottled liquids through static contact, showing great potential for applications in robotic perception. The research findings were published in the journal npj Robotics under the title “Liquid classification in robotic fingers with multimodal tactile sensor system.”
Figure 1: Robotic finger multimodal tactile sensor system based on machine learning for classification of bottled liquids
Design and working principle of the multimodal tactile sensor system and machine learning process
To achieve liquid classification, the researchers designed a multimodal tactile sensor system that utilizes thermal conduction and vibration perception machine learning models. This system includes a thermal sensor module (for sensing thermal conduction) and a vibration sensor module (for sensing the frequency response of object vibration), both of which can operate independently. The overall design of the system and the data obtained are shown in Figure 2. The configuration of the thermal sensor module is illustrated in Figure 2 (a). Specifically, the thermal sensor module employs a thermoelectric device (TED) that operates based on the Peltier effect. Through this effect, the TED can generate heat under voltage input, causing a temperature change between the thermal sensor module and the contacted object. Additionally, the thermal sensor module is equipped with a resistance temperature detector (RTD) to measure temperature changes. The RTD can detect temperature changes by measuring the voltage output generated from linear resistance changes, providing high precision and stability in measurements.
Figure 2: Architecture of the multimodal tactile sensor system
The working principle of the thermal sensor module is shown in Figure 2 (b). The raw voltage data obtained when using the thermal sensor module to grasp a water bottle is illustrated in Figure 2 (c). To convert the voltage signal into temperature changes and smooth the output data from the thermal sensor module, data preprocessing was performed, with results shown in Figure 2 (d).
The configuration of the vibration sensor module is shown in Figure 2 (e). This module operates based on the piezoelectric effect. The choice of polyvinylidene fluoride (PVDF) film is due to its ability to support electromechanical coupling, enabling the generation and perception of vibrations. The module consists of an actuator and a sensor, where the actuator uses a PVDF film with a thickness of 110 μm to ensure high voltage tolerance, while the sensor uses a PVDF film with a thickness of 28 μm to achieve high sensitivity. The actuator is designed in a bipedal structure to concentrate and enhance the vibration signal. The vibration sensor module is mounted on the robotic finger using a silicone finger sleeve, measuring the frequency response related to the liquid when grasping the bottle. Through the piezoelectric interaction between the actuator and the sensor, the vibration sensor module can capture different frequency responses generated by the liquid.
When a sine signal is input to the actuator, deformation occurs due to the piezoelectric effect, generating vibrations. The produced vibrations propagate through the grasped object and act on the sensor. When the sensor undergoes mechanical deformation due to vibrations, it measures these mechanical deformations as voltage changes using the piezoelectric effect. The recorded voltage changes are analyzed to identify the characteristic frequency response of the liquid bottle. The working principle of the vibration sensor system when grasping a water bottle and the raw voltage data are shown in Figures 2 (f) and 2 (g), respectively.
Classification of objects using a neural network model. The neural network model consists of a feature learning part and a classification part. The feature learning part of the model is illustrated in Figure 2 (i). To achieve accurate classification, the model combines the features extracted by each tactile sensor through flattening and concatenation, followed by processing through fully connected layers. The classification part of the model is shown in Figure 2 (j).
Classification of water volume
To validate the performance of the multimodal tactile sensor system in liquid classification, the researchers conducted two tests: (i) classification of water volume in bottles; (ii) classification of liquid types in bottles. First, in the water volume classification experiment, commercially available 500 mL PET bottles (Evian, Danone, France) were selected due to their widespread use. Six bottles were filled with 0, 100, 200, 300, 400, and 500 mL of water, respectively. The robotic finger grasped the bottles in a parallel manner, with the left finger corresponding to the thermal sensor module and the right finger corresponding to the vibration sensor module. The grasping point was located between the 100 and 200 mL water levels. Each water bottle was grasped 200 times, and each sensor module collected 200 sets of data for each bottle. During data collection using the multimodal tactile sensor system, the gripper holding each bottle is shown in Figure 3 (a). The average temperature change and total temperature change of the thermal sensor module are shown in Figures 3 (b) and 3 (c), respectively. The average frequency response and total frequency response of the vibration sensor module are shown in Figures 3 (d) and 3 (e), respectively.
Figure 3: Photos and charts of water volumes to be classified
Each of the 200 samples from each bottle was randomly divided into training, validation, and test sets in a ratio of 8:1:1. Using the proposed multimodal tactile sensor module and a dual-parallel 1D-CNN (convolutional neural network) model, a classification accuracy of 97.5% was achieved. The confusion matrix is shown in Figure 4 (c), demonstrating the excellent classification capability of the multimodal tactile sensor system.
Figure 4: Confusion matrix of the test dataset for water volume classification
Classification of liquid types
To classify liquid types, six identical PET bottles were filled with acetone, water, ethanol, FR3 oil, canola oil, and Mictrans oil, respectively. Each bottle contained 200 mL of liquid, a volume that minimizes the distance between the sensor module and the liquid surface, thus producing the highest peak amplitude in the frequency response. Similarly, each bottle was grasped 200 times, resulting in 200 sets of data collected for each sensor module for each type of liquid. The average temperature change and total temperature change recorded by the thermal sensor module are shown in Figures 5 (b) and 5 (c), respectively. The average frequency response and total frequency response of the vibration sensor module are shown in Figures 5 (d) and 5 (e), respectively.
Figure 5: Photos and charts of liquids to be classified
The proposed multimodal tactile sensor system combined with the dual-parallel 1D-CNN model achieved a maximum classification accuracy of 97.5%. The confusion matrix is shown in Figure 6 (c), highlighting the excellent performance of the system and model in classification. These results confirm the effectiveness of the system in distinguishing different types of liquids.
Figure 6: Confusion matrix of the test dataset for liquid type classification
In summary, this study proposes a machine learning-based robotic finger multimodal tactile sensor system for the classification of bottled liquids, achieving multimodal perception of temperature changes and frequency responses through a unified architecture. The developed system consists of a multimodal tactile sensor system and an optimized machine learning process. The thermal sensor module measures temperature changes caused by thermal conduction using a thermoelectric device (TED) and a resistance temperature detector (RTD), while the vibration sensor module detects frequency responses generated by the piezoelectric effect using PVDF film. Both sensor systems employ various data preprocessing techniques, such as polynomial regression and fast Fourier transform (FFT). Furthermore, the dual-parallel 1D-CNN is specifically designed to effectively extract and integrate different multimodal features from the sensor systems. The performance of the multimodal tactile sensor system was validated through two experiments: “water volume classification” and “liquid type classification.” In conclusion, the proposed multimodal tactile sensor system provides a biomimetic-inspired solution for the classification of bottled liquids, demonstrating broad application prospects in robotic finger perception.
Paper link:
https://doi.org/10.1038/s44182-025-00051-2
