🔍 Latest Overview of Vision-Based Tactile Sensors: Hardware, Algorithms, Applications, and Future Perspectives

Based on the IEEE T-IM 2025 frontier review “A Survey of Vision-Based Tactile Sensors: Hardware, Algorithm, Application and Future Direction”

🧩 1. Research Background: From “Touch” to “Smart Touch”

Traditional tactile sensors (resistive, capacitive, piezoelectric, etc.) have limitations in terms of resolution, sensitivity, and long-term stability, making it difficult to meet the fine operation requirements of robots in complex, unstructured environments.
Vision-Based Tactile Sensors (VBTS) convert contact deformation into high-resolution images, combined with computer vision and deep learning algorithms, achieving “visible touch”.
Key advantages:

Ultra-high spatial resolution: Capturing micron-level surface geometric features
Multimodal information: Simultaneously perceiving force, pressure, texture, temperature, etc.
Flexibility and adaptability: Suitable for grasping irregular and fragile objects

🏗️ 2. Hardware System: System Optimization from Materials to Optics

⚙️ 2.1 Structure and Core Modules

Perception layer: Multi-layer structure supporting, contacting, marking, reflecting, and protecting
Camera module: Monocular/binocular/depth/event cameras
Lighting module: RGB LEDs, multi-angle uniform illumination

💡 2.2 Working Principle

VBTS converts mechanical information into visual signals through optical means, mainly in three categories:

Light guide plate type
Marker displacement type
Reflective film type

Among them, marker displacement type has become mainstream due to its high precision and robustness.

🏷️ 2.3 Representative Devices

GelSight series: Pioneers in micron-level morphology detection
TacTip / DIGIT: Balancing low cost and high resolution
OmniTact, FingerVision: Supporting multi-directional touch and complex terrains

Future direction: Miniaturization, low cost, modularization, and strengthening optical optimization and anti-fouling wear resistance.

🧠 3. Algorithm Ecosystem: Seven Core Functions and Technological Evolution

Function	Key Algorithms	Application Value
Shape Reconstruction	Optical flow, U-Net, Transformer, Implicit Neural Representation (INR)	Obtaining accurate 3D point clouds, providing geometric basis for grasping positioning
Slip Detection	Physical models, ConvLSTM, Video Transformer	Real-time prediction of slip, ensuring grasp stability
Force Sensing	ResNet, TransForce, iFEM	Millinewton-level three-dimensional force measurement
Contact Area Segmentation	PSPNet, DeepLabV3	Providing precise contact area input
Pose Estimation	DGML, multimodal CNN + particle filter	Improving six degrees of freedom grasping accuracy
Object Recognition	Transformer, diffusion models, GAN	Supporting zero-shot and cross-modal recognition
Material Property Sensing	kNN, Bayesian NN, deep fusion models	Identifying hardness, texture, temperature, viscosity, and other characteristics

Trend Insights

From traditional image processing → deep learning → generative and multimodal large models
Cross-modal fusion of touch-vision-language (e.g., UniTouch, GPT-4V) has become a research hotspot

🌐 4. Application Landscape: A Panoramic Layout from Laboratory to Industry

🏭 4.1 Smart Manufacturing

Defect and crack detection
Micron assembly and pressure monitoring

🩺 4.2 Medical Rehabilitation

Minimally invasive surgery and tumor palpation
Home rehabilitation and hand strength training

🧵 4.3 Textiles and Agriculture

Fabric defect detection, texture assessment
Fruit ripeness grading, non-destructive harvesting

🏠 4.4 Other Frontiers

Infrastructure inspection (bridge cracks, underwater structures)
Service robots (household chores, feeding, Braille recognition)

VBTS is a key technological foundation for promoting flexible manufacturing, smart healthcare, green agriculture, and public safety.

⚠️ 5. Development Challenges: From Technical Bottlenecks to Industrialization Issues

Insufficient hardware standardization: Heterogeneous designs make algorithms difficult to migrate across platforms
Limited dataset scale: Lack of public tactile datasets similar to ImageNet
Algorithm real-time performance and computational cost: Industrial sites require millisecond-level response
Insufficient robustness: Long-term stability affected by lighting, temperature and humidity changes, and material aging

🚀 6. Future Trends: The Evolution Path of Tactile Intelligence

New materials and optical breakthroughs: Developing flexible, wear-resistant, and highly transparent elastomers, optimizing optical paths and miniature cameras
Multimodal large models: Building large foundational models (LTM) that integrate vision, touch, and language
Large-scale open datasets: Such as FoTa, Touch100k, to promote algorithm generalization and zero-shot learning
Industrial standardization and commercialization: Unifying interfaces and maintenance systems to accelerate mass production and implementation

🔑 7. Conclusion: Moving Towards a Future of “Seeing and Feeling”

Vision-based tactile sensors are reshaping the interaction between robots and their environments, from manufacturing to healthcare, from agriculture to service industries, serving as the core technological foundation for future intelligent industries. With the continuous breakthroughs in deep learning, generative models, and multimodal perception, the next generation of robots will truly achieve “visible touch and intelligent actions”, providing safer, more efficient, and flexible solutions for an intelligent society.

References: He, K. (2025). A Survey of Vision-Based Tactile Sensors: Hardware, Algorithm, Application and Future Direction. IEEE Transactions on Instrumentation and Measurement, 74, 9533221.

💭 Thoughts and Discussions

Are the existing vision-based tactile sensors the best solution?
Can tactile datasets largely be solved with simulated data?
Is tactile data collection more difficult to standardize than visual data in practical deployment?

Feel free to leave your thoughts in the comments, and share this article to let more people understand the future of vision-based tactile robots!

📚 Recommended Further Reading

Dahiya et al. (2010): Tactile Sensing—from Humans to Humanoids
Li et al. (2023): Tactile Information Hierarchy in Robotics
DIGIT: An Open-Source, Compact, High-Resolution Optical Tactile Sensor
TVL Dataset: Touch, Vision, Language Dataset for Crossmodal Learning
He, K. (2025): A Survey of Vision-Based Tactile Sensors: Hardware, Algorithm, Application and Future Direction. IEEE T-IM, 74, 9533221