Reported by Machine Heart
Projecting content from a smartphone onto any surface for “touchscreen” interaction feels both familiar and a bit magical…
Since the advent of smartphones, interacting with touch and digital content has become ubiquitous. However, so far, touchscreen technology has mainly been limited to portable devices.
Recently, a research team from several universities in Japan proposed a new low-cost method to turn any surface into a touchscreen, providing new possibilities for interaction with the digital world.
Previous works that allowed touch manipulation of projected images mostly relied on special input devices, multiple sensors, or image processing algorithms, making it difficult to handle chaotic or confusing visual content. The new system proposed by the research only requires a camera connected beneath the projector, determining whether a finger has touched the screen surface from a single camera image, and because this method only captures an area slightly above the screen, the projected image of the system remains robust and unaffected by visual interference.
Paper link:https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9495800
The key to this system is a clever optical trick that ensures only motion above the projection surface is detected. This allows the system to track the user’s finger when pressing a button while ignoring all other content in the camera’s field of view. Researchers hope this technology can be used to create large interactive displays anywhere.
This touch sensing system consists of a projector, a camera, and a microcontroller, requiring no additional cameras, depth sensors, or light sources. The projector serves a dual purpose: 1) projecting images onto the surface; 2) synchronizing with the camera to image only the light source slightly above the projection screen.
Compared to other camera-based methods, this approach has privacy advantages, as the device only captures light from the space directly above the projection surface. In other words, the system does not capture privacy-sensitive areas outside the area of interest, such as faces or other identifiable details.
The system identifies touch sensing by recognizing the position of fingertips in the scene images captured by the camera, and the researchers developed a method to selectively capture only the area slightly above the projection screen.
There are two main challenges to overcome:
The first challenge is that the visual information of the projected image can interfere with hand detection. In particular, the projection colors overlap on the fingertip skin, making it difficult to extract the fingertip area from the image. Additionally, if the projected image content includes hands or people, the system cannot distinguish between a fake hand in the projected image and a real hand touching the screen. One way to overcome this challenge is to use additional devices, such as infrared cameras or depth sensors, to detect real hand regions as non-planar objects. However, this extra hardware increases the complexity of device configuration.
The second challenge arises from performing touch detection and fingertip localization based on a single fixed camera image. While multiple cameras can triangulate the 3D coordinates of the finger, this also increases the size and computational complexity of the interactive device.
As shown in Figure 2, the following parts do not need to be perceived: projected content, hand areas outside the fingertips, surrounding scenery, hovering fingertips, and fingertips outside the touchscreen range.
To selectively capture only the plane and all related optical interactions, the research utilized the Slope Disparity Gating method.
Figure 3. (a) Conceptual illustration of Slope Disparity Gating, allowing the camera to capture a single plane in 3D space [7]. (b) Imaging selected areas. While the projector casts an image surrounded by the blue line in perspective projection, the imaging technology of this research enables it to capture only the area surrounded by the red line.
By synchronizing these two devices, the horizontal plane of light emitted by the projector intersects with the horizontal plane received by the camera. Due to a slight offset between the two devices, the depth of the overlapping points can be calculated using triangulation principles.
This enables calibration settings so that the camera only picks up light at a specific distance from the projector, which can be set to hover directly above the projected image. This means that while the camera tracks the user’s finger when pressing the area of the projected image, it also ignores the rest of the visual scene.
Additionally, the paper details how to combine this setup with simple image processing algorithms to track the user’s finger relative to the projected image, and this tracking information can serve as input for any touch-based application.
One of the research members, Jayasuriya, stated: “The initial prototype of this system cost around $500, and if commercialized, the cost could be greatly reduced further.”
Currently, the device can only track one finger at a time, but enabling multi-touch would only require the researchers to replace the image processing algorithm with a smarter one. The research team hopes that in future iterations, the device will achieve more complex gesture recognition.
Interested readers can read the original paper for more research details.
Reference link: https://spectrum.ieee.org/any-surface-a-touchscreen
Training Larger Models Using DTR and Mixed Precision Techniques
On September 15, from 19:00 to 20:00, algorithm researcher Xiao Shaoran from Megvii Research Institute and Megvii MegEngine architect Chen Zhenhuan will give a detailed explanation of how MegEngine, powered by DTR technology, excels in the field of large model training.
The live broadcast will take place on the Mobile Group video account, welcome everyone to make an appointment and join the group.
© THE END
For reprint, please contact this public account for authorization
Submission or reporting: [email protected]
