Source: China Optics
Author: Light New Media
In the movie “Iron Man 2”, there is a scene where the protagonist Tony Stark taps on a floating 3D map, causing dazzling changes. The “information” left by his deceased father—a 3D atomic structure model of a new type of artificial element—is extracted.
Figure 1: Scene from the movie “Iron Man 2”
Many similar scenes in science fiction films reflect people’s endless imagination of 3D display and interaction. Fortunately, such imagination may soon become a reality.
Recently, a team led by Professor Wang Qionghua from Beihang University developed a real-time interactive system for floating true 3D display, showcasing the prototype of this 3D display and interaction system. The system can dynamically display naked-eye 3D images, achieving 30 frames per second of real-time and accurate gesture interaction at a 4K resolution for 3D sources, and supports independent interaction of multiple hands with multiple objects through predefined various gestures.
This achievement is titled “Real-time Interactive System for Floating True 3D Display Based on Leap Motion Gesture Recognition” and was published in Liquid Crystal and Display (ESCI, Scopus indexed, Chinese core journal), 2022, Issue 5.
1. Composition of the Floating True 3D Display Real-time Interaction System
The system is mainly divided into a display module and an interaction module, connected and processed by a computer.
Figure 2: System Structure
Source: Liquid Crystal and Display, 2022, 37(5): 658. Fig. 6
The display module is based on integral imaging 3D display, a technology traceable to the integral photography proposed by Lippmann G. in 1908. Integral imaging 3D display utilizes the principle of reversible light paths, using a lens array to converge and then diverge light at specific spatial points, simulating the situation where light appears to emanate from a real object’s surface, thus being considered a naked-eye true 3D display.
On this basis, the researchers applied a floating device—dihedral mirror array. Light emitted from a point source is reflected twice by each dihedral mirror unit, converging to form an image floating in the air. Therefore, the 3D image of integral imaging 3D display will be projected out of the entire system, providing a stronger visual impact and facilitating freer interaction.
Figure 3: Principle of Floating True 3D Display
Source: Liquid Crystal and Display, 2022, 37(5): 655. Fig. 2
Figure 4: Dihedral Mirror Unit
Source: Liquid Crystal and Display, 2022, 37(5):655. Fig. 3
The interaction module collects gesture data through the motion controller Leap Motion. Leap Motion is a motion controller based on binocular computer vision that captures images through a transparent window on top, extracting the joint features of the user’s hand, thus reflecting the gestures and postures of the entire hand. The data collected by Leap Motion is sent to the computer for processing, converting the information reflected by the user’s gestures into commands that control the changes of the 3D source, allowing the 3D source to dynamically change with the gestures, thereby enabling the 3D images to change dynamically with the gestures in conjunction with the display module.
Figure 5: Leap Motion and Reference Coordinate System
Source: Leap Motion Official Website
2. Implementation Method of Accurate Gesture Interaction
Since the reference coordinate system of Leap Motion does not originally correlate with the reference coordinate system of the 3D image, objects floating to the left of the user’s body may interact with the hand located to the right of the user’s body, naturally leading to discomfort and inconvenience in interaction.
To solve this problem, researchers mapped the coordinates of Leap Motion to the reference coordinate system of the 3D image through calibration and matrix transformation; essentially generating a virtual hand in the virtual space where the 3D image is located to control the changes of the 3D image, achieving unity between the virtual and reality since the user’s hand is in the same spatial position.
3. Independent Interaction of Multiple Hands with Multiple Objects
Currently, some studies on floating true 3D display gesture interaction technology have been conducted both domestically and internationally, but most only consider the interaction of a single 3D object or the overall interaction of a 3D scene, without considering the independent interaction of multiple 3D objects in complex 3D scenes. So how do we understand independent interaction? Imagine using a personal computer in our daily lives; we create two windows and only drag the upper window, which does not affect the lower window; conversely, we consider these two window objects to have independent interactions.
Figure 6: Mouse Interaction in 2D Display
In 2D display mouse interaction, overlapping areas of two windows will cause the upper window to block the lower one, and we can only interact with the upper window; moreover, the mouse can only interact with one point at a time, unable to drag two windows simultaneously, thus independent interaction is relatively easy to achieve. However, in the 3D display and gesture interaction system, once 3D objects 1 and 2 overlap, the overlapping area belongs to both object 1 and object 2. Therefore, sometimes when the user only needs to interact with object 1, they may accidentally pass through object 2, leading to object 2 also being interacted with; we refer to this phenomenon as interaction crosstalk. When the system expands from single-hand interaction to multi-hand interaction, each hand needs to perform different interactions, thus requiring different objects to respond to the corresponding hand’s interaction without interference.
To this end, researchers proposed an interaction channel mechanism. Each 3D object has a channel used to connect to a virtual hand. Successfully connected 3D objects establish an interaction relationship with the hand, and the interaction channel of that object is occupied by the data from that hand; thus, other hands cannot occupy this interaction channel, and other 3D objects will not allow that hand to occupy the interaction channel, making the successfully connected 3D object and hand an independent interaction combination until the hand actively relinquishes the interaction.
Figure 7: Channel Mechanism for Multi-object Interaction
Source: Liquid Crystal and Display, 2022, 37(5): 657. Fig.5
By utilizing the interaction channel mechanism, the interaction crosstalk between multiple hands and multiple objects can be effectively reduced, achieving independent interaction; of course, under specific gestures, several objects in the 3D scene can also interact in the same way globally or locally. The interaction channel mechanism makes the design and implementation of interactions more flexible, further stimulating users’ creativity.
Video: Real-time Interaction Effect of Floating True 3D Display
4. Application Prospects of Floating True 3D Display Real-time Interaction System
The combination of floating true 3D display and gesture interaction aligns with the natural perception of human interaction with real objects in the real world, thus allowing the overlap or even confusion between the real and virtual worlds, achieving both virtual and real information exchange; its applications in audio-visual entertainment, conference office, cultural exhibitions, etc., provide a possible direction for the development of the metaverse. On the other hand, the interaction operations of this system can avoid unnecessary contact, particularly beneficial for reducing the risk of disease transmission, thus possessing potential convenience and safety advantages in the context of the COVID-19 pandemic.
Paper Information
Lin Xingyu, Xing Yan, Zhang Hanle, Wang Qionghua. Real-time Interactive System for Floating True 3D Display Based on Leap Motion Gesture Recognition[J]. Liquid Crystal and Display, 2022, 37(5):654-659.
https://cjlcd.lightpublishing.cn/thesisDetails#10.37188/CJLCD.2022-0060
Corresponding Author Introduction
Wang Qionghua, Professor/PhD Supervisor at the School of Instrument Science and Optoelectronic Engineering, Beihang University, Distinguished Professor of the Chang Jiang Scholars Program, recipient of the National Outstanding Youth Science Fund, leader of the National “Ten Thousand Talents Plan” for scientific and technological innovation, Fellow of the Society for Information Display (SID), Vice Chairman of the Liquid Crystal Division of the Chinese Physical Society, Deputy Director of the 3D Imaging and Display Committee of the Chinese Society of Image and Graphics, Associate Editor of international journals such as the Journal of the Society for Information Display, and editorial board member of journals such as Liquid Crystal and Display. He received his bachelor’s, master’s, and doctoral degrees from the University of Electronic Science and Technology of China in 1992, 1995, and 2001, respectively, served as assistant, lecturer, and associate professor at the University of Electronic Science and Technology from 1995 to 2001, worked as a Research Scientist at the Optical Center of the University of Central Florida from 2001 to 2004, and was a professor and PhD supervisor at Sichuan University from 2004 to 2018. His main research directions are display and imaging technology, having completed more than 20 national-level scientific research projects, and currently is the head of national key research and development projects and major scientific research instrument development projects. He has developed naked-eye 3D displays, 3D cameras, new liquid crystal displays, liquid crystal lenses, variable-focus liquid lenses, and continuous optical zoom microscopes; received six provincial and ministerial science and technology awards, granted over 150 patents in the US and China, published three books, and authored over 300 papers indexed by SCI.
Editor: Lemon
