Banner | Promotion
Near Eye Display (Near eye display, NED) or Head-Mounted Display (Head mounted display, HMD) is a basic device for achieving Virtual Reality (Virtual reality, VR) and Augmented Reality (Augmented Reality, AR), providing immersive and interactive experiences that seamlessly integrate the digital world with the physical world, and is expected to become the next generation of augmented reality display terminals. Currently, a key issue facing NED is the convergence-accommodation conflict, which can lead to eye fatigue and discomfort during prolonged viewing.Retinal Projection Display (Retinal Projection Displays, RPD) technology features always in focus (panoramic focus) images, naturally resolving the convergence-accommodation conflict, making it a research hotspot in the near-eye display field.
Compared to traditional direct-view displays, RPD has many advantages due to its unique principle of directly imaging the source of the image into the pupil of the human eye. First, RPD gathers nearly all the light into the human eye, resulting in high optical efficiency, especially suitable for outdoor use. Second, by increasing the numerical aperture of the eyepiece, a large field of view can be directly obtained for near-eye displays. Third, the panoramic focus feature of RPD naturally resolves the convergence-accommodation conflict present in near-eye displays, enabling AR near-eye displays without visual fatigue. In some AR applications, such as vehicle-assisted driving, drivers can clearly see virtual images when focusing their eyes at different distances outside the vehicle, avoiding the risks caused by back-and-forth focusing between the external road conditions and virtual images, thus enhancing driving safety.
Recently, Associate Researcher Wang Zi and Professor Lv Guoqiang from Hefei University of Technology published a review article titled “Research Progress on Retinal Projection Display Technology” in the 5th issue of Liquid Crystal and Display (ESCI, Scopus indexed, a core journal in Chinese) in 2022.
This article reviews the development of RPD technology, explains the basic working principles of RPD, summarizes the latest advancements in RPD and its exit pupil expansion, and discusses its future prospects.
1. Retinal Projection Display (RPD) Technology and Its Principles
RPD was first introduced in an experiment conducted by physicist Maxwell in 1860. He directly imaged a light source through a lens onto the pupil, observing that the lens was uniformly illuminated by the light. This method of imaging the light source onto the pupil is known as the Maxwellian viewing method. Based on this technology, Kollin et al. incorporated an image source and developed the first prototype of an RPD display. Subsequently, RPD technology has garnered widespread attention from researchers.
Fig.1: (a) SLM-based retinal projection display; (b) LBS-based retinal projection display.
Image source: Liquid Crystal and Display, 2022, 37(5): 640. Fig.1
Figure 1 shows the basic principle diagram of traditional RPD. In Figure 1 (a), a point light source is collimated by a lens, parallelly illuminating a Spatial Light Modulator (Spatial light modulator, SLM) loaded with digital image information, converging into a light point at the pupil of the human eye through the eyepiece, directly projecting onto the human retina without being affected by the eye’s accommodation. The parallel light source imaging at the pupil produces an exit pupil diameter smaller than the pupil diameter, significantly increasing the depth of field of the near-eye display system.
Figure 1 (b) shows another RPD display principle using the LBS (Laser beam scanning, LBS) method. A Micro-Electro-Mechanical System (Micro-Electro-Mechanical System, MEMS) scanning mirror performs two-dimensional directional scanning deflection of the laser beam while synchronously modulating the intensity of the laser beam, loading image information to achieve laser beam scanning projection. The eyepiece then converges the light point at the pupil of the human eye to realize retinal projection display. The LBS method utilizes the high collimation of lasers to achieve a small exit pupil diameter, making it an active display method compared to the passive optical system in Figure 1 (a).
Due to the light being converged to a point, an excessively small eyebox size means that the human eye’s pupil must be precisely located on the light point to receive the image, and a slight shift will cause the image to disappear. Therefore, exit pupil expansion is a key issue that RPD needs to address. Regarding this issue, scholars at home and abroad have proposed various solutions, mainly divided into two categories: geometric optics and diffraction optics.
2. RPD Based on Geometric Optics
Geometric optics-based RPD uses geometric optical elements such as lenses to converge the light beam into the pupil of the human eye. The exit pupil expansion methods of geometric optics-based RPD can be divided into viewpoint duplication and viewpoint steering.
Fig.2: Retinal projection exit pupil expansion; (a) beam splitter array; (b) mechanical steering mirror; (c) LED array
Image source: Liquid Crystal and Display, 2022, 37(5): 641. Fig.2
Viewpoint duplication generates multiple viewpoints to cover a larger eye movement range. The beam splitter array method shown in Figure 2 (a) creates a 3×3 viewpoint array by multiple reflections of the light path to expand the exit pupil. This viewpoint duplication method is simple and effective, but it requires special attention to the matching between the viewpoint spacing and the pupil diameter. As shown in Figure 3, if the viewpoint spacing is smaller than the pupil diameter, it will lead to crosstalk between viewpoints; if the spacing is larger than the pupil diameter, it will result in image loss. Additionally, since the pupil diameter of the human eye changes with the intensity of ambient light, the design of viewpoint spacing also faces challenges. Moreover, the more viewpoints duplicated, the lower the brightness of each viewpoint image.
Fig.3: (a) Viewpoint spacing too small leads to crosstalk; (b) viewpoint spacing too large leads to image loss
Image source: Liquid Crystal and Display, 2022, 37(5): 641. Fig.3
In contrast, as shown in Figure 2 (b), the viewpoint steering method dynamically changes the position of the viewpoint based on the position of the pupil. Figure 2 (c) shows a method using an array of Light-Emitting Diodes (Light-emitting diode, LED) to implement viewpoint steering, dynamically controlling different LED positions to generate viewpoints at different locations on the pupil surface according to pupil tracking information, thus expanding the exit pupil. On the other hand, the LED array scheme can also achieve viewpoint duplication. By rapidly switching LED light sources and synchronously refreshing the images loaded on the SLM, viewpoint duplication can be realized.
3. RPD Based on Holographic Optical Elements
Using geometric lenses is not conducive to thinning the optical system, while Holographic Optical Elements (Holographic optical element, HOE) as an off-axis optical combiner can effectively address the thinning issue. As shown in Figure 4, a lens HOE prepared through optical interference serves as a planar optical element, simultaneously providing focusing and reflecting optical effects. Additionally, due to the wavelength selectivity of the Bragg grating, the transmittance of the lens HOE for ambient light approaches 100%. By using lens HOE to converge the beams of image sources such as LBS and LCOS, a compact and high optical efficiency RPD system can be achieved.
Fig.4: (a) Lens HOE converging LBS image source beams; (b) lens HOE converging SLM image source beams
Image source: Liquid Crystal and Display, 2022, 37(5): 642. Fig.4
Similar to the exit pupil expansion of the aforementioned geometric lens RPD, in HOE-based RPD, as shown in Figure 5 (a), lens array HOE generates point light source arrays to duplicate viewpoints, or mechanical steering mirrors are commonly used to deflect viewpoints. Additionally, as shown in Figure 5 (b), the angle multiplexing characteristic of HOE provides another possible solution, recording multiple converging beams into the same HOE, thereby converging the signal light to three different viewpoints.
Fig.5: (a) Point light source array generated by lens array HOE; (b) multiple converging beams recorded in the same HOE
Image source: Liquid Crystal and Display, 2022, 37(5): 642. Fig.5
In recent years, polarization-based volume grating devices (PVG) or liquid crystal HOE (LCHOE) have also been used to achieve exit pupil expansion for RPD. Unlike traditional HOE that records the intensity of the interference beam, due to the inherent anisotropic properties of liquid crystals, PVG is very sensitive to the polarization state of the signal light. This characteristic, combined with the polarization modulation capability of liquid crystal devices, provides new possibilities for exit pupil expansion in RPD systems. As shown in Figure 6 (a), the polarization selection characteristic of reflective liquid crystal holographic optical elements (LCHOEs) can dynamically switch the incident light’s polarization state by controlling the polarization converter (PC), allowing left-handed LCHOE and right-handed LCHOE to function separately, thus achieving dynamic switching of the RPD viewpoint position. As shown in Figure 6 (b), based on polarization selectivity, using a transmissive polarization grating (PG) device and a polarization converter to dynamically switch the light beam direction in conjunction with HOE can produce two sets of switchable viewpoints, alleviating the potential crosstalk and image loss that may occur with traditional HOE viewpoint duplication.
Fig.6: (a) Reflective PVG; (b) Transmissive PVG
Image source: Liquid Crystal and Display, 2022, 37(5): 643. Fig.6
4. Holographic RPD Based on Wavefront Modulation
In addition to traditional two-dimensional image sources, as shown in Figures 7 (a) and (b), utilizing the holographic wavefront modulation capability to generate three-dimensional image sources and converging them to the human eye via lenses or HOE can achieve holographic RPD near-eye display with depth perception. As shown in Figure 7 (c), by combining multiplexing coding technology, it is possible to achieve color dynamic holographic RPD using only a 60 Hz spatial light modulator.
Fig.7: (a)(b) RPD of holographic image sources; (c) multiplex coding technology achieves color RPD
Image source: (a) ACM Transactions on Graphics (Tog), 2017,36(4): 1-16. Fig14; (b) SID Symposium Digest of Technical Papers. Oxford, UK: Blackwell Publishing Ltd, 2011,42(1): 591-594. Fig1; (c) Opt. Express, 2021,29, 8098-8107. Fig.1
As shown in Figure 8, holographic RPD can easily achieve exit pupil expansion by adding different plane carriers to the phase hologram and using lenses to converge the reconstructed image to different viewpoint positions.
Fig.8: Principle of exit pupil expansion using phase hologram: (a) perspective view; (b) top view
Image source: Opt. Lett. 2022, 47: 445-448. Fig.1
Figure 9 demonstrates the use of conjugate light of amplitude holograms to expand the exit pupil of holographic RPD, transforming traditionally useless conjugate light interference into a viewpoint array, achieving double viewpoint duplication.
Fig.9: Conjugate light term coding to achieve exit pupil expansion in holographic retinal projection
Image source: Optics Letters, 2021,46(22): 5623-5626. Fig.5
Fig.10: (a) Holographic RPD principle (b) Holographic RPD exit pupil expansion using multi-spherical wave coding
Image source: Liquid Crystal and Display, 2022, 37(5): 643. Fig.7
Figure 10 (a) shows a novel lens-free wavefront-controlled holographic RPD method, where the target image is treated as amplitude, multiplied by the converging spherical wave phase, and then through Fresnel diffraction and introducing reference light interference, the final amplitude hologram is obtained. This method eliminates the use of lenses and directly implements retinal projection through SLM wavefront modulation.
By combining eye tracking, the three-dimensional coordinates of viewpoints can be freely and precisely manipulated through encoding the spherical wave phase, offering advantages such as lens-free aberration, high system degrees of freedom, and low computational resource requirements. Additionally, since spherical wave phases replace traditional random phases, speckle noise is well suppressed. The flexible wavefront control characteristics of lens-free holographic RPD can also easily achieve viewpoint duplication. As shown in Figure 10 (b), by multiplying several spherical wave phases of different directions, the light can be converged to multiple viewpoints.
It can be seen that, unlike traditional viewpoint duplication methods, the lens-free holographic RPD utilizes wavefront coding to achieve beam convergence, deflection, and duplication, allowing flexible control over the three-dimensional coordinates, quantity, and spacing of viewpoints to match changes in pupil location and size, effectively addressing issues of viewpoint crosstalk and image loss.
By encoding three-color RPD wavefront information into a single hologram, low speckle noise color dynamic lens-free holographic RPD display and lateral exit pupil expansion can be achieved. Figure 11 (a) shows a color reconstructed image with AR effects, clearly visible at different depths. As shown in Figure 11 (b), encoding multiple virtual image sources into a single hologram can achieve full-color dynamic multi-channel holographic near-eye displays, allowing users to quickly switch between different video channels simply by rotating their eyes.
Additionally, the lens-free holographic RPD with super multi-view (super multi view, SMV) display characteristics can converge multiple parallax images multiplied by corresponding spherical waves into the pupil, providing monocular depth cues. Figure 11 (c) shows the reconstructed results at different depths.
Fig.11 (a) Color reconstructed image with AR effects generated by spherical wave-based holographic RPD; (b) Full-color dynamic multi-channel holographic near-eye RPD; (c) Holographic super multi-view RPD reconstruction results
Image source: (a) Optics Letters, 2021, 46(17): 4112-4115; (b) Optics Letters, in review; (c) Optics Letters, 2022, 47(10): 2530-2533.
5. Summary and Outlook
RPD near-eye displays feature high optical efficiency, large fields of view, and their panoramic focus characteristics naturally resolve the convergence-accommodation conflict, enabling AR near-eye displays without visual fatigue. For the exit pupil expansion issue, geometric optics methods often employ point light source arrays and mechanical steering mirrors to achieve viewpoint duplication or deflection, but with a certain level of complexity.
The unique angle and polarization multiplexing characteristics of HOE help achieve lightweight RPD systems with large fields of view and large exit pupils. Holographic RPD directly implements retinal projection through SLM wavefront modulation, allowing free and precise control over viewpoints. However, the exit pupil and field angle of holographic RPD systems are still limited by the SLM devices. In the future, by combining the advantages of holographic wavefront control and HOE, it is expected to achieve lightweight RPD near-eye displays with large fields of view, large exit pupils, and high degrees of system freedom.
Paper Information
Zhang Xu, Wang Zi, Tu Kefeng, Chen Tao, Pang Yujian, Lv Guoqiang, Feng Qibin. Research Progress on Retinal Projection Display Technology [J]. Liquid Crystal and Display, 2022, 37(5):639-646.
https://cjlcd.lightpublishing.cn/thesisDetails#10.37188/CJLCD.2022-0040
Corresponding Author Profile 
Wang Zi, Associate Researcher at Hefei University of Technology, Master’s supervisor. He obtained a Bachelor’s and Doctorate in Physics from the University of Science and Technology of China in 2012 and 2017, respectively, and is primarily engaged in research on computational holography, 3D displays, and near-eye displays. He has undertaken multiple research projects including the National Natural Science Foundation, National Engineering Technology Research Center projects, and major projects in Anhui Province. He has published over 30 SCI academic papers in renowned journals such as Optics Letters, Optics Express, Applied Physics Letters, etc.
E-mail: [email protected]
Supervised by | Zhang Ying, Zhao Yang
Edited by | Zhao Wei
Welcome to Submit to the Research Group – Press Release
Reprint/Cooperation/Research Group Submission, WeChat:447882024
Banner Promotion
Click Here 👇 Follow Us
Click[Looking], to receive our content updates in a timely manner