1. Introduction The concept of 3D imaging technology has existed for decades, but the commercialization of products began after 2000, when many major film studios announced the use of the latest HD cameras to achieve 3D imaging effects. Since then, not only in the consumer market but also in the field of industrial machine vision, 3D imaging technology has made leaps in speed, error accuracy, and resolution. With the rise of “Industry 4.0”, traditional 2D technology is very limited in terms of error accuracy and distance measurement, especially in complex object recognition, size measurement, and interactive applications, such as human-computer interaction and the increasing demand for 3D vision. In the field of factory automation, 3D vision can enhance the autonomy and effectiveness of robots/machine systems. In fact, 3D vision is essential for higher precision quality inspection, reverse engineering, or size detection of objects limited by 2D vision. Additionally, the use of vision system-guided robotics technology is growing, requiring better remote guidance, obstacle recognition, and accurate movement through 3D vision. 3D vision also protects workers in factories from obstructions in dense human-computer interaction systems and solves hazardous situations, while also counting and recognizing natural persons and robots through monitoring systems. 3D vision permeates all aspects of society, providing end-users with safer, higher performance, and more effective assistance systems. For example, 3D vision is applied in advanced automotive autonomous driving systems, autonomous vehicles, collaborative robots…
This article aims to explore 3D imaging technology and the advanced embedded CMOS sensor technology in industrial 3D applications.
2. Overview of 3D Imaging Currently, users can obtain 3D through several technologies and solutions. The following sections aim to briefly outline each technology along with its advantages and disadvantages.
2.1 Stereo Vision Technique: Similar to how the human brain controls distance perception, two cameras are installed at the same location but at different angles, using visual disparity and calibration techniques to extract depth information of objects.
Figure 1 Stereo Vision System Principle
2.2 Structured Light Technique: Composed of a camera and a projector. The projector casts striped light onto the object, and the camera captures the striped light. The depth information is calculated based on the distortion of the stripes on the object.
Figure 2 Structured Light System Principle
2.3 Laser Triangulation: A laser triangulation system consists of a 2D camera, lens, and laser. It determines the position of the measured object by measuring the light reflected from the object’s surface. The laser projects a light spot onto the object, and the reflected light is focused onto a photosensitive device through the imaging camera. Depending on the distance of the laser relative to the object’s surface, the laser point appears at different positions within the camera’s field of view. This technique is called triangulation, forming a triangle with the laser point, camera, and laser emitter. High-resolution lasers are typically used in applications requiring high precision, stability, and low temperature drift for displacement and position monitoring.
Figure 3 Laser Triangulation Principle
2.4 Time-of-Flight Principle: The ToF principle utilizes the synchronization of the light source and image sensor, calculating distance based on the time difference between the emitted light pulse and the light returning to the image sensor.
Figure 4 Time-of-Flight System Principle
2.5 Comparison of 3D Imaging Technologies: As mentioned above, each technology has its advantages and disadvantages. Therefore, it is important to choose the most suitable solution based on the application, especially regarding distance range and depth accuracy requirements. A relative comparison is made in Table 1.
Table 1 Comparison of 3D Imaging Technologies
Currently, 3D imaging systems mainly rely on stereo vision, structured light, and laser triangulation solutions. These systems are primarily used at fixed working distances and require a high level of calibration in specific detection fields. Time-of-Flight (ToF) systems can overcome these difficulties and are more flexible from an application perspective, but they are currently limited by resolution.
3. Unique 3D Imaging Solutions from Teledyne e2v in the Industrial Market
Based on the successful accumulation of line scan cameras and area array sensors in the machine vision field, Teledyne e2v is building a unique platform dedicated to 3D imaging to support the latest industrial applications, such as vision-guided robots, automated guided vehicles (AGV), factory monitoring and security, handheld scanners, and outdoor applications. Teledyne e2v aims to provide a variety of 3D imaging technologies and solutions to meet customer application needs.
Teledyne e2v mainly offers the following three 3D imaging solutions: • Laser Triangulation • Stereo Vision and Structured Light • Time of Flight Technology
3.1 Laser Triangulation Solution
Teledyne e2v can provide standard products and also special custom high-speed sensors: • A wide range of selectable resolutions: 2k to 8k points per profile line • High frame rate sensors: up to 20k fps • High sensitivity, low noise: QE>60%, SNR of 43dB • Multiple high dynamic range performance • Unique on-chip processing technology: digital processing on the readout channel, on-chip sub-pixel peak calculation • Maximizing channel capacity in fast recording: up to 32 Gbps LVDS interface
3.2 Stereo Vision and Structured Light Systems
Teledyne e2v provides standard CMOS sensors with the following key features:
• High level SWAP-C (size, weight, power, and cost): small optical size, multiple packaging methods to optimize size and cost, low power consumption 200mW @ full frame; • Sapphire/Ruby series: WVGA, 1.3 & 2MP, 1/1.8” • Emerald series: 1.3 million pixels to 16 million pixels, pixel size 2.8µm (where the 16MP chip is 1” optical size, 8MP is 2/3” – unique worldwide)
• Visible light and near-infrared, high sensitivity versions: each chip series shares the same platform – pin-to-pin compatible
• Able to capture 3D images from fast-moving objects: up to 125 fps, high global shutter efficiency, high-speed dual shutter;
• Image sensor consistency: each sensor series shares a platform – pixel-to-pixel compatible;
• 3D processing: multi-ROI (multi-ROI function, 4 windows), binning, histogram of image data generated directly by the sensor chip.
High global shutter efficiency is applied to fast-moving objects.
Teledyne e2v has been committed to the innovation and development of high-efficiency global shutter image sensors, which can avoid artifacts in capturing fast-moving objects. When capturing moving objects, this phenomenon is likely to occur when the integration time is much shorter than the readout time, as the upper part of the image will begin integration before the lower part; and between these intervals, the object has already moved.
This requires pixel memory storage elements to store charges after integration until they are read out. This technology is based on a 5T structure. The downside is a reduced fill factor, and limited methods for reducing CDS-type temporal noise. Due to these two points, under G/S, the SNR is slightly worse compared to 4T ERS for a given pixel size.
However, using pinned junctions in the pixel storage register reduces temporal noise, while there is no longer a gap between dark current, fixed noise, and SNR. Considering fill factor, advancements and optimizations of micro-lenses on the pixel (such as zero gap) can achieve significant improvements in sensitivity to wavelength response.
Near-infrared high sensitivity chips are used for high-precision detection.
Teledyne e2v standard CMOS chips have high quantum efficiency (QE). For example, in the following image, the QE of Teledyne e2v’s Sapphire and Ruby series; Sapphire is mainly used for visible light, Ruby has enhancements and optimizations in both visible light and near-infrared. Under 5T technology, pixel sizes of 5.3µm x 5.3µm, and QE of up to 80% can be considered top-level CMOS chips.
Figure 5 QE versus wavelength response of new global-shutter CMOS imagers. In summary of these innovative technology segments, Teledyne e2v standard CMOS chips can be widely used for high-precision 3D detection of fast-moving objects in visible and near-infrared light under day/night and outdoor conditions.
3.3 Time-of-Flight Based 3D Imaging Technology Solution: Teledyne e2v launched the first 3D ToF solution, based on a 1.3 million pixel, 1-inch optical size, high sensitivity, and high dynamic range CMOS chip. • 1.3 million pixels: depth map under full frame, accuracy < 3cm under full frame, high-speed dual shutter; • Capturing 3D images of fast-moving objects: up to 120 fps, full frame depth map at 30 fps, high global shutter efficiency; • Large range 3D detection: high near-infrared sensitivity, high dynamic range 60DB, detection range [0.5m to 5m]; • High sensitivity chips in visible light and near-infrared: 50% QE at 850 nm, high dynamic range mode: low light/day and night; • Embedded 3D processing: Multi-ROI (2 windows), Binning, embedded on-chip image processing.
The ToF technology based on 5T CMOS sensors for active imaging is an imaging method realized by synchronizing control of the light source. For example, the infrared signal used in our smartphones helps to determine the distance under low light conditions for assisted autofocus. Active imaging is mainly applied to capture images or 3D images in harsh weather conditions (fog). Range gating imaging and Time-of-Flight (ToF) are two main types of active imaging technologies. Range gating technology has two requirements: pulse wavefront and high-speed shutter camera. When the reflection returns from the reflective plane, light is sent towards the target, and the camera’s high-speed electronic shutter opens at the right moment. After the light pulse travels a certain distance to the object, it will be partially reflected and received by the camera. This requires precise control of the camera’s high-speed shutter so that the camera sensor can just open a shutter when the pulse returns, receiving the measurement signal. By synchronizing the settings of the light source and the camera, the desired distance for image acquisition can be selected. Under harsh testing conditions, such as rainy days, foggy days, or strongly scattering media, only a small portion of photons can pass through the medium and return to the camera. Range gating imaging is an effective method to overcome harsh testing conditions and improve system signal-to-noise ratio. ToF directly measures the flight time of light to infer the distance of the reflective plane. Unlike range gating technology, ToF does not select the image plane, allowing direct distance imaging. As shown in Figure 6, the implementation of distance gating image capture is based on a synchronized camera source light system. The camera has a very fast global shutter, in the microsecond range. At T0, the camera sends a trigger to the synchronized light source to emit a pulse. After a period of time T1, the light pulse reaches the preset measurement distance range, and if an object exists within that range, it will be reflected; otherwise, no reflection signal will occur. In the case of reflection, the light returns to the camera at T2. At T3 = T0 + 2τ, the camera’s shutter opens and captures the reflected light. This process is repeated to accumulate enough signal. The generated image corresponds only to objects present within that range. If a depth image is needed, a series of images of distance gating patterns must be scanned at several depths, or delays can be adjusted. The distances are then calculated from this set of images.
Figure 6 Schematic Diagram of Range Gating Technology. Figure 7 shows the working principle of the global shutter image sensor pixel. Signal integration is not achieved in one go, but by continuously accumulating enough detected photons through multiple micro-exposures (micro-integration) to achieve range gating imaging or ToF imaging.
Figure 7 Working Principle of Global Shutter Image Sensor Pixel. Teledyne e2v patented technology, based on T5 pixels, generates timing on alternating lines to achieve a Δt of about 10 nanoseconds. This means a significant improvement in time resolution. Teledyne e2v’s 1.3 million pixel CMOS image sensor has high sensitivity/low noise, including multiple integration or “stacking” modes. Additionally, pinned photodiodes and storage registers require a high parasitic light sensitivity ratio (PLS), also known as extinction ratio, to reduce background noise generated by the scene to produce sharp images. This is achieved by reducing parasitic light or crosstalk during the camera shutter “off” period.
Figure 8 CMOS Sensor with Timing and Synchronization Circuits Requires Adequate Extinction Ratio to Reduce Background Noise.
Teledyne e2v BORA Image Sensor – Leading Art-Grade 1.3 Million Pixel Time-of-Flight (ToF). Teledyne e2v’s next-generation TOF 1.3 million pixel image sensor, named BORA, features electronic global shutter optimized for multi-integration mode, thus achieving superior detection performance in low-light environments over short distances and ranges; this TOF CMOS chip, with a 3D depth of 1.3 million pixels and a 1-inch optical size, is dedicated to obtaining more information and higher precision in industrial applications, just like the first-generation ToF.
Table 2 Comparison Table of Teledyne e2v ToF Demonstration Systems
4. Conclusion To improve the effectiveness and autonomy of industrial systems, the demand for vision systems that guide robots and machines is growing; some applications require 3D vision (object recognition, precision). These technologies are also demanding, requiring high-performance sensors with complex features. Teledyne e2v provides a wide range of unique 3D vision solutions for the industrial market, such as factory automation, logistics, measurement applications, etc. Our unique expertise in high-performance CMOS image sensors not only provides standard 3D solutions but also offers custom services capable of achieving 3D precise detection from fast-moving objects under day/night operating conditions, thus overcoming the current challenges of 3D vision.
If you wish to purchase the Time-of-Flight (ToF) 3D imaging demonstration board/development board, please email wangyi#memsconsulting.com (replace # with @), or call 17898818163.
Recommended Conference: On September 11, 2017, the “Micro Words and Great Meaning” seminar on 3D camera technology and applications, hosted by MEMS Consulting, will be grandly held in Shanghai (concurrent exhibition: 2017 China (Shanghai) Sensor Technology and Application Exhibition). This seminar will cover topics such as 3D camera applications and market analysis, 3D camera principles and technology routes, 3D camera module analysis and algorithm analysis, etc. Companies such as Intel, Viavi Solutions, Espros, Turing Technology, Juyou Intelligent, Xi’an Zhiwei Sensor, and Aipuke have been invited to give speeches. If your company wishes to participate in speeches or exhibitions, please contact: Contact person: Wang Yi Email: [email protected] Phone: 17898818163 Seminar details and registration: www.MEMSeminar.com
Further Reading: “3D Imaging and Sensing – 2017 Edition”
“Infrared LED and Laser Technology, Applications and Industry Trends”
“Intel RealSense 3D Camera and STMicroelectronics Infrared Laser Emitter” “Lenovo Phab 2 Pro 3D Time-of-Flight (ToF) Camera” “Apple iPhone 7 Plus Rear Dual Camera Module”
