Technical Articles
2025-01-18

How Image Sensors Drive the Development of Embedded Vision Technology

Click below Card, follow the “New Machine Vision” public account

Heavyweight dry goods, delivered first time

New imaging applications are booming, from collaborative robots in Industry 4.0 to drone firefighting or agriculture, to biometric facial recognition, and handheld medical devices for home care. A key factor in the emergence of these new application scenarios is that embedded vision is more prevalent than ever. Embedded vision is not a new concept; it simply defines a system that includes a vision setup that controls and processes data without an external computer. It has been widely used in industrial quality control, with the most familiar example being the “smart camera.”

In recent years, the development of economical hardware components from the consumer market has significantly reduced the bill of materials (BOM) costs and product size compared to previous computer-based solutions. For example, small system integrators (OEMs) can now purchase single-board computers or module systems like NVIDIA Jetson in small batches; larger OEMs can directly obtain image signal processors like Qualcomm Snapdragon. On the software level, available software libraries can speed up the development of dedicated vision systems, reducing configuration difficulty, even for small batch production.

The second change driving the development of embedded vision systems is the emergence of machine learning, which allows neural networks in laboratories to be trained and then uploaded directly to processors so they can automatically recognize features and make decisions in real time.

Providing solutions suitable for embedded vision systems is crucial for imaging companies targeting these high-growth applications. Image sensors play an important role in large-scale adoption because they can directly affect the performance and design of embedded vision systems, and their main driving factors can be summarized as: smaller size, weight, power consumption, and cost, abbreviated in English as “SWaP-C” (decreasing Size, Weight, Power, and Cost).

1

Reducing Costs is Crucial

The accelerator for the rapid advancement of new embedded vision applications is the price that meets market demand, while the cost of vision systems is a major constraint in achieving this requirement.

1

Saving Optical Costs

The first way to reduce the cost of vision modules is to shrink the product size for two reasons: firstly, the smaller the pixel size of the image sensor, the more chips can be manufactured from a wafer; on the other hand, sensors can use smaller, lower-cost optical components, both of which can reduce inherent costs. For example, Teledyne e2v’s Emerald 5M sensor reduces pixel size to 2.8µm, allowing S-mount (M12) lenses to be used on five-megapixel global shutter sensors, leading to direct cost savings—entry-level M12 lenses cost around $10, while larger C-mount or F-mount lenses cost 10 to 20 times more. Therefore, reducing size is an effective method to lower the cost of embedded vision systems.
For image sensor manufacturers, this reduced optical cost has another impact on design because, in general, the lower the optical cost, the less ideal the sensor’s angle of incidence. Therefore, low-cost optics require specific displacement microlenses to be designed above the pixels to compensate for distortion and focus light from wide angles.

2

Cost-Effective Sensor Interfaces

In addition to optical optimization, the choice of sensor interface also indirectly affects the cost of vision systems. The MIPI CSI-2 interface is the most suitable choice for achieving cost savings (initially developed by the MIPI Alliance for the mobile industry). It has been widely adopted by most ISPs and has begun to be adopted in the industrial market because it provides a lightweight integration from low-cost system-on-chip (SoC) or system-on-module (SOM) from companies like NXP, NVIDIA, Qualcomm, Rockchip, Intel, and others. Designing a CMOS image sensor with a MIPI CSI-2 sensor interface allows for direct data transmission from the image sensor to the host SoC or SOM of the embedded system without any adapter bridge, thus saving costs and PCB space; of course, this advantage is even more pronounced in multi-sensor-based embedded systems (like 360-degree panoramic systems).
However, these benefits come with some limitations. Currently, the widely used MIPI CSI-2 D-PHY standard in the machine vision industry relies on cost-effective flat ribbon cables, which have the drawback of limiting connection distances to 20 centimeters, which may not be optimal for remote pan-tilt setups where the sensor is far from the main processor, as is often the case in traffic monitoring or surround view applications. One solution to extend connection distances is to place additional repeater boards between the MIPI sensor board and the host processor, but this comes at the cost of miniaturization. Other solutions come not from the mobile industry but from the automotive industry: the so-called FPD-Link III and MIPI CSI-2 A-PHY standards support coaxial or differential pairs, allowing connection distances of up to 15 meters.

3

Reducing Development Costs

When investing in new products, rising development costs are often a challenge, which can cost millions of dollars in non-recurring engineering (NRE) and put pressure on time to market. For embedded vision, this pressure becomes greater because modularity (i.e., whether a product can switch to using multiple image sensors) is an important consideration for integrators. Fortunately, one-time development costs can be controlled by providing a certain degree of cross-compatibility between sensors, for example, by defining merged/shared pixel structures for stable optoelectronic performance, sharing a single frontend structure through the same optical center, and compatible PCB components (by means of size compatibility or pin compatibility), thereby accelerating evaluation, integration, and supply chains, as shown in Figure 1.
How Image Sensors Drive the Development of Embedded Vision Technology
Figure 1: Image sensor platforms can be designed to provide pin compatibility (left) or size compatibility (right) for proprietary PCB layout designs
Today, with the widespread release of so-called module and board-level solutions, the development speed of embedded vision systems is faster and more affordable. These one-stop products typically include a sensor board that can be integrated at any time, sometimes including a preprocessing chip, a mechanical front, and/or a lens interface. These solutions benefit applications by providing highly optimized sizes and standardized connectors, allowing them to connect directly to ready-made processing boards, like NVIDIA Jetson or NXP i.MX ones, without needing to design or manufacture intermediate adapter boards. By eliminating the need for PCB design and manufacturing, these module or board-level solutions not only simplify and accelerate hardware development but also significantly shorten software development time, as they are often provided with Video4Linux drivers.
Therefore, original equipment manufacturers and vision system manufacturers can skip weeks of development time to make image sensors communicate with the main processor, allowing them to focus on their unique software and overall system design. Optical modules, such as those provided by Teledyne e2v, offer complete packaging from optics to drivers to sensor boards by integrating the lens within the module, eliminating tasks related to lens assembly and testing, further advancing the development of one-stop solutions.
How Image Sensors Drive the Development of Embedded Vision Technology
Figure 2: The new module (right) allows for direct connection to ready-made processing boards (left) without designing any other adapter boards.

2

Improving Autonomous Energy Efficiency

Due to external computers hindering portable applications, devices powered by miniature batteries are the most obvious application benefiting from embedded vision. To reduce system energy consumption, image sensors now incorporate various functions that allow system designers to save energy.
From the sensor’s perspective, there are several ways to reduce the power consumption of the vision system without lowering the acquisition frame rate. The simplest way is to minimize the dynamic operation of the sensor itself at the system level by using standby or idle modes for as long as possible, thus reducing the power consumption of the sensor itself. Standby mode reduces the power consumption of the sensor to below 10% of its operational mode by turning off the emulation circuit. The idle mode can halve the power consumption and allows the sensor to restart image acquisition in microseconds.
Another energy-saving method is to adopt more advanced lithography node technology to design sensors. The smaller the technology node, the lower the voltage required to switch transistors, which reduces dynamic power consumption since power consumption is proportional to the square of the voltage. Thus, pixels produced ten years ago using 180nm technology not only reduced the size of transistors to 110nm but also lowered the voltage of digital circuits from 1.8V to 1.2V. The next generation of sensors will use 65nm technology nodes, making embedded vision applications more energy-efficient.
The last point is that by selecting the appropriate image sensor, LED lighting consumption can be reduced under certain conditions. Some systems must use active illumination, such as generating 3D maps, motion pauses, or simply using sequential pulses at specified wavelengths to enhance contrast. In such cases, reducing the noise of the image sensor in low-light environments can achieve lower power consumption. By reducing sensor noise, engineers can determine whether to reduce current intensity or the number of LED lights integrated into the embedded vision system. In other cases, when image capture and LED flashing are triggered by external events, selecting an appropriate sensor readout structure can significantly save energy. When using traditional rolling shutter sensors, the LED lights must be fully on during full-frame exposure, while global shutter sensors allow the LED lights to be activated only in certain parts of the frame. Therefore, using a global shutter sensor instead of a rolling shutter sensor when using pixel-wise correlated double sampling (CDS) can save lighting costs while still maintaining low noise levels like those used with CCD sensors in microscopes.

3

On-Chip Functions Pave the Way for Vision System Programming

Some extended concepts of embedded vision guide us to comprehensively customize image sensors to integrate all processing functions (system-on-chip) in a 3D stacked manner for optimized performance and power consumption. However, the cost of developing such products is very high, and achieving this level of integration with fully customized sensors is not entirely impossible in the long run, but we are currently in a transitional phase that includes embedding certain functions directly into the sensor to reduce computational load and speed up processing time.
For example, in barcode reading applications, Teledyne e2v has patented technology that embeds a proprietary barcode recognition algorithm into the sensor chip, which can identify the location of barcodes within each frame, allowing the image signal processor to focus only on these areas, improving data processing efficiency.
How Image Sensors Drive the Development of Embedded Vision Technology
Figure 3. Teledyne e2v Snappy five-megapixel chip automatically identifies barcode locations
Another function that reduces processing load and optimizes “good” data is Teledyne E2V’s patented fast exposure mode, which enables the sensor to automatically adjust exposure time to avoid saturation under changing lighting conditions. This feature optimizes processing time as it adapts to fluctuations in illumination within a single frame, and this rapid response minimizes the number of “bad” images that the processor needs to handle.
These functions are often specific and require a good understanding of the customer’s application. As long as there is sufficient understanding of the application, various other on-chip functions can be designed to optimize embedded vision systems.

4

Reducing Weight and Size for Minimal Application Space

Another major requirement of embedded vision systems is the ability to fit into tight spaces or be lightweight for use in handheld devices or to extend battery-powered product operating time. This is why most embedded vision systems currently use low-resolution small optical format sensors ranging from 1MP to 5MP.
Reducing the size of pixel chips is just the first step in reducing the size and weight of image sensor packages. The current 65nm process allows us to reduce global shutter pixel sizes to 2.5µm without sacrificing optoelectronic performance. This production process enables full HD global shutter CMOS image sensors to meet the mobile market’s requirement of less than 1/3 inch specifications.
Another major technology to reduce the weight and footprint of sensors is to shrink package sizes. Wafer-level packaging has rapidly grown in the market over the past few years, particularly in mobile, automotive, and medical applications. Compared to traditional ceramic (CLGA) packages commonly used in the industrial market, wafer-level fan-out packaging and chip-scale packaging can achieve higher density connections, making them excellent solutions for the lightweight and miniaturization challenges of embedded system image sensors. For Teledyne e2v’s 2MP sensor, wafer-level packaging combined with smaller pixel sizes has reduced the size to a quarter within just five years.
How Image Sensors Drive the Development of Embedded Vision Technology
Figure 4: Typical evolution of image sensor sizes since 2016, with improvements in packaging technology and pixel size reduction
Looking to the future, we expect new technologies to further achieve smaller sensor sizes required for embedded vision systems.
3D stacking is an innovative technology for semiconductor device production, which manufactures various circuit chips on different wafers and then uses copper-to-copper connections and through-silicon vias (TSV) technology for stacking and interconnection. 3D stacking allows devices to achieve smaller package sizes than traditional sensors due to its multi-layer overlapping chips. In 3D stacked image sensors, readout and processing blocks can be moved below the pixel array and row decoders. This reduces the footprint of the sensor due to the smaller readout and processing blocks and allows more processing resources to be added to reduce the load on the image signal processor.
How Image Sensors Drive the Development of Embedded Vision Technology
Figure 5: 3D chip stacking technology enables pixel arrays, emulation, and digital circuits to overlap, even adding additional specific application processing layers while reducing sensor area.
However, for 3D stacking technology to gain widespread application in the image sensor market, there are still some challenges to face. First, this is an emerging technology, and second, it has a higher cost due to the additional process steps, making chip costs more than three times higher than chips using traditional technologies. Therefore, 3D stacking will mainly be the choice for high-performance or very small package size embedded vision systems.
In summary, embedded vision systems can be summarized as a “lightweight” vision technology that can be used by various types of companies, including OEMs, system integrators, and standard camera manufacturers. “Embedded” is a generic description that can be applied to different applications, thus cannot provide a list of its features. However, there are several applicable rules for optimizing embedded vision systems, namely, generally, market drivers do not come from super-fast speeds or ultra-high sensitivity, but rather from size, weight, power consumption, and cost. Image sensors are the main drivers of these conditions, so careful selection of the appropriate image sensor is needed to optimize the overall performance of embedded vision systems.
Appropriate image sensors can provide embedded designers with more flexibility, not only saving bill of materials costs but also reducing the footprint of lighting and optical components. But more importantly than image sensors, the emergence of ready-to-apply board-level solutions in the form of imaging modules paves the way for further optimizing size, weight, power consumption, and cost, significantly reducing development costs and time with cost-acceptable, deep-learning-optimized image signal processors from the consumer market, without adding extra complexity.
Written by: Marie-Charlotte Leclerc
Source: Teledyne Imaging

This article is for academic sharing only. If there is any infringement, please contact to delete the article.

—THE END—
How Image Sensors Drive the Development of Embedded Vision Technology

Related posts

Leave a Comment

Your email address will not be published. Required fields are marked *