New imaging applications are thriving, from collaborative robots in Industry 4.0 to drone firefighting or agricultural uses, to biometric facial recognition and handheld medical devices for home care. A key factor for the emergence of these new application scenarios is that embedded vision is more prevalent than ever. Embedded vision is not a new concept; it simply defines a system that includes a vision setup that controls and processes data without an external computer. It has been widely used in industrial quality control, with familiar examples like “smart cameras.”

In recent years, the development of affordable hardware components from the consumer market has significantly reduced the bill of materials (BOM) costs and product sizes compared to previous computer-based solutions. For example, small system integrators or OEMs can now purchase small quantities of single-board computers or module systems like NVIDIA Jetson; larger OEMs can directly obtain image signal processors like Qualcomm Snapdragon. On the software side, market software libraries can accelerate the development speed of dedicated vision systems and reduce configuration difficulties, even for small batch production.

The second change driving the development of embedded vision systems is the emergence of machine learning, which allows neural networks in laboratories to be trained and then directly uploaded to processors so they can automatically recognize features and make decisions in real-time.

Providing solutions suitable for embedded vision systems is crucial for imaging companies targeting these high-growth applications. Image sensors play a significant role in large-scale introduction due to their ability to directly affect the performance and design of embedded vision systems, and their main driving factors can be summarized as: smaller size, weight, power consumption, and cost, abbreviated as “SWaP-C” (decreasing Size, Weight, Power, and Cost).

The accelerator for new applications of embedded vision is the price that meets market demand, and the cost of vision systems is a major constraint to achieving this requirement.

The first way to reduce the cost of vision modules is to minimize product size for two reasons: first, as the pixel size of image sensors decreases, more chips can be manufactured from a wafer; on the other hand, sensors can use smaller, lower-cost optical components, both of which can reduce inherent costs. For example, Teledyne e2v’s Emerald 5M sensor reduces pixel size to 2.8 µm, allowing S mount (M12) lenses to be used on five-megapixel global shutter sensors, resulting in direct cost savings—entry-level M12 lens costs about $10, while larger C or F mount lenses cost 10 to 20 times more. Therefore, reducing size is an effective way to lower the cost of embedded vision systems.

For image sensor manufacturers, this reduction in optical costs has another impact on design because, generally speaking, the lower the optical cost, the less ideal the incident angle of the sensor. Therefore, low-cost optics require designing specific displacement microlenses above the pixels to compensate for distortion and focus light from wide angles.

In addition to optical optimization, the choice of sensor interface also indirectly affects the cost of vision systems. The MIPI CSI-2 interface is a suitable choice for achieving cost savings (it was originally developed by the MIPI Alliance for the mobile industry). It has been widely adopted by most ISPs and has begun to be used in the industrial market because it provides lightweight integration from low-cost system-on-chip (SoC) or system-on-module (SOM) from companies like NXP, NVIDIA, Qualcomm, Rockchip, Intel, and others. Designing a CMOS image sensor with a MIPI CSI-2 sensor interface allows for direct data transfer from the image sensor to the host SoC or SOM of the embedded system without any bridging, thus saving costs and PCB space; of course, this advantage is even more pronounced in multi-sensor embedded systems (like 360-degree panoramic systems).

However, these benefits are limited. The widely used MIPI CSI-2D-PHY standard in the machine vision industry relies on cost-effective flat ribbon cables, which have the disadvantage of a connection distance limit of 20 centimeters, which may not be ideal for remote pan-tilt setups where the sensor is far from the main processor, as is often the case in traffic monitoring or surround view applications. One solution to extend connection distances is to place additional repeater boards between the MIPI sensor board and the host processor, but this comes at the cost of miniaturization. There are also other solutions, not from the mobile industry, but from the automotive industry: the so-called FPD-Link III and MIPI CSI-2A-PHY standards support coaxial or differential pairs that allow connection distances of up to 15 meters.

When investing in new products, the continually rising development costs are often a challenge; they can cost millions of dollars in non-recurring engineering (NRE) and put pressure on time-to-market. For embedded vision, this pressure becomes greater because modularity (i.e., whether the product can switch between various image sensors) is an important consideration for integrators. Fortunately, one-time development costs can be controlled by providing a certain degree of cross-compatibility between sensors, for example, by defining merged/shared pixel structures for stable photoelectric performance, sharing a single frontend structure through the same optical center, and compatible PCB components (by means of size compatibility or pin compatibility), thus accelerating evaluation, integration, and supply chain, as shown in Figure 1.

Figure 1: Image sensor platforms can be designed to provide pin compatibility (left) or size compatibility (right) for proprietary PCB layout designs.

Today, with the widespread release of so-called module and board-level solutions, the development speed of embedded vision systems is faster and more affordable. These one-stop products typically include a sensor board that can be integrated at any time, sometimes including a preprocessing chip, a mechanical front, and/or a lens interface. These solutions benefit applications by highly optimized sizes and standardized connectors, allowing them to connect directly to off-the-shelf processing boards such as NVIDIA Jetson or NXP i.MX ones without the need to design or manufacture intermediate adapter boards.

By eliminating the need for PCB design and manufacturing, these module or board-level solutions not only simplify and accelerate hardware development but also greatly shorten software development time, as they are often provided with Video4Linux drivers. As a result, original equipment manufacturers and vision system manufacturers can skip weeks of development time to make the image sensor communicate with the main processor, allowing them to focus on their unique software and overall system design. Optical modules, such as those provided by Teledyne e2v, offer a complete package from optics to drivers to sensor boards by integrating the lens within the module, further driving the development of one-stop solutions.

Figure 2: New modules (right) allow for direct connection to off-the-shelf processing boards (left) without designing any additional adapter boards.

Devices powered by micro batteries benefit from embedded vision, as external computers hinder portable applications. To reduce system power consumption, image sensors now incorporate multiple functions that allow system designers to save energy.

From the sensor’s perspective, there are various ways to reduce the power consumption of vision systems without compromising frame acquisition rates. The simplest method is to minimize the dynamic operation of the sensor itself at the system level by using standby or idle modes for as long as possible, thereby reducing the power consumption of the sensor itself. Standby mode reduces the sensor’s power consumption to below 10% of the working mode by shutting down simulation circuits. Idle mode can halve power consumption and allows the sensor to restart to capture images in a few microseconds.

Another energy-saving method is to use more advanced lithography node technology to design sensors. The smaller the technology node, the lower the voltage required to switch transistors, which reduces dynamic power consumption since power consumption is proportional to the square of the voltage: P_dynamic∝C×V². Therefore, pixels produced using 180 nm technology ten years ago not only reduced transistors to 110 nm but also lowered the voltage of digital circuits from 1.8V to 1.2V. The next generation of sensors will use 65nm technology nodes, making embedded vision applications more energy-efficient.

Lastly, by selecting the appropriate image sensor, LED light consumption can be reduced under certain conditions. Some systems must use active lighting, such as generating 3D maps, motion pauses, or simply using sequential pulses of specified wavelengths to enhance contrast. In these cases, reducing the image sensor’s noise in low-light environments can achieve lower power consumption. By reducing sensor noise, engineers can determine whether to reduce current intensity or the number of LEDs integrated into the embedded vision system. In other cases, when image capture and LED flashing are triggered by external events, choosing the right sensor readout structure can significantly save energy. When using traditional rolling shutter sensors, the LED lights must be fully on during full-frame exposure, while global shutter sensors allow the LED lights to be activated only in certain parts of the frame. Therefore, using a global shutter sensor instead of a rolling shutter sensor when using pixel-wise correlated double sampling (CDS) can save lighting costs while maintaining low noise similar to CCD sensors used in microscopes.

Some extended concepts of embedded vision guide us to fully customize image sensors, integrating all processing functions (system-on-chip) in a 3D stacked manner for optimized performance and power consumption. However, the cost of developing such products is very high, and achieving this level of integration with fully customized sensors is not entirely impossible in the long term; we are currently in a transitional phase, which includes embedding certain functions directly into the sensor to reduce computational load and accelerate processing time.

Figure 3: Teledyne e2v SNAPPY five-megapixel chip automatically identifies barcode locations.

Another feature that reduces processing load and optimizes “good” data is Teledyne e2v’s patented fast exposure mode, which allows the sensor to automatically adjust exposure time to avoid saturation under varying lighting conditions. This feature optimizes processing time as it adapts to fluctuations in lighting within a single frame, and this rapid response minimizes the number of “bad” images the processor needs to handle.

These features are often specific and require a good understanding of the customer’s application. As long as there is sufficient understanding of the application, various other on-chip functions can be designed to optimize embedded vision systems.

Another major requirement for embedded vision systems is the ability to fit into tight spaces or to be lightweight, to facilitate handheld devices extending product working time. This is why most embedded vision systems currently use low-resolution small target sensors ranging from 1MP to 5MP.

Reducing the size of pixel chips is just the first step in reducing the size and weight of image sensor packages. The current 65 nm process allows us to reduce the global shutter pixel size to 2.5µm without compromising photoelectric performance. This manufacturing process enables full HD global shutter CMOS image sensors to meet mobile market requirements of less than 1/3 inch specifications.

Another major technology for reducing sensor weight and footprint is shrinking package size. Wafer-level packaging has rapidly grown in the market over the past few years, particularly in mobile devices, automotive, and medical applications. Compared to traditional ceramic (CLGA) packages commonly used in the industrial market, wafer-level fan-out packaging and chip-level packaging can achieve higher density connections, making them excellent solutions for lightweight and miniaturized embedded system image sensors. For Teledyne e2v’s 2-megapixel sensor, wafer-level packaging combined with smaller pixel sizes has been able to reduce size to a quarter within five years.

Figure 4: Typical evolution of image sensor size since 2016 with improvements in packaging technology and reduced pixel sizes.

In summary, embedded vision systems can be summarized as a “lightweight” vision technology that can be used by various types of companies, including OEMs, system integrators, and standard camera manufacturers. “Embedded” is a generic description that can be used for different applications, so it cannot provide a list of its characteristics. However, there are several applicable rules for optimizing embedded vision systems: generally speaking, market drivers do not come from super-fast speeds or ultra-high sensitivity but from size, weight, power consumption, and cost. Image sensors are the main drivers of these conditions, so careful selection of suitable image sensors is required to optimize the overall performance of embedded vision systems.

Suitable image sensors can provide more flexibility for embedded designers, not only saving bill of materials costs but also reducing the footprint of lighting and optical components. But more importantly than image sensors, the emergence of ready-to-apply board-level solutions in the form of imaging modules paves the way for further optimization of size, weight, power consumption, and cost, significantly reducing development costs and time through cost-effective, deep-learning-optimized image signal processors from the consumer market, without adding extra complexity.