The photography capability of mobile phones is an important reference factor for many users when purchasing a phone. Generally speaking, people tend to pay more attention to parameters such as pixel count, aperture size, and even CMOS area when considering a phone’s photography capabilities. In fact, the DSP processor inside the phone also silently contributes to mobile photography, and to a large extent, the processing power of the DSP determines the aesthetic quality of the photos taken by the phone. So, what exactly is DSP? What capabilities do high-end DSPs on the market currently have? This article will take you through it.
When it comes to mobile photography, most people will instinctively compare pixels, apertures, and other parameters. In reality, there is another important component in mobile phones that plays a crucial role in photography effects, which is the DSP. DSP stands for Digital Signal Processor, which translates to a digital signal processor. So what is DSP, and what role does it play in mobile photography?
The Hidden Functions Behind the Scenes — DSP
In people’s impression, taking photos with a mobile phone is a simple task. We just need to open the camera app, frame the shot, and press the shutter button to complete the operation, which may take less than five seconds. However, behind this simplicity lies a complex process that is often unimaginable. If we exclude the operation and focusing parts, the process of how the phone processes image information after pressing the shutter button can be simply divided into the following four steps:
The first step is CMOS collecting light signals. When people pick up their phone and open the camera app, the CMOS starts collecting light information. The CMOS converts the intensity of light into analog electrical signals, which are highly correlated with the light information.
▲The raw signal data collected by CMOS is vastly different from the final image we see.
The second step is that the CMOS converts the electrical signals into digital signals using an analog-to-digital converter (ADC). Since ordinary analog signals cannot undergo further calculations, they need to be converted into digital signals. Some CMOS may perform preliminary processing on the light information at this time. The ISP, or Image Signal Processor, also processes the image in this step, including noise reduction, sharpening, color temperature adjustment, color correction, white balance, and exposure compensation.
The third step is that the DSP processes the digital signals. The DSP continues to adjust the image based on algorithms and primarily performs complex calculations. For example, it further optimizes lighting effects, reduces noise and adjusts exposure in special scenes like night shots, compensates for backlighting, and uses optimization algorithms to further adjust the image’s color and details. The DSP can also perform complex processes, including image content recognition, lens distortion correction, HDR composition, etc. In terms of data processing, the DSP can also handle data compression and encoding/decoding, including JPEG encoding/decoding and video encoding/decoding.
▲DSP is a dedicated device for processing digital signals; the image shows a common audio DSP chip.
The final step is that the CPU further processes the image based on existing algorithms. The CPU then outputs the adjusted image to memory and displays it on the screen through components like the VPU, while also storing the image in the phone’s memory for the user to view and use later.
▲The architecture of DSP is significantly different from that of CPU and GPU; the image shows a photo of a DSP chip.
Through the simplified description of the process, we can see that the DSP plays a crucial role in image processing, serving as a vital link in managing and correcting the output of images after the ISP. It can be said that without the DSP, the image quality seen by users may be greatly compromised.
Moreover, due to architectural advantages, DSPs can provide a much higher energy efficiency ratio compared to CPUs, not only improving the processing speed of mobile phones but also significantly saving energy, achieving a balance between performance and power consumption.
I Pretend to Be Unfeeling, But I Actually Hate My Own Affection
After reading the previous text, everyone should have a certain understanding of the tasks performed by DSPs. So what advantages does DSP have that make it so valued by manufacturers?
From the development of DSPs, early DSPs merely performed preliminary processing of digital signals, such as filtering and screening data, without requiring high precision or powerful computing capabilities. However, as calculations have become increasingly complex and applications have gradually increased, DSPs have become important data processing components in the eyes of professionals due to their unique architectural advantages. From an application perspective, DSP has two very prominent advantages:
First, DSPs are designed specifically for digital processing, so their architecture is oriented towards fixed processing methods, maximizing the work done per cycle. In the current mobile computing architecture, the architecture of DSPs differs from that of GPUs and CPUs. DSPs need to handle a large amount of parallel data, and the methods for executing data are relatively simple; therefore, in circuit design, DSPs can complete massive data calculations quickly and efficiently using fixed processing methods, commonly seen in fast Fourier transforms, numerous accumulate-and-add operations, and convolution calculations.
▲A DSP architecture diagram, showing that DSP’s tasks are mostly related to digital processing, requiring many high-efficiency fixed unit modules.
Generally speaking, fixed processing pipelines are much more efficient than programmable pipelines. DSPs face relatively few digital signal processing algorithms, and fixed algorithms can be optimized using fixed methods, which is a unique architectural advantage of DSPs.
Secondly, the power consumption advantage of DSPs is also quite evident. Besides the high efficiency brought by fixed algorithms, DSPs generally operate at lower frequencies. The lower the frequency, the lower the driving voltage, and power consumption is directly proportional to the square of voltage and frequency. Thus, high frequency requires high voltage, while DSP’s low frequency means relatively lower voltage. If a large amount of data were to be processed by the CPU, not to mention that the CPU’s pipeline architecture is not well-suited for handling massive parallel data, even if forced to do so, the CPU’s high power consumption and relatively slower speed compared to DSPs would greatly hinder the battery life of mobile devices.
With these two significant advantages, DSPs are increasingly being valued today. Besides image processing, DSPs are more efficient than general-purpose CPUs in various applications including video, audio, sensors, and computer vision, thus their application range is expanding. A good DSP on mobile devices can provide users with better photo quality and superior user experience. So, what characteristics do high-end DSP products currently have?
Qualcomm Hexagon 685 DSP
The Qualcomm Snapdragon 845 processor is currently the preferred product for many high-end smartphones. In the Snapdragon 845, besides the powerful CPU and GPU components, its DSP part is also very noteworthy.
The DSP used in the Snapdragon 845 is called Hexagon 685. Hexagon 685 is an upgraded version of Hexagon 682 used in Snapdragon 835, and their technology is primarily derived from Hexagon 680 released by Qualcomm in Snapdragon 820. From an industry development perspective, Qualcomm is very forward-looking; during the Hexagon 680 (Snapdragon 820) era, Qualcomm enhanced the processor’s vector computing capabilities by introducing HVX, or Hexagon Vector eXtensions.
▲The Snapdragon 845 processor uses Hexagon 685 DSP
In Hexagon 680, Qualcomm added two parallel HVXs, each containing four 1024-bit SIMD pipelines, capable of processing large amounts of vector data quickly and in parallel. Years later, with the rise of deep learning and machine learning, these areas require a lot of vector data for computation. However, traditional mobile CPU and GPU architectures are not well-suited for efficiently running such content, especially algorithms like stochastic gradient descent required in deep learning.
▲Qualcomm has detailed the architecture and details of Hexagon 680 DSP. Currently, there is less information available about Hexagon 685, but it has some relation to Hexagon 680. The image shows the architecture diagram of Hexagon 680, which illustrates the VLIW parallel pipeline design.
In this context, the Hexagon 685, which inherits and comprehensively strengthens the Hexagon 680 architecture, not only continues to be powerful in image processing and sensor data handling but also brings deep learning capabilities that are several times stronger than traditional DSPs, allowing Hexagon 685 to achieve better performance in various deep learning-related applications.
In addition to its powerful vector processing capabilities, the instruction set system of Hexagon 685 has also been improved. Due to further adjustments in control code, Hexagon 685 achieves higher efficiency than traditional VLIW and employs techniques to manage idle and stalled threads, making data execution more effective. Additionally, Hexagon 685 can achieve zero-latency loop thread scheduling, meaning that the processing efficiency of the DSP will be significantly enhanced.
▲Qualcomm provided detailed information about Hexagon 685 DSP when launching Snapdragon 845
With the hardware setup completed, Qualcomm is preparing to introduce software support for Hexagon 685. Currently, Qualcomm is working on bringing TensorFlow support to Snapdragon 845 and Hexagon 685, which will accelerate a large number of AI applications. Qualcomm claims that Hexagon 685’s performance is 25 times faster than that of non-Hexagon devices. In a demonstration using the GoogleNet Inception deep neural network algorithm, Hexagon 685 was able to recognize more images and do so faster. This allows Hexagon 685 to quickly locate the required targets in image processing and achieve specified optimizations and enhancements.
With the addition of deep learning capabilities, Hexagon 685, combined with appropriate software, has the potential to bring AI technology into photography and imaging processes, greatly improving the current automatic processing of photographic images. Typical application scenarios include face recognition, real-time noise reduction, multi-frame synthesis, and real-time camera filters.
For example, when users take portraits at night, it can be challenging to illuminate the night scene while also ensuring the face is well-lit. If a flash is used to illuminate the face, the night scene can become very dim. Now, users can utilize deep learning capabilities to allow the machine to recognize and separate the portrait and the night scene, ultimately applying different color and lighting processing methods to both, and then merging them into a single photo, achieving a harmonious coexistence of the portrait and the night scene.
Additionally, in animal photography, special scene recognition (such as different white balance scenes at the beach or desert), and color post-processing, the addition of AI technology can lead to superior photographic results, eliminating the need for extra filters to adjust and process images, making the process easier and more convenient.
Enhanced Functions, Embracing AI Cadence Tensilica Vision Q6
In addition to established manufacturers like Qualcomm, Cadence, a well-known EDA software company, has also launched a series of products in the DSP field. Looking at Cadence’s development in DSP, the company previously released excellent DSPs such as P6, which have been widely used in SoCs from manufacturers like Huawei and MTK. The recent release of its new product, Q6, brings even more powerful computing capabilities.
▲Cadence, as an established EDA manufacturer, has also made achievements in the DSP field.
The Vision P6 attracted industry attention after the release of Huawei’s P20 series smartphones. As Huawei used Vision P6 as its DSP in the Kirin 970, the Huawei P20 Pro showcased powerful photography capabilities, with excellent photo quality, accurate colors, and rich details. Achieving such photographic results was largely due to the Vision P6 DSP, which, according to current information, provides features including mechanical image stabilization (anti-shake), night mode, multi-frame noise reduction, and low-light shooting capabilities, significantly enhancing the photography effects of Huawei phones.
▲Cadence DSP can deeply intervene in various processes of image capture
Following the Vision P6 DSP, Cadence released the Vision Q6 DSP. From the parameters, Vision Q6 has significantly enhanced many features compared to Vision P6, including a 1.5 times increase in AI capabilities, a 1.5 times increase in frequency, and a 1.25 times improvement in energy efficiency. In terms of detailed parameters, the microarchitecture of Vision Q6 has also been adjusted, increasing the pipeline depth from 10 stages to 13 stages; it has comprehensively improved branch prediction functionality, ensuring that deeper pipelines do not negatively impact IPC. Other improvements include strengthening the storage system, enhancing load/store units, increasing storage bandwidth, and refining ISA for image processing and machine learning.
▲The currently released Vision Q6 is the design version of the previous generation P6, which can be paired with the C5 AI acceleration module.
After enhancing the vector processing hardware, Cadence announced support for the Android NN API, including support for TensorFlow, TensorFlow Lite, and existing Caffe frameworks. This means that Vision Q6, like the previously introduced Hexagon 685, has strengthened support for deep learning, allowing Vision Q6 to leverage AI technology to enhance image processing and achieve better image quality.
▲Vision Q6’s architectural improvements
▲Vision Q6 also supports AI-related computational acceleration
AI is Just Beginning, DSP is Still Evolving
Currently, DSPs are further enhancing traditional image, audio, and other signal processing functions while also strengthening AI-related capabilities. After all, with the addition of AI, operations that were previously difficult or impossible to handle can now be managed by AI for judgment and recognition, which will positively enhance the photography and video shooting capabilities on mobile devices, helping users achieve better experiences and obtain more outstanding or realistic photos and videos.
However, the application of AI in DSPs is just beginning. From the development trends in the industry, there is still no clear answer as to whether AI should operate as a co-processor or whether DSPs should enhance their AI capabilities. However, with the advancement of technology, AI is gradually unleashing its increasingly powerful potential. Whether DSP and AI will merge, diverge, or mutually strengthen each other remains to be seen.
