Understanding the Roles of Modem, ISP, DSP, and NPU in SoC
Click above to follow us
In the Android system ecosystem, the performance of a simple processor (CPU + GPU) does not solely determine the final user experience. It also involves optimizations at the network, system, and application levels (such as various games and network acceleration engines), multi-core scheduling capabilities, and security design. Next, let’s explore the roles of other components within the SoC.Related Articles:Hardcore Science! Why is the performance of SoC dependent on architecture and process?Why is Snapdragon 865 the most powerful? Let’s understand the CPU and GPU architecture!Modem – The Network PioneerFor PCs to access the internet wirelessly, a wireless network card must be installed. However, in the mobile SoC field, the “wireless network card” (baseband, also known as modem) is one of the internal components of the SoC, which determines the network standards supported by a phone as well as internet and download speeds.Integrated or ExternalThe modem is not a mandatory option for the SoC. Chip manufacturers can choose to integrate the modem directly into the SoC or use it as a standalone chip soldered onto the phone’s motherboard.For example, the Qualcomm Snapdragon 855 directly integrates the Snapdragon X24 modem, while the Snapdragon 865 removes the built-in modem design, requiring an additional Snapdragon X55 to connect to the internet.Generally speaking, SoCs with integrated modems save motherboard space and perform better in terms of energy consumption.In 2020, new smartphones will be equipped with 5G SoCs, which integrate or externally support 5G modems.For comparisons of network performance among 5G SoCs and the factors affecting their maximum speeds, interested readers can refer to the article “Isn’t the 5G network speed just three times different?! What exactly is millimeter wave?!“Like the modem, the Wi-Fi and Bluetooth modules of the SoC can also be integrated directly into the SoC or realized through external chips.For instance, the Snapdragon 855 directly integrates Wi-Fi and Bluetooth units, while the Snapdragon 865 requires an external standalone FastConnect 6800 chip to support the latest Wi-Fi 6 and Bluetooth 5.1 networks. The MediaTek Dimensity 1000+ is a representative that integrates the Wi-Fi 6 unit directly within the SoC.Rationally Viewing Modem PerformanceCurrently, the three major operators have implemented speed limiting strategies for both 4G and 5G networks, with the maximum downlink speed for 4G being 300Mbps, while the 5G network is limited to just 1Gbps.This means that no matter how powerful the modem performance integrated or external to the SoC, it cannot exceed this limit in the short term.Therefore, when purchasing SoCs and smartphones, we should not overly focus on the modem’s maximum theoretical speed, but rather pay attention to its support for 5G functionalities, such as whether it supports dual-mode networking, how many bands it supports, and whether it can achieve dual SIM 5G full network compatibility, etc.ISP – The Imaging KeyNowadays, many smartphones are equipped with the same 48MP or 64MP sensors, but the actual imaging results vary significantly.The imaging algorithm optimizations from mobile phone manufacturers represent “soft power,” while the ISP unit integrated within the SoC determines the “hard power.”Significance of ISPISP stands for Image Signal Processor. The capability of a SoC to support several cameras, the maximum pixel count of sensors it can support, the resolution and frame rates (such as 8K/30FPS) for video recording, the FPS for slow-motion capture, HDR video support, and image processing calculations all depend on the ISP.In other words, the stronger the ISP specifications, the more advanced cameras it can support, and with the same cameras, it possesses better imaging capabilities (not absolute, as imaging algorithms require complex soft-hard coordination; otherwise, it could lead to “negative optimization”).ISP Naming StandardsIn the naming of ISPs, Qualcomm Snapdragon’s naming is perhaps the most standardized. For example, the ISP unit integrated into the Snapdragon 710 is called Spectra 250, the Snapdragon 765 is Spectra 350, and the Snapdragon 865 is Spectra 480, allowing us to easily identify which is stronger from the suffix numbers.The Kirin family prefers to use x.0 notation, such as the Kirin 980/Kirin 810 integrating ISP 4.0, while the Kirin 990 upgraded to ISP 5.0. MediaTek’s ISPs tend to “stack cores,” for example, the Helio G90 series integrates a 3-core ISP, while the Dimensity 1000’s ISP has five cores.DSP – No Longer SimpleDSP stands for Digital Signal Processor. It was originally mainly used for processing audio signals, such as voice noise reduction, digital-to-analog conversion, and special audio effects. However, in the Qualcomm Snapdragon family, the role of the DSP unit has been further elevated.Familiar Yet Strange DSPWe all know that DSP is one of the components of the SoC, but in the architecture diagrams of HiSilicon Kirin, MediaTek, and Samsung SoCs, we do not see the shadow of the DSP unit. However, in the Qualcomm Snapdragon mobile platform, the DSP is often highlighted, from the Hexagon 680 of Snapdragon 660 to the Hexagon 698 of Snapdragon 865, with higher suffix numbers indicating stronger performance.DSP’s Cross-Boundary ApplicationsWith the rise of VR/AR applications and AI technology, Qualcomm has endowed the DSP unit with more capabilities, including vector extension (HVX) and tensor accelerators.Unfortunately, Snapdragon SoCs have never integrated standalone NPU units, and the main selling point of the “Qualcomm AI Engine” requires collaboration among the CPU, GPU, DSP, memory, cache, and other modules. Other chip manufacturers’ SoCs introduced NPU modules long ago, so there was no need for significant changes to the DSP.NPU – Intelligence Relies on ItAt the end of 2017, Huawei introduced the concept of NPU (Neural Processing Unit) with the Kirin 970, giving SoCs stronger local (edge-side) AI computing capabilities (similar to “hardware decoding”), with execution efficiency surpassing the “software decoding” of CPUs. Since then, the NPU unit has become synonymous with AI.From the single-core NPU of Kirin 970, the dual-core NPU of Kirin 980, to the latest Kirin 990 5G, which has integrated a 2+1 tri-core NPU.AI EverywhereInitially, AI was mainly used for photography, such as the intelligent scene recognition feature when framing, allowing the system to quickly identify objects and scenes being captured and automatically optimize settings.Subsequently, AI functionality gradually expanded, covering handheld super night scenes, AI voice assistants, AI gaming engines, AI network acceleration, AI energy optimization, AI smart recognition, AI image translation… Almost all application scenarios rely on AI for accelerated computing.Therefore, a SoC chip without sufficient AI capability is not even worthy of stepping out to greet others.Different AI UnitsDifferent brands of SoCs have slightly different naming and implementation methods for AI units. Except for the Qualcomm Snapdragon AI Engine, which does not have a standalone NPU unit, MediaTek’s NeuroPilot AI technology introduced in Helio P60/P90 was also initially achieved through collaborative computing of multiple units (APU + CPU + GPU; at this time, the integrated APU was not standalone hardware, but more like the tensor accelerators newly integrated into the Snapdragon DSP).However, starting with Dimensity 1000, MediaTek also launched a standalone AI processor (APU 3.0), which has already surpassed Kirin 990 5G’s scores in AI benchmarks.Samsung began integrating standalone NPU units into SoCs starting with Exynos 9820, and the latest Exynos 980 and Exynos 990 have further enhanced NPU performance.Composition of Samsung Exynos 980 SoCIt is worth noting that AI units in mobile phones are still in their early application stages, and their importance is far less than that of CPUs, GPUs, and ISPs, making them more of a supplementary presence.Others – The Unsung HeroesIn addition to the aforementioned “well-known” units, many functional modules are also integrated within mobile SoCs. Although they may not be very eye-catching, the tasks they perform are also very important.Memory/Storage ControllerWe all know that the soon-to-be-mass-produced LPDDR5 memory outperforms the current mainstream LPDDR4X, and the reading speed of UFS 3.0 flash memory can also surpass that of UFS 2.1, but not all SoCs support these latest technology standards. For example, only Snapdragon 865 and Exynos 990 currently qualify for LPDDR5 memory, while Kirin 990 and MediaTek Dimensity 1000 only support LPDDR4X.CoprocessorMany SoCs integrate a unit called “coprocessor” (such as the Sensor Hub integrated into Qualcomm Snapdragon SoCs, and the i8 integrated into Kirin 980). They are mostly built on the more power-efficient ARM Cortex-M series architecture and serve to assist the CPU by continuously monitoring, collecting, and processing data from sensors like accelerometers, gyroscopes, compasses, and new barometers 24/7, allowing you to safely use step counting, GPS positioning, and other features without worrying about power consumption.Security ModuleThanks to the convenience of mobile payments, many users have already become accustomed to “cashless consumption.” However, who ensures the security of private data and payment-related authentication on mobile phones? To address this issue, many SoCs have begun to integrate dedicated security modules, such as the inSE security chip built into Kirin chips and the SPU security processor integrated into Snapdragon chips.Their role is to keep facial recognition, fingerprint information, ISP, DSP, memory, and all other security data generated by participating modules stored locally without uploading to the cloud, thus maximizing system security.Other ModulesSoCs also integrate display, audio, and video units, which affect the resolutions of screens that mobile phones can use and the formats of audio/video codecs they support. Additionally, the fast charging technology compatible with a smartphone is also influenced by certain units within the SoC. For instance, the Snapdragon 710 only supports QC4 fast charging, while the Snapdragon 675 can support the more efficient QC4+. Fortunately, these unit modules have almost no impact on SoC performance, and we only need to understand them briefly.
Click “Read the Original” for more exciting content
