Since the explosion of generative AI technology represented by ChatGPT at the end of last year and the beginning of this year, various participants in the edge market have been looking for ways to leverage the momentum of generative AI. For instance, chip companies and OEM manufacturers in the PC sector are discussing how lightweight laptops can run generative AI; even companies in the embedded market that produce SoC/MPU chips are talking about generative AI…Recently, MediaTek released the Dimensity 9300 chip, which is marketed as the “Dimensity 9300 flagship 5G generative AI mobile chip”—let’s analyze this term. 5G is a standard feature of the Dimensity 9000 series, so there’s no need to elaborate on that; the terms “flagship” and “generative AI” particularly correspond to (1) a full big-core CPU architecture design, and (2) the ability to run generative AI.
These two points are astonishing for a couple of reasons. On one hand, the CPU big.LITTLE or big.medium.small core designs have been prevalent in Android phones for many years, with Arm’s Cortex-A7, A53, and A55 being classic small core designs that have been validated by the market. Now, removing the small cores represents a breakthrough for MediaTek, which has traditionally followed Arm’s IP.On the other hand, running generative AI on the edge is not surprising in itself—what is truly astonishing is that MediaTek claims its Dimensity 9300 can handle LLMs with 7 billion, 13 billion, and even 33 billion parameters without breaking a sweat. This is indeed an impressive achievement for a mobile chip that pursues low power consumption, and it gives real value to local inference of generative AI.We will mainly look at the Dimensity 9300 in detail from these two aspects.Overview of Dimensity 9300 Configuration and HighlightsFirst, let’s summarize the configuration of the Dimensity 9300, which has a total of 22.7 billion transistors, and compare it with the previous generation Dimensity 9200.
In our view, there are several highlights. The first is the removal of the small core cluster in the CPU. Although it is still an 8-core design, it features 4 Cortex-X4 super big cores and 4 Cortex-A720 big cores. MediaTek claims that compared to the Dimensity 9200, the multi-core peak performance has improved by 40%, while power consumption has decreased by 33% when reaching the peak performance of the Dimensity 9200.The presentation even directly showed that the multi-core score on Geekbench 6 is between 7500-8000, surpassing “Company A’s A series chips” and “Company Q’s flagship chip of 2023″—a quick check of the Geekbench 6 database reveals that these refer to Apple’s A17 Pro and Qualcomm’s Snapdragon 8 Gen 3.
Although the scale of this chart starts at a score of 6500, making the lead appear exaggerated, it indeed achieves comprehensive leadership in CPU multi-core performance.The second highlight is the AI acceleration unit APU 790, which can run generative AI—whether at the press conference or in pre-briefings with the media, MediaTek devoted considerable time to discussing the APU, which is a first for MediaTek to highlight it so prominently, even placing it before the GPU in discussions, which will be elaborated on later.Its paper specifications include “8x speed generative AI Transformer operator acceleration,” “mixed precision INT4 quantization technology,” “NeuroPilot Compression memory hardware compression technology,” 2x integer performance, 2x floating-point performance improvement, and a 45% reduction in power consumption. MediaTek announced an ETHZ AI Benchmark v5.1 score of 2109. However, during the media communication session, MediaTek emphasized that if memory test scores are included, the system performance score would increase by over 1000 points. After all, running large models is very demanding on storage performance.
Therefore, the support for LPDDR5T-9600MT memory is also a highlight. The “T” in LPDDR5T was first announced by SK Hynix in January this year, indicating that their new LPDDR5 product has a bandwidth of 9.6Gbps, which is 13% higher than LPDDR5X, hence the name LPDDR5 Turbo, with a supply voltage range of 1.01-1.12V. Samsung and Micron are expected to have similar products.Recently, foreign media reported that Micron’s LPDDR5X-9600 and SK Hynix’s LPDDR5T-9600 have been verified to be compatible with Snapdragon 8 Gen 3. However, during this media communication session, MediaTek stated: “The 9600 scale will be the first to be implemented globally on our chip platform, truly commercially available for end consumers.” They also mentioned that “competitors” have not actually implemented this design and that no terminal products with LPDDR5T-9600 will appear this year.The support for LPDDR5T-9600 may also be related to the application of generative AI on this generation of chips. As we know, AI large models have high demands for RAM bandwidth and capacity.Another highlight in the configuration table is the upgrade of the GPU to Immortalis-G720 MC12. MediaTek stated that this GPU’s peak performance has improved by 46% compared to the Immortalis-G715 MC11 of the Dimensity 9200, and the power consumption has decreased by 40% when reaching the peak performance of the latter; they also presented two GFXBench graphics tests compared to Snapdragon 8 Gen 3 and A17 Pro:
We know that Apple has been squeezing performance out of its GPU for two years, and the performance upgrade of Qualcomm’s Snapdragon 8 Gen 3 GPU this year has been quite significant, but it is surprising that Immortalis is performing so well now. It is worth mentioning that the two tests in the image above show that the Aztec Ruins test is geometrically less complex than the Manhattan test, but the shader load is significantly higher. This could serve as a basis for comparing the performance focus differences between Immortalis and Adreno.Additionally, the GPU of the Dimensity 9300 has seen a 46% improvement in ray tracing performance and an 86% improvement in VRS performance—more details on ray tracing support and the Dimensity graphics ecosystem will be discussed later.Other highlights of the Dimensity 9300 chip include an upgrade of the ISP to Imagiq 990. MediaTek specifically mentioned that the AI semantic segmentation video engine achieves “16-layer image semantic segmentation,” which is several layers more than other chips; as well as “full pixel focus,” “2x lossless zoom,” “OIS optical anti-shake core,” and support for “3-microphone high dynamic recording noise reduction.” Technical upgrades related to the display engine and connectivity will not be elaborated on here; in fact, Wi-Fi 7 enhanced Mtra Range 2.0 and dual Bluetooth fast connection technology can also be considered highlights.CPU: The Rationality of the “Full Big-Core” DesignThis year, Arm recommended the TCS23 solution at its new product launch, suggesting mobile AP SoCs with CPU big.medium.small core combinations including 1+3+4, 1+5+2, and a new 10+4+0 scheme aimed at laptops. Qualcomm has clearly chosen the 1+5+2 scheme, which consists of 1 Cortex-X4 super big core, 5 Cortex-A720 (further divided into two groups based on frequency) big cores, and 2 Cortex-A520 small cores.
MediaTek did not choose any of these but opted for a 4+4+0 scheme—more akin to a laptop scheme. In fact, MediaTek expressed last year that it was investing in SoC chips for the PC platform. Previously, MediaTek also had the Kompanio 1380 chip aimed at Chromebooks. In May of this year, Electronic Times reported that MediaTek is collaborating with NVIDIA to develop Windows on Arm platform products for laptops.If this news is reliable, it suggests that MediaTek’s achievements in PC chips should benefit the mobile platform (although we should generally say the opposite). Looking at it this way, doesn’t the 4+4+0 scheme of the Dimensity 9300 seem much more reasonable for the CPU part? (Especially with the landing of LPDDR5T, doesn’t it seem more plausible?)Generally, we say that the value of the “small core” in mobile platforms is to save power—mainly to maintain lower power consumption and higher efficiency compared to big cores in low-load scenarios; and to achieve as high a performance gain as possible with a smaller area. Arm’s small cores have so far been sequential architectures, designed to maintain lightness, small size, and low power consumption.However, in reality, there have been historical precedents for mobile platforms not adopting “small cores.” For example, Qualcomm’s early self-developed CPU architectures, at least up to the Kryo era of Snapdragon 820, still adhered to homogeneous cores without any small cores—and did not suffer much in terms of battery life. Moreover, we have previously discussed in analysis articles that the last self-developed Kryo architecture of Snapdragon 820 was more a case of being born at the wrong time.Another excellent example is that although we always say that Apple’s A-series chips have a “big.little” heterogeneous CPU, their “small cores” have historically maintained an out-of-order (OoO) design throughout, and their scale is not “small” at all. For instance, the performance of the A15’s “small core” exceeds that of the Cortex-A76 and is even comparable to the low-frequency Cortex-A78.
The most obvious value of a full big-core design is the significant improvement in multi-threaded performance. Therefore, it is not surprising that MediaTek announced a 40% improvement in multi-core peak performance for the Dimensity 9300 compared to the 9200, also surpassing this year’s competitors from Apple and Qualcomm. This includes a lab environment AnTuTu score exceeding 2.2 million, which serves as promotional material.The other value of stacking cores is multi-tasking performance: at the media conference, MediaTek mentioned that while achieving 60fps frame rate in the game “Genshin Impact” at ultra-high quality, it could also conduct a WeChat video call. This is difficult to achieve on traditional architectures.In fact, the key to this performance is whether the system design of small-sized devices like smartphones can fully leverage the full performance of the Dimensity 9300, especially the 4 Cortex-X4 big cores. This is a question worth paying attention to in the future. From MediaTek’s announcement of GB6 performance results, it can be seen that the Dimensity 9300 may test the system design capabilities of OEM manufacturers.The next question is whether, after removing the small cores, battery life and efficiency can still be guaranteed? MediaTek’s explanation and logic at the press conference was, “Big cores work fast and rest fast, leading to low power consumption.” This logic is certainly valid, and it is something we often say: many times, it is not about high power consumption, but rather matching it with sufficiently strong performance, which can actually allow tasks to be completed more quickly—this is how Apple has operated in the past.What we refer to as “power consumption” generally refers to the energy consumed per unit time (in watts), rather than the energy consumed to complete a task (in joules). The actual test of smartphone battery life is the total energy consumed by all tasks; as long as it is not always in a high power consumption state, “the faster it is done, the more power is saved.”Thanks to architectural updates and process improvements, as well as efficient performance in high-load scenarios, the Dimensity 9300 also sees a corresponding reduction in power consumption. MediaTek mentioned that when reaching the peak performance of the Dimensity 9200, the power consumption of the Dimensity 9300 is reduced by 33%—this is completely intuitive, as more big cores inherently have higher efficiency in high-load scenarios.
Thus, we see the power consumption of the Dimensity 9300 in various usage scenarios has decreased. However, this still does not explain how low power consumption is achieved in low-load scenarios. Arm’s design of such 3-cluster architectures aims to solve problems with different cores in different load ranges—because each core only achieves maximum efficiency within a specific performance range, as shown in the following graph of performance versus power consumption.
The Cortex-A520 small core is designed to achieve lower power consumption compared to A720 and X4 in low-load scenarios. This is the original intention of Arm’s design for such architectures. The power consumption reductions we discussed earlier are difficult to claim for low-load scenarios. So how does the Dimensity 9300’s 4+4+0 scheme ensure efficiency during standby and low-load periods?First, let’s discuss our speculation. The first is that since the Cortex-A510, Arm’s small core design seems to have entered a bottleneck period. The efficiency improvement of A510 compared to the earlier A55 is minimal, with performance gains largely at the cost of power consumption; and we believe its area efficiency is not ideal. The A520 has made significant improvements in efficiency, including enhanced branch prediction and data prefetch performance, restructured the storage subsystem, and removed some performance enhancements—such as eliminating an ALU pipeline. However, we believe that Arm’s small cores are no longer the classics of the past.Secondly, this may be related to manufacturing processes. MediaTek stated at the media conference: “In recent years, we have undergone a significant conceptual shift in architecture design, realizing that small cores are not necessarily more power-efficient. The more advanced the process, the smaller the transistors—if they are too small, they cannot contain leakage current. Therefore, the power consumption of small cores may not be better than that of big cores. This issue will become more serious as we move forward. The faster we can shut down the system, the more power we save.”Another possibility is that with the development of the mobile ecosystem, the power consumption models of operating systems and application designs have changed significantly compared to a few years ago. The level of competition among chips has reached a point where chip design companies generally care about upper-level application scenarios. MediaTek may have done more work in this area.“In military terms, you need to accurately assess the enemy and know which enemy to eliminate first, then quickly eliminate them.” MediaTek stated, “What we can do is define some key programs and quickly resolve them, allowing the army to rest afterward.” It is said that this also involves a lot of collaboration and optimization with its ecosystem partners.“In the future, we will work on ensuring that the system can sleep deeper when saving power. We have spent three years on this. Besides hardware design, how to optimize the software and system to find key scenarios, gather applications together, quickly execute them, and then quickly enter deep sleep. This is not a simple task; it must be integrated from end to end, from hardware to software to the system.”“The system background has many small programs, and when those programs pop up, the CPU will be awakened—this will definitely generate power consumption. What we need to do is either combine those small fluctuations in the system or move them aside, possibly using other hardware or SCP (system control processor) methods. This involves more detailed work on how to further save power. This technology is continuously being improved, and we will keep doing this.”Although we are still unclear about how MediaTek ensures energy efficiency in extremely low-load scenarios, this statement suggests that MediaTek should have made technical supplements in other areas. Finally, MediaTek has repeatedly emphasized, “We believe in the full big-core architecture; starting next year, everyone will move in this direction. The future is a full big-core era.”APU: Generative AI on Mobile PhonesAnother key update of the Dimensity 9300 is undoubtedly the APU 790. From MediaTek’s schematic diagram, the APU 790 has added a “generative AI engine,” which easily brings to mind the Transformer engine that NVIDIA specifically added in this generation of the Hopper architecture. This is a response to the trend of the AI era, equivalent to adding a more specialized accelerator within the accelerator.During the media communication session, MediaTek discussed that the APU 790 is “deeply adapted to Transformer models,” especially optimizing the Softmax + LayerNorm operators, achieving processing speeds that are 8 times faster than the previous generation. This is significant because Softmax + LayerNorm accounts for a large portion of the processing in Transformers.
Currently, the commonly discussed Transformer structure employs a self-attention mechanism to capture global correlations and the relationships between different elements within a queue. The Softmax function is used to help determine the amount of attention or the importance of each word in a sentence. LayerNorm is used for normalization across neurons at a layer level.Previously, the industry generally believed that the structural characteristics of Transformers were more suitable for NLP (natural language processing); while the convolutional characteristics of CNNs (convolutional neural networks) determined their strengths in visual tasks such as image classification and object recognition. Therefore, it is clear that LLMs (large language models) are based on the Transformer structure.In 2021, the Google Research team published a paper proposing a solution to cut an image into small pieces, treating each piece as a single character or token, allowing Transformers to handle image-related tasks. Vision Transformers became quite popular. Models like Stable Diffusion also have traces of the Transformer structure. This should also be the basis for the APU 790 emphasizing the “generative AI engine.”It seems that in the past, we associated keywords like LLM, Transformer, and generative AI with large data center clusters or at least flagship desktop GPUs with TDPs in the hundreds of watts. However, in recent months, there has been an increasing amount of promotion for running generative AI on edge devices, whether mobile phones or PCs. The so-called “running AI” or “operating AI” on these edge devices generally refers to AI inference.Proper commercial large model training still needs to be run on server clusters—this can be understood as the process of building a model. Once the model is built, it needs to be used—this is the inference process. Inference sometimes requires significant computing power, such as the AI inference of ChatGPT being conducted on server clusters.However, small model inference does not require high computing power; many IoT embedded devices also emphasize AI inference capabilities, which is also referred to as running AI. The Dimensity 9300 running generative AI refers to performing inference on generative AI models of a certain scale—although these models may not be as large as ChatGPT, they still have a certain magnitude.For example, MediaTek mentioned in this promotion that the Dimensity 9300 can run a 7 billion parameter AI large language model at a generation speed of 20 tokens/second; and it has “successfully run a 13 billion parameter AI large language model on the edge”; and achieved the “first mobile chip to successfully run a 33 billion parameter AI large language model.”However, we are unclear about the operating environment and exact practicality of the latter two scales of LLMs—the 33 billion figure seems to be mentioned to benchmark against the Snapdragon 8 Gen 3’s support for 10 billion parameter scale.Nonetheless, the 20 tokens/second for the 7 billion parameter LLM is indeed practical—and MediaTek also mentioned collaboration with vivo to achieve the edge deployment of the 7 billion parameter LLM. Many may not have a clear concept of parameter scale; previously, we experienced running a 60 billion parameter ChatGLM2, a 130 billion parameter Llama 2, and a 155 billion parameter StarCoder using a laptop’s Core processor without an accelerator—however, it is important to note that this was purely CPU-based. This can serve as a reference for scale.Additionally, MediaTek stated that the Dimensity 9300 achieves “high-speed generative AI image generation” with Stable Diffusion in less than 1 second—this expression is somewhat vague, as Stable Diffusion’s image generation involves various issues such as iteration count and image resolution.Overall, to achieve the aforementioned scale of models running on a mobile chip, relying solely on the APU’s Transformer model operator acceleration is likely insufficient. In our view, the full big-core CPU and high-frequency LPDDR5T memory also play a role, and MediaTek has made other related technical supplements.
Firstly, there is the memory hardware compression technology NeuroPilot Compression. The quantifiable data indicates that the 13GB memory originally required for the 13 billion parameter LLM, when combined with the memory requirements for the Android operating system and other apps, is no longer sufficient with the current flagship phones’ 16GB memory. Therefore, NeuroPilot Compression can compress it down to 5GB after quantization and compression.Secondly, in terms of development ecosystem and toolchain, MediaTek likely needs to do a lot of related work, including model quantization, optimization, and conversion, ultimately simplifying and lowering the performance requirements for local inference on the edge. This includes collaboration on models, with mentions of deep adaptation and optimization for mainstream large language models such as Android AI foundational model, Baidu’s Wenxin Yiyan, Baichuan large model, and Llama 2, specifically for deployment on the edge.Additionally, it is worth mentioning that MediaTek referred to “generative AI edge ‘skill expansion'”—based on a large model, allowing the edge AI to learn more skills through cloud training, enabling the Dimensity 9300 to become the “first chip to support generative AI edge LoRA fusion,” which allows for the integration of multiple functions based on a large model through cloud training. LoRA (Low-Rank Adaptation) is generally used in model fine-tuning, characterized by reducing the number of parameters. The logic and process of “edge LoRA fusion” mentioned here are not very clear to us—based on the introduction, LoRA fusion seems to be part of the NeuroPilot development toolchain.Finally, we must discuss applications. After all, shouting about generative AI ultimately needs to translate into applications on mobile phones. MediaTek demonstrated using a phone with the Dimensity 9300 chip to generate poetry, create images and meme stickers based on input requirements, and generate personalized GIF stickers from personal photos.Another more practical application is the AI assistant developed in collaboration with vivo, which completes tasks in a conversational format: for example, semantic search, drawing, copywriting, photo editing, information retrieval, generating mind maps, and scheduling tasks with a single click.
MediaTek’s own listed generative AI applications also include generative super-resolution for images (in collaboration with ASR), real-time generation of 3D avatars based on real human images during video calls (in collaboration with Morpho), and anti-glare for night photos (in collaboration with TETRAS.AI), etc. The early ecosystem of edge generative AI is theoretically somewhat fragmented, and this period is also a time for chip companies to seize opportunities. MediaTek’s ecosystem partners showcased at the press conference include Kuaishou, Jianying, Meitu Xiuxiu, Huya Live, Douyin, iQIYI, QQ Music, and more.In the future, the technological achievements of generative AI on mobile phones are likely to gradually become visible in these applications. The truly killer applications will depend on the ideas of these application layer participants—MediaTek provides the hardware foundation and development platform. Mobile users can look forward to it.GPU: This Time It Really WorksThe Dimensity 9300 launch conference made us feel for the first time that MediaTek’s product and technology points were so numerous that they couldn’t keep up with the explanations, as it seemed there were too many highlights. Just the introduction of the CPU and APU took over 40 minutes. This left the GPU, ISP, and other equally worthy sections to be discussed relatively briefly. This has likely never happened in MediaTek’s history of mobile AP SoC launch conferences.For example, since September of last year, MediaTek has been collaborating with Discovery to produce documentary programs and has held film festivals, which has made the ISP introduction seem more substantial. However, it is clear that in this launch conference, the ISP’s spotlight was completely overshadowed by the APU. Due to space limitations, we will not elaborate on ISP, display engine, wireless communication support, dual security chips, and MediaTek’s regular low-power supporting technologies.Here, we will finally discuss the GPU and MediaTek’s graphics ecosystem, as many users interested in Dimensity chips are mobile gamers. As mentioned earlier, MediaTek has equipped the Dimensity 9300 with a 12-core Immortalis-G720, whose peak performance has improved by 46% compared to the Immortalis-G715 of the Dimensity 9200, and the power consumption has decreased by 40% when reaching the peak performance of the latter. Overall graphics performance also surpasses that of the A17 Pro and Snapdragon 8 Gen 3—this should be the first time MediaTek, and also Arm, has comprehensively surpassed the other two competitors in absolute performance.This figure is better than we expected, as Arm mentioned an average improvement of 15% in sustained and peak performance when releasing the Immortalis-G720. The Immortalis-G720 design emphasizes system-level energy efficiency, especially reducing memory bandwidth usage—this generation’s DVS (Deferred Vertex Shading) significantly reduces bandwidth requirements and the proportion of DRAM power consumption in games, with average power consumption reduced by 20%. The performance improvement of the Dimensity 9300’s GPU is likely largely due to the addition of an extra core.
Other values are generally consistent with Arm’s announcements, including reduced bandwidth requirements, improved VRS performance, and MSAA optimizations. In terms of system performance data, MediaTek mentioned that even in the ultra-high quality of the “Genshin Impact” game on the Sumeru map, the Dimensity 9300 can maintain 60fps for 30 minutes; and at 840p resolution with ultra-high quality, “Honkai: Star Rail” can run at full frame for 30 minutes. The latter is a game that clearly relies more on GPU performance. If it weren’t for the APU stealing the spotlight, the GPU of the Dimensity 9300 would definitely deserve more attention.Regarding ray tracing, the PPT mentioned a 46% improvement in hardware ray tracing performance. The Immortalis architecture determines that more shader cores can lead to higher ray tracing performance.The ray tracing performance of the Dimensity 9200 currently ranks first in the Basemark InVitro test (the model is likely the soon-to-be-released vivo X100)—Arm also emphasized the leading position of Basemark InVitro at its press conference. MediaTek believes that Basemark InVitro is “closer to complex ray tracing scenarios,” where ray tracing occupies a significant proportion of the scene, reflecting actual ray tracing performance.From known tests, the A17 Pro has better performance in the 3DMark Solar Bay ray tracing test compared to the Dimensity 9200. Based on the 46% improvement value, the Dimensity 9300 should also be able to take the lead in this test. The differences in rankings across different tests may still be related to whether ray tracing tests only examine ray tracing reflections, the involvement of rasterization in the scene, or whether more complex ray tracing tests are included.Many users interested in mobile gaming ray tracing are more concerned about game support and ecosystem development. Regarding ecosystem-related issues, MediaTek announced that the Dimensity 9300 “first supports 60fps ray tracing in ‘Escape from Tarkov'”; additionally, the ray tracing version of “Sword and Fairy World” is also set to debut on the Dimensity platform. MediaTek stated that over 50 games have joined the Dimensity ray tracing ecosystem, and the ecosystem has achieved full coverage of the three major engines: Unity, Unreal, and Messiah. This is currently a unique achievement in the mobile industry. Ray tracing has finally become useful.
Another highlight related to the graphics ecosystem is the support for global illumination, in collaboration with Unreal Engine and Arm, integrating Vulkan Ray Query + “desktop-level rendering Shader Model 5” into Unreal 5 Lumen. Specifically, in collaboration with Tencent Games, the Smart GI technology for global illumination is applied in the game “Escape from Tarkov.” The popularization of such technologies is something mobile gamers are eager to see.Additionally, updates related to graphics technology include the MAGT game adaptive control technology upgraded to the “Star Speed Engine.” It is said that after applying the Star Speed Engine, “Honor of Kings” achieved a 15% reduction in power consumption and a 30% improvement in loading speed. These data still show that MediaTek’s graphics ecosystem is deeply influencing the domestic mobile gaming industry.
Future Personalization Needs for Mobile PhonesThe Dimensity 9300 indeed brings surprises from various dimensions. However, the most surprising aspect remains the enhancement of generative AI within it. In media interviews, MediaTek envisioned the application of generative AI on mobile phones: “Mobile phones are carried with you; they know your personalized needs and your location. When answering your questions, they can include all information to provide personalized responses that are truly needed by you.”“A generative AI mobile phone is full of imagination. The inputs are diverse and complex, including voice, text, gestures, etc. As flagship phones have evolved, everyone should gradually detach them from benchmark scores.”On November 13, the vivo X100 will be the first to launch with the Dimensity 9300 chip. This may mark a new chapter for generative AI mobile phones.
