With the introduction of artificial intelligence (AI), especially generative AI, the automotive industry is undergoing a transformative shift. A recent survey conducted by McKinsey among automotive and manufacturing executives revealed that over 40% of respondents are investing up to €5 million in generative AI research and development, with more than 10% of respondents investing over €20 million.
As the industry continues to evolve towards software-defined vehicles (SDVs), the number of lines of code in cars is expected to increase from 100 million per vehicle to approximately 300 million by 2030. Generative AI for automotive applications, combined with SDVs, can jointly achieve in-vehicle use cases that enhance performance and comfort, thereby improving the driving experience.
This article will introduce a generative AI use case developed in collaboration between Arm and Amazon Web Services (AWS) and detail its implementation.
Use Case Introduction
As vehicles become increasingly sophisticated, owners can now receive updates for features such as parking assistance or lane-keeping after delivery. The challenge that arises is how to keep owners informed about new updates and features in a timely manner. Traditional methods of updating through paper or online manuals have proven inadequate, leading to owners not fully understanding the potential of their vehicles.
To address this challenge, AWS has combined the powerful capabilities of generative AI, edge computing, and the Internet of Things (IoT) to develop a demonstration of in-vehicle generative AI. The solution showcased in this demonstration is an in-vehicle application supported by a small language model (SLM), designed to enable drivers to obtain the latest vehicle information through natural voice interaction. This demonstration application can operate offline after deployment, ensuring that drivers can access important information about their vehicles even without an internet connection.
The solution integrates multiple advanced technologies to create a more seamless and efficient product experience for users. The application in this demonstration is deployed on a local small language model within the vehicle, which enhances performance using routines optimized by Arm KleidiAI. The response time of a system not optimized by KleidiAI is around 8 to 19 seconds, whereas the inference response time of the KleidiAI-optimized small language model is between 1 to 3 seconds. By utilizing KleidiAI, the application development time was reduced by 6 weeks, and developers did not need to focus on optimizing the underlying software during the development period.
Arm Virtual Hardware supports access to many popular IoT development kits on AWS. When physical devices are unavailable, or teams around the world cannot access physical devices, developing and testing on Arm Virtual Hardware can save time in embedded application development. AWS successfully tested the demonstration application on the automotive virtual platform, where Arm Virtual Hardware provided a virtual instance of a Raspberry Pi device. The same KleidiAI optimization can also be applied to Arm Virtual Hardware.
One of the key features of this generative AI application running on edge devices is its ability to receive OTA wireless updates, with some updates being received using AWS IoT Greengrass Lite, ensuring that drivers are always provided with the latest information. AWS IoT Greengrass Lite occupies only 5 MB of RAM on edge devices, making it highly memory efficient. Additionally, the solution includes automated quality monitoring and feedback loops to continuously assess the relevance and accuracy of the small language model’s responses. A comparison system is employed to flag responses that exceed expected quality thresholds for review. The collected feedback data is visualized in near real-time through a dashboard on AWS, allowing the OEM’s quality assurance team to review and identify areas for improvement and initiate updates as needed.
This generative AI-powered solution offers advantages beyond providing accurate information to drivers. It also embodies a paradigm shift in SDV lifecycle management, enabling a more continuous improvement cycle where OEMs can add new content based on user interactions, and the small language model can be fine-tuned using update information seamlessly deployed over the air. This ensures that users have access to the latest vehicle information, enhancing the user experience, while OEMs have the opportunity to introduce and guide users on new features or purchasable add-ons. By leveraging the powerful capabilities of generative AI, IoT, and edge computing, this generative AI application can serve as a guide for automotive users, showcasing methods that contribute to a more connected, informed, and adaptive driving experience in the SDV era.
End-to-End Upper-Level Implementation Solution
The solution architecture shown in the figure below is used for fine-tuning the model, testing the model on Arm Virtual Hardware, and deploying the small language model to edge devices, including a feedback collection mechanism.
Figure: Architecture Diagram of the Generative AI-Based Automotive User Guide Solution
The numbered items in the figure correspond to the following:
Model Fine-Tuning: The AWS demonstration application development team selected TinyLlama-1.1B-Chat-v1.0 as its base model, which has been pre-trained for conversational tasks. To optimize the driver’s automotive user guide chat interface, the team designed concise and focused responses to accommodate the limited attention drivers can spare while driving. The team created a custom dataset containing 1,000 question-answer pairs and fine-tuned it using Amazon SageMaker Studio.
Storage: The fine-tuned small language model is stored in Amazon Simple Storage Service (Amazon S3).
Initial Deployment: The small language model is initially deployed to an Ubuntu-based Amazon EC2 instance.
Development and Optimization: The team developed and tested the generative AI application on the EC2 instance, using llama.cpp for small language model quantization and applying the Q4_0 scheme. KleidiAI optimization was pre-integrated with llama.cpp. Meanwhile, the model achieved significant compression, reducing the file size from 3.8 GB to 607 MB.
Virtual Testing: The application and small language model were transferred to the virtual Raspberry Pi environment of Arm Virtual Hardware for initial testing.
Virtual Validation: Comprehensive testing was conducted in the virtual Raspberry Pi device to ensure functionality.
Edge Deployment: The generative AI application and small language model were deployed to physical Raspberry Pi devices using AWS IoT Greengrass Lite, with deployment management facilitated by AWS IoT Core jobs.
Deployment Orchestration: AWS IoT Core is responsible for managing tasks deployed to the edge Raspberry Pi devices.
Installation Process: AWS IoT Greengrass Lite handles the downloading of software packages from Amazon S3 and completes the installation automatically.
User Interface: The deployed application provides voice-based interaction capabilities for end users on the edge Raspberry Pi device.
Quality Monitoring: The generative AI application implements quality monitoring of user interactions. Data is collected through AWS IoT Core, processed via Amazon Kinesis Data Streams and Amazon Data Firehose, and then stored in Amazon S3. OEMs can monitor and analyze the data through Amazon QuickSight dashboards to promptly identify and address any quality issues with the small language model.
Next, we will delve deeper into KleidiAI and the quantization scheme adopted in this demonstration.
Arm KleidiAI
Arm KleidiAI is an open-source library designed for AI framework developers. It provides optimized performance-critical routines for Arm CPUs. This open-source library was initially launched in May 2024 and now offers optimizations for matrix multiplication across various data types, including 32-bit floating point, Bfloat16, and ultra-low precision formats such as 4-bit fixed point. These optimizations support multiple Arm CPU technologies, such as SDOT and i8mm for 8-bit computations, and MLA for 32-bit floating point operations.
With four Arm Cortex-A76 cores, the Raspberry Pi 5 demonstration utilized KleidiAI’s SDOT optimization, which is one of the first instructions designed for AI workloads based on Arm CPUs, introduced in Armv8.2-A released in 2016.
The SDOT instruction also demonstrates Arm’s ongoing commitment to enhancing AI performance on CPUs. Following SDOT, Arm has gradually introduced new instructions for running AI on CPUs, such as i8mm for more efficient 8-bit matrix multiplication and Bfloat16 support, aiming to improve 32-bit floating point performance while halving memory usage.
For the demonstration conducted using the Raspberry Pi 5, a block quantization scheme was employed, utilizing integer 4-bit quantization (also known as Q4_0 in llama.cpp) to accelerate matrix multiplication, with KleidiAI playing a key role.
Q4_0 Quantization Format in llama.cpp
The Q4_0 matrix multiplication in llama.cpp consists of the following components:
The left-hand side (LHS) matrix stores activation content in the form of 32-bit floating point values.
The right-hand side (RHS) matrix contains weights in 4-bit fixed point format. In this format, the quantization scale is applied to a block of data consisting of 32 consecutive 4-bit values, encoded using 16-bit floating point values.
Thus, when referring to 4-bit integer matrix multiplication, it specifically pertains to the format used for weights, as illustrated in the figure below:
At this stage, neither the LHS nor RHS matrices are in 8-bit format. How does KleidiAI utilize the SDOT instruction designed for 8-bit integer dot products? Both input matrices must be converted to 8-bit integer values.
For the LHS matrix, an additional step is required before the matrix multiplication routine: dynamic quantization to 8-bit fixed point format. This process dynamically quantizes the LHS matrix to 8 bits using a block quantization scheme, where the quantization scale is applied to a block of 32 consecutive 8-bit integer values, stored as 16-bit floating point values, similar to the 4-bit quantization method.
Dynamic quantization minimizes the risk of accuracy loss, as the quantization scale factor is calculated during inference based on the minimum and maximum values in each data block. In contrast, the scale factor for static quantization is predetermined and remains constant.
For the RHS matrix, no additional steps are required before the matrix multiplication routine. In fact, 4-bit quantization serves as a compressed format, while the actual computation is performed in 8 bits. Therefore, before passing the 4-bit values to the dot product instruction, they are first converted to 8 bits. The computational cost of converting from 4 bits to 8 bits is minimal, as it only requires simple shifting/masking operations.
Given the high efficiency of the conversion, why not use 8 bits directly and avoid the conversion hassle?
Using 4-bit quantization offers two key advantages:
Reducing Model Size: Since 4-bit values require only half the memory of 8-bit values, this is particularly beneficial for platforms with limited available RAM.
Enhancing Text Generation Performance: The text generation process relies on a series of matrix-vector operations, which are often constrained by memory limitations. In other words, performance is limited by the speed of data transfer between memory and the processor, rather than the computational capability of the processor. Since memory bandwidth is a limiting factor, reducing data size can minimize memory traffic, significantly improving performance.
How to Combine KleidiAI with llama.cpp?
It is very simple; KleidiAI is integrated into llama.cpp. Therefore, developers do not need additional dependencies to fully leverage the performance of Arm CPUs based on Armv8.2 and later architecture versions.
The integration of the two means that developers running llama.cpp on mobile devices, embedded computing platforms, and servers based on Arm architecture processors can now experience better performance.
Are There Other Options Besides llama.cpp?
For running large language models on Arm CPUs, while llama.cpp is a great choice, developers can also utilize other high-performance generative AI frameworks optimized with KleidiAI. For example (listed alphabetically): ExecuTorch, MediaPipe, MNN, and PyTorch. Just choose the latest version of the framework.
Therefore, if developers are considering deploying generative AI models on Arm CPUs, exploring the above frameworks can help optimize performance and efficiency.
Conclusion
The integration of SDVs and generative AI is ushering in a new era of automotive innovation, making future vehicles smarter and more user-centric. The in-vehicle generative AI application demonstration introduced in this article, optimized by Arm KleidiAI and supported by AWS services, showcases how emerging technologies can help address real challenges in the automotive industry. This solution achieves response times of 1 to 3 seconds and reduces development time by several weeks, proving that more efficient and offline-capable generative AI applications are not only achievable but also highly suitable for in-vehicle deployment.
The future of automotive technology lies in creating solutions that seamlessly integrate edge computing, IoT capabilities, and AI. As vehicles continue to evolve and software becomes increasingly complex, potential solutions like the one presented in this article will be key to bridging the gap between advanced automotive features and user understanding.