Introduction:
On November 14th at 2 PM, the Black Sesame Intelligent A1000 Chip Basic Software Development Online Seminar successfully concluded its live broadcast. The seminar was hosted by two technical experts: Li Kun, Technical Application Manager at Black Sesame Intelligent, and Li Yupeng, Automotive Electronics Basic Software Planning Architect & Head of Automotive Operating System and Basic Software Planning at ZTE Communications. The topics covered were “Basic Software Development of Visual Perception Data Flow on the A1000 Platform” and “Adapting ZTE Automotive Operating System SafetyLinux on the A1000 Platform”.
This article is a整理 of the main lecture by Li Yupeng, Automotive Electronics Basic Software Planning Architect & Head of Automotive Operating System and Basic Software Planning at ZTE Communications.If you need a replay of this online seminar and the Q&A session, you can click Read the Original to watch.
Li Yupeng:Good afternoon, everyone. I am Li Yupeng from ZTE Communications Automotive Electronics, responsible for the planning of automotive operating systems and basic software. Today, I will share the topic of adapting ZTE Communications Automotive Operating System SafetyLinux on the Black Sesame Intelligent Huashan II A1000 platform.I would like to thank Black Sesame Intelligent and the platform provided by Zhixingxing for this communication opportunity, and I hope to work with all of you online to promote the construction of the basic software and hardware platform for intelligent driving and its ecosystem.Today, I will share from the following three aspects:1. The adaptation of ZTE Automotive Operating System SafetyLinux to the Black Sesame Intelligent A1000 chip2. The intelligent driving cooperation case of ZTE Automotive Operating System SafetyLinux and Black Sesame Intelligent A1000 chip
3. Product planning and solutions of ZTE Automotive Operating System
1
Matching of ZTE Automotive Operating System SafetyLinux with Black Sesame Intelligent A1000 Chip
First, let me briefly explain why we need to carry out this adaptation work.As you know, the Black Sesame Intelligent A1000 chip comes with a corresponding SDK and relevant materials, and the default target hardware for the SDK is the evaluation board provided by Black Sesame Intelligent.However, in actual application environments, different domain controllers have different requirements and configurations, and there are also corresponding requirements for the underlying operating system and software, which necessitates a third-party operating system that can adapt to the Black Sesame Intelligent A1000 platform.
ZTE SafetyLinux has been adapted to the Black Sesame Intelligent A1000 chip. So what specific adaptation work needs to be done?Before introducing the specific adaptation work, let’s first look at the main content of the Black Sesame Intelligent A1000 chip SDK, which is the basis and prerequisite for SafetyLinux to adapt to the A1000 chip.
From the materials released by Black Sesame Intelligent, we can see that the SDK for the A1000 includes a Linux kernel. Based on this kernel, it is possible to develop on-chip drivers for the A1000 SoC, as well as some kernel functions and customized user-space components.Additionally, the SDK also includes a file system closely related to the Black Sesame Intelligent A1000 chip, which includes the necessary root file system, Black Sesame Intelligent software development components, and other customized functional components. These components are quite rich.Furthermore, the SDK contains other materials that are essential for adapting to the Black Sesame Intelligent A1000 chip, such as Linux kernel source code, accompanying development toolkits, binary firmware, sample code, project templates, and accompanying development documentation.Overall, the materials provided by the Black Sesame Intelligent A1000 chip SDK are quite rich and comprehensive, and combined with the SDK and evaluation board, they provide excellent assistance and guidance for third-party application development or third-party OS porting.Next, I will introduce the specific work of adapting ZTE SafetyLinux to the Black Sesame Intelligent A1000, which mainly includes the following parts:1. Adaptation Analysis
First, it is necessary to clarify the Linux kernel version of the Black Sesame Intelligent A1000 chip SDK. Compare it with the ZTE SafetyLinux kernel to identify the modification or change points between the two kernels and BSP driver code, preparing for subsequent kernel porting and patch creation.Secondly, it is essential to clarify the kernel configuration, which means that after some driver development is completed in the Black Sesame Intelligent A1000 SDK, kernel configuration modifications need to be made to enable the BSP drivers. This is also a key focus when porting the OS to the Black Sesame Intelligent A1000 hardware platform.Next, it is necessary to distinguish the device tree configurations corresponding to the on-chip and off-chip drivers in the SDK. For example, for the on-chip device tree, we need to prepare how the ZTE SafetyLinux and the target domain controller’s device tree should be configured. Here, I will briefly mention that the main consideration for distinguishing between on-chip and off-chip device trees is that the device tree corresponding to the on-chip SoC drivers is closely related to the Black Sesame Intelligent A1000. Regardless of how the domain controller’s functions are designed, these drivers must be integrated.However, off-chip devices like smart cameras, LiDAR, or other smart peripherals often have their drivers and device tree configurations based on the requirements of the domain controller. For instance, if a domain controller does not include LiDAR, the corresponding device tree configuration may not be necessary. If the number or type of cameras varies, the corresponding device tree configuration will also differ. This is another consideration for distinguishing between on-chip and off-chip driver device trees.Finally, an analysis of the SDK compilation outputs is required. Different chip kernels generate different images after compilation, including corresponding device tree dtb, ko libraries, etc. Analyzing and clarifying these variations or differences is necessary to provide essential support for the actual kernel porting, adaptation, and compilation later. This is a fundamental method for any complex work.2. Kernel PortingNext, we look at the second step, which is the core, namely the kernel porting work of ZTE SafetyLinux adapting to the Black Sesame Intelligent A1000.
First, based on the previous difference analysis, clarify the differences between the Linux kernel version in the Black Sesame Intelligent A1000 SDK and the SafetyLinux kernel, and then create a detailed and comprehensive difference list.Then, based on this list, generate a modification list for the SafetyLinux kernel corresponding to the Black Sesame Intelligent A1000, creating patches for the SafetyLinux kernel or on-chip drivers. Here, the term “patch” refers to a general term for changed code or changed code packages, which is not entirely the same as what we usually understand as a patch. By using patches, we can easily integrate the changes or unique features from the Black Sesame Intelligent A1000 SDK into the ZTE SafetyLinux project framework (Yocto framework).Next, a crucial step is to resolve potential issues that arise when merging these kernels and on-chip drivers, as well as patches into ZTE SafetyLinux. These issues mainly arise because the kernel in the Black Sesame Intelligent A1000 SDK and the ZTE SafetyLinux kernel may have version differences, and the branches may also differ. For example, Black Sesame Intelligent may have made some features or enhancements based on the SDK, while ZTE has also made some enhancements in SafetyLinux. These enhancements need to be organically integrated to form a better version after porting. Therefore, resolving these issues is quite critical. Especially after merging, a preliminary compilation of the kernel is required, which may also lead to some exceptions. These exceptions may arise from various reasons, requiring OS kernel development team engineers to manually handle and analyze them one by one. The analysis process must be cautious, as the differences may be numerous or involve a wide range, requiring OS kernel experts to analyze and resolve them one by one. Sometimes, it may also be necessary to consider the overall features of the OS major version and provide solutions for the differences while also considering logical and performance demands.Thus, kernel porting is a key step in the entire adaptation work. Of course, the ZTE OS kernel team has years of development experience and is quite adept at kernel porting. I will also detail a successful case later.Once the kernel porting is completed, the kernel and some driver code and basic change features will be integrated into ZTE SafetyLinux.3. Kernel Configuration and Version Building
The third step is kernel configuration and version building.First, it is necessary to create patches for board-level or off-chip drivers, including functional enhancements or feature patches made by Black Sesame Intelligent. The patches mentioned here refer to integrating changes or new features, including system boot patches.Once the patches are completed, corresponding compilation and configuration need to be done in the kernel to integrate the patches into the kernel, making the changes effective. These patches may also include some default configurations, such as RT function configurations, initialization configurations, etc.After completing these configurations, a complete version of ZTE SafetyLinux adapted to the Black Sesame Intelligent A1000 can be compiled and built based on the Yocto framework. After building, preliminary testing of the version and verification of key functional processes are also required. Of course, various issues may be discovered during testing and verification, requiring us to re-conduct regression testing, testing, and version release.After this step, the basic work for adapting ZTE SafetyLinux to the Black Sesame Intelligent A1000 hardware environment is basically completed. Of course, there may be some performance optimizations, security enhancements, and information security patches that need to be considered, which should be carried out in conjunction with the entire technical software platform’s business system.
At this point, the foundational work for adapting ZTE SafetyLinux to the Black Sesame Intelligent A1000 platform is complete.
2
Cooperation Case of ZTE Automotive Operating System and Black Sesame Intelligent A1000
Next, I will introduce our specific operations for porting and the results and progress achieved by combining ZTE SafetyLinux and the Black Sesame Intelligent A1000 hardware platform.ZTE SafetyLinux and Black Sesame Intelligent A1000 have a good cooperation case in the L2+ autonomous driving scenario.
The diagram above shows our cooperation scheme architecture. As can be seen, the underlying architecture is a single Black Sesame Intelligent A1000 SoC chip, equipped with a high-performance MCU to perform intelligent driving L2+ NOA and parking functions. The smart peripherals adopt a 5R10V configuration, along with other associated components and peripherals.On the Black Sesame Intelligent A1000 SoC chip (A core), the ZTE SafetyLinux operating system is deployed, using the LTS long-term evolution branch version 5.10, supporting RT patches, and integrating ZTE’s enhancements for vehicle scenarios in computing, memory, storage, file systems, networks, maintenance, and containers.At the basic software layer of the solution architecture, it includes AUTOSAR AP/CP, DDS, SOA functional frameworks, and functional safety monitoring frameworks, etc.The application layer includes intelligent driving business applications implemented together with third-party and OEM partners, such as the previously mentioned driving and parking assistance.The above is a brief introduction to the overall cooperation scheme of ZTE SafetyLinux and Black Sesame Intelligent A1000.
In this scheme, one of the core functions is the driver adaptation of the underlying smart peripherals. As can be seen from the above diagram, the main peripheral is the camera, but it also includes radar, GNSS, IMU, and other devices that are not shown, which also need to be integrated into the entire system. Additionally, the drivers for NorFlash, eMMC, and Ethernet also need to be integrated.In the integration of the camera driver, besides the camera itself, the integration and adaptation of the corresponding deserializer for the board-end camera, as well as the configuration of the corresponding device tree, are also key tasks we need to perform. In addition, the previously mentioned GNSS, IMU time synchronization peripherals and process integration support, as well as the secure boot framework ATF and uboot integration. Other peripheral integrations, such as millimeter-wave radar, may be carried out on different SoCs or MCUs. The above describes the integration and adaptation configuration of smart peripherals in the entire case.
In addition to the underlying drivers, the integration of Black Sesame Intelligent ISP, AI framework, and third-party components is also essential. As the underlying basic software and hardware platform, it must provide necessary support for upper-layer applications and basic software frameworks.The Black Sesame Intelligent ISP and AI framework mainly include image transformation acceleration libraries, video encoding libraries, deep learning libraries, camera ISP processing libraries, IPC communication libraries, display libraries, GPU driver libraries, etc., and also include the integration of Black Sesame Intelligent AI operator libraries. Some of these libraries are also core independent intellectual property rights of Black Sesame Intelligent, released externally in the form of libraries.During the integration process with ZTE SafetyLinux, to ensure the consistency of kernel code and programs, we also requested Black Sesame Intelligent to compile these libraries based on ZTE’s OS toolchain, which received excellent support and cooperation from Black Sesame Intelligent.In terms of third-party library integration, it includes basic software libraries and third-party libraries required by the basic software framework, integrating relevant third-party libraries involved in perception, planning control, and parking business.These third-party libraries undergo security governance during the integration into ZTE SafetyLinux. Security governance refers to scanning for security vulnerabilities in third-party libraries, especially open-source libraries, and applying patches or making security enhancements to necessary libraries. After modifications, generalization or basic integration testing is conducted, which is part of what ZTE SafetyLinux needs to do to ensure that the integrated libraries can be correctly applied on the entire operating system platform without security vulnerabilities or basic functional bugs, thus providing good support for upper-layer application development.
This page shows some effects achieved in the cooperation case of ZTE SafetyLinux and Black Sesame Intelligent A1000, especially the display effects of the camera.The column above shows the output effects of the camera after driver integration, which helps to determine whether the functionality, drivers, and related components are integrated correctly.From this image, we can see that we have achieved good output effects on both the surround view camera and the panoramic camera. Of course, in different external environments, such as under good outdoor lighting conditions or in dim indoor scenes, both the surround view and panoramic cameras also perform well.In the initial results of the camera, some cropping or distortion correction processing is also implemented based on the software components provided by Black Sesame Intelligent. From this image, we can see that good integration has been achieved, resulting in favorable outcomes.The row below shows higher-level applications. We use the YOLOv5 model sample to validate and test the effects of AI perception and inference. From the camera images and traffic video stream image recognition, it can be seen that both ordinary objects and vehicles on the road can be well recognized.
This also indicates that the basic software platform of ZTE SafetyLinux and Black Sesame Intelligent A1000 has achieved good results in supporting the integration of upper-layer components and business, effectively supporting the development of upper-layer applications.
3
ZTE Automotive Operating System
Product Planning and Solutions
Here, I will briefly introduce ZTE’s automotive operating system.Today, we mainly introduce ZTE’s SafetyLinux. In fact, ZTE has a relatively complete layout in the underlying OS, including microkernel operating systems, SafetyLinux systems, virtualization hypervisors, and accompanying toolchains. Through the combination of these three products, we can form a good OS solution for intelligent driving, intelligent cockpit, and intelligent vehicle control, providing solid support for typical application scenarios at the vehicle end.Additionally, ZTE’s microkernel operating system has also achieved good ecological adaptation and integration with the Black Sesame Intelligent A1000 platform, currently achieving a seamless process for AI perception inference and achieving favorable results. This is also thanks to the excellent support and cooperation from the Black Sesame Intelligent team and hardware platform.
Finally, I would like to add a brief note about ZTE’s understanding of intelligent driving operating system solutions.In fact, in addition to the intelligent driving solution jointly created by ZTE SafetyLinux and Black Sesame Intelligent A1000, we also have other intelligent driving operating system solutions.For example, the combined monitoring solution displayed on the right side of the above image. This solution mainly considers the functional safety enhancement of SafetyLinux. Since ZTE’s microkernel operating system has passed the ISO26262 ASIL-D management process and product certification, it can provide a good foundation for functional safety certification. Through this combined monitoring, the functional safety characteristics of the entire intelligent driving solution can be further enhanced.In terms of long-term solution development, ZTE also has its own ideas. We analyze that in the future, we will base the underlying OS for intelligent driving on an enhanced microkernel, mainly considering that the public safety requirements for high-level autonomous driving will significantly increase. ZTE’s microkernel operating system can meet the ASIL-D functional safety requirements, thus providing conditions to achieve a high functional safety level for the entire intelligent driving underlying OS solution or the entire intelligent driving solution. This is also ZTE’s basic idea for the evolution of intelligent driving.That concludes my presentation today. Thank you all.