Why Major Companies Use Yocto?

Source: Public Account [Osprey Talks Microcontrollers]

Author: Osprey

ID: emOsprey

What is Yocto? Many friends are probably confused when they hear about it for the first time. When I first learned about this technology from my boss, I had no idea what it was and could only feel my way forward.

The Bitter History of Buildroot

Previously, my boss had just asked me to learn Buildroot to compile kernel images because we had an old project that used it for image compilation.

In the past, these images were maintained by foreigners (I currently work in a Fortune 500 foreign company, and if you want to chat about what I learned during these years in a foreign enterprise, feel free to follow Osprey Talks Microcontrollers), and we only needed to fetch the images from a designated location and use tools to burn the SD card images.

Later, my boss wanted us to maintain the images domestically and compile them ourselves, so he asked me to learn image compilation (I had talked with my boss about my career direction, and he knew I wanted to develop in the Linux direction, so he gave me this opportunity).

I had learned about Buildroot many years ago and had compiled it, thinking it wouldn’t be particularly difficult, and that I could complete the task quickly.

But unexpectedly, it took me more than half a month to get it done, and the most memorable part was downloading around 2.5 GB of open-source resources. I had never encountered such issues before; it usually took me half a day to a day to complete.

Looking back now, there are two reasons for this:

1. Back then, GitHub was accessible, and many foreign resources could be easily downloaded.

2. The current images rely on too many and too old open-source libraries. Previously, I only needed the basic components to learn the Linux kernel.

Thus, I smoothly compiled uboot, the kernel, and the file system (the three major components of the image).

Now, some resources download too slowly, and I had to search for various resources, eventually transferring from Gitee, and finally completed the downloads (luckily, offline installation was supported).

However, I still couldn’t install the qt functionality in the end due to some unknown reason, but it wasn’t important, so I didn’t pursue it.

Additionally, many resources have specified hash values, and you can only download resources with the same hash value. Otherwise, even if you modify the hash value to adapt to new resources, it might lead to system incompatibility.

Furthermore, the installation of the Buildroot environment itself was a bit troublesome, and it took some time to get familiar with it.

The Suffering History of Yocto

After switching to Yocto development, I found that the learning difficulties increased tenfold compared to Buildroot, and the resource (disk space) consumption was also about tenfold. However, once familiarized, it felt like it was truly born for embedded systems.

At that time, my Ubuntu version was 18.04 because an old project needed this version to compile normally.

However, the recommended system for Yocto should be 20.04 or higher, with at least 120 GB of free disk space:

IMX_YOCTO_PROJECT_USERS_GUIDE.pdf

I didn’t want to reinstall the Ubuntu system (at that time, I still didn’t know about using Docker), so I took a detour and encountered various software compatibility issues, such as with gcc (I could only download the source code and reinstall) and python, where some recommended software versions couldn’t be installed smoothly. The repo tool itself also needed to use domestic mirrors due to the network environment in China.

In the end, when generating the SDK, I got stuck at the last step and couldn’t find the dnf Python tool. Luckily, I generated the toolchain through another command; otherwise, it would have been a big problem if I got stuck at the last step of compiling the image (later Docker solved these pain points).

bitbake meta-toolchain

In short, there were many problems.In a few days, although the development environment issues were resolved (with Docker, it could be done in half a day), like Buildroot, I encountered download issues again, and this time it was even worse, directly 30 GB of resources to download (because Yocto is rarely used in China, and even when it is, most use imx6ull, so it’s hard to find suitable resources).

This left me with no temper; I could only seek help from the vendor and got the resources they had downloaded in advance.

With the resources in hand, I could finally start compiling.

The compilation process went relatively smoothly, but compared to Buildroot, it was extremely slow, and resource consumption was enormous (a very simple process might require hundreds of MB of resources, which shows why the entire image needs more than 100 GB of space).

Here’s a very useful command that allows other tasks to continue running without immediately interrupting when an error occurs.

bitbake xxx -k

Additionally, I recommend a Yocto introductory video: Bilibili:Building Embedded Linux Systems with Yocto for iMX6ULL

In summary, after setting up the development environment and pre-downloading resources, the compilation went quite smoothly.

All three major components were accounted for, and I could also generate SD card images.

Why is Yocto so difficult to learn, consumes so many resources, yet so many major companies use it?

For instance, NXP, BMW, Amazon, Facebook (Meta), General Electric, and others.

For modern computers, resource consumption is no longer a major issue; maintainability, flexibility, customizability, compatibility, and ecosystem are the primary considerations. And Yocto meets all these requirements.

Flexibility and Customizability: The Yocto Project provides a flexible toolkit and development environment, allowing developers to freely choose resource usage or improve specific components (including specific versions of these components). This flexibility enables developers to customize Linux distributions according to their needs, thus meeting the requirements of specific hardware platforms. Avoiding Redundant Work: The Yocto Project encourages innovation across various applications and device types, supporting multiple hardware architectures such as x86, ARM, RISC-V, PPC, and MIPS. By sharing technologies, software stacks, configurations, and best practices, the Yocto Project helps developers avoid starting from scratch, reducing redundant work and unnecessary maintenance. Powerful Build System: The core of the Yocto Project is BitBake, an efficient task scheduler and execution system capable of handling complex dependencies and optimizing the build process. BitBake ensures that build tasks are executed in the correct order, thus improving build efficiency and predictability. Rich Ecosystem: The Yocto Project has an active community and abundant documentation resources, providing strong support for developers. Community members include hardware manufacturers, operating system vendors, and independent consultants who contribute resources and information to form a robust ecosystem. Standardization and Compatibility: The Yocto Project offers standardized build processes and tools, which facilitate collaboration among different developers and teams. By using the Yocto Project, developers can ensure that their customized Linux systems have a high level of compatibility and portability across different hardware platforms. Optimized Resource Utilization: The Yocto Project allows developers to precisely create the content needed for embedded devices, adding only the necessary functional support or packages. This helps optimize system performance, reduce unnecessary resource usage, and provide more efficient solutions for embedded and IoT devices. Maintainability and Scalability: The design of the Yocto Project allows users to use self-developed operating systems or commercial operating systems without losing optimizations or functionalities during migration. Additionally, commercial operating systems can also use the Yocto Project as upstream resources, ensuring maximum code reuse. Supporting Multiple Build Targets: The Yocto Project can not only be used to create Linux distributions but also generate real-time operating system (RTOS) toolchains for bare-metal development, as the Zephyr project does.

The above is the answer provided by AI.

Next, let’s talk about why the Yocto method is great based on Osprey’s actual development experience.

1. Development Environment

The Docker technology solved the environment setup issue, so environment setup is no longer a problem.

Years of ignorance about the benefits of Docker

2. Can Compile Various Codes

We know that a project involves various programming languages, such as C/C++, Python, Java, etc., and various platforms like u-boot, Linux, MCU, qt, and various architectures, arm, risc-v, x86, etc., while Yocto can compile all these codes and package them into a compressed file for direct use.

Moreover, the same set of meta can easily compile for both x86 and arm, allowing some development and debugging work to be done even without an arm board, making automated testing easier.

3. Supports QEMU Debugging

Native support for QEMU debugging, although I haven’t quite figured this out yet; once understood, it would greatly enhance development efficiency.

4. Customization

This is probably the reason why various major companies favor Yocto.

Often, our work is based on others’ foundations; for example, I want to add a qt interface in Ubuntu.

In this case, if there’s already a ready-made Ubuntu, it’s much easier to add qt.

Similarly, the Yocto project contains many Linux distributions with various components (possibly hundreds or thousands), allowing for self-service addition or deletion of certain components, and these component versions are maintained and updated by the community.

For example, a specific version of the tool vim might require a certain version of the Linux kernel or above; the compatibility of these commonly used tools will be managed, and you only need to download a poky (the reference implementation of Yocto) distribution, specify a version, and you can compile the entire image (although downloading may take some time again, as the versions of dependent components may differ and require re-downloading).

Chip manufacturers will also add their special customizations based on these foundations.

In addition to the overall image customization, individual components or lines of code can also be modified.

The most common modifications include device trees, kernel code, and drivers, which can all be added as patches to the base Linux code (without modifying the source code).

Moreover, upgrading the Linux kernel to the latest version is also straightforward. Typically, poky will synchronize the latest kernel and upgrade the dependent components simultaneously, so your project modifications may only require a single commit change to Poky (if kernel code is modified, minor adjustments may be needed), but components like vim, nano, ssh, etc., will be upgraded without requiring your attention.

Yocto-generated images also come in various choices, such as:

core-image-minimal: This is a very basic image that includes necessary system tools and the Linux kernel, suitable as a foundation for customizing more complex systems. core-image-sato: This is a standard image optimized for desktop environments, containing more software packages and tools, suitable for desktop or display devices. core-image-full-cmdline: This image includes a complete command-line toolset but does not contain a graphical interface, suitable for embedded devices requiring a full command-line environment. core-image-multimedia: This image includes multimedia-related software packages, suitable for devices requiring audio and video processing capabilities.

You can choose to add custom functionalities based on these image foundations.

If you dislike the code of some components, you can modify it at any time; such modifications will not change the source code but will be applied as patches to another compilation environment, ensuring the purity of the source code.

5, Project Practicality

If you have a Linux board V1, with code adapted to the current hardware, and find some issues that require hardware changes, resulting in V2, there may also be other MCU codes that need to be modified concurrently, while the old version board is still in use.

How to handle these dependency issues?

The old method is to manually maintain the dependency differences between two environments; a more advanced approach is to use scripts or repo tools for maintenance.

However, Yocto, with the help of the kas tool, can perfectly handle such situations. The meta repository can contain information for two types of boards on the same branch; as long as you set up the required machine type before compilation, the corresponding dependencies can be handled properly, eliminating the occurrence of Linux and MCU incompatibility.

Furthermore, using kas, you can make differential modifications for V2 based on V1, simply by adding the differences, avoiding redundant work or copying or branching.

6. SDK

You can easily generate various SDKs, suitable for upper-layer application development after installation, without occupying too much space, so you won’t need to install the entire Yocto development environment for just a small functionality.

Standard SDK: This SDK contains artifacts for application development, whether for bootloaders, Linux kernel development, or other user-space software. It provides a cross-development toolchain and libraries optimized for specific images. You can use the standard SDK to independently develop and test code intended to run on a target machine. Extensible SDK: In addition to all the features of the standard SDK, the extensible SDK also provides tools that allow you to easily add new applications and libraries to images, modify the source code of existing components, test changes on target hardware, and seamlessly integrate applications into the OpenEmbedded build system. The extensible SDK can be installed on any machine and can be used to develop applications, images, and kernels. Native SDK: This SDK is intended to run on the target device, suitable for embedded devices powerful enough to be used as a development environment. The native SDK includes headers, libraries, and toolchains, allowing for direct application development and testing on the target device. Cross-development SDK: This SDK is used to generate binaries for the target machine within the development host. It contains cross-compilation toolchains, libraries, headers, and binaries, along with other utilities and applications.

This is truly a comprehensive mix, containing all the resources needed for embedded software development, compilation, and debugging; it’s all taken care of for you, and you deserve it.

7. Ecosystem

There are many open-source metas that can be used directly, and many chip manufacturers also have adaptations for the Yocto environment, with numerous developers maintaining Yocto, resulting in a robust ecosystem.

Lastly, let’s discuss the drawbacks.

1. High resource consumption (each task has a separate compilation environment).

2. The initial compilation speed is very slow because it needs to compile various tools on the host.

3. Steep learning curve, but suitable for maintaining complex projects.

4. Deep resource nesting, which is not easy to find or understand during the compilation process.