Detailed Guide to Embedded Linux Development on Raspberry Pi 4

• 1. Overview

• 2. Overview of the Development Environment

• 2.1 Installing the Virtual Machine Environment

• 2.2 Setting Up the Raspberry Pi Development Environment

• 3. Installation of Cross-Compilation Tools and Compilation of U-Boot

• 3.1 Installing the ARM 64-bit Cross-Compilation Environment

• 3.2 Compiling U-Boot for Raspberry Pi

• 3.3 Running U-Boot on Raspberry Pi

• 4. Compiling and Downloading Linux on Raspberry Pi 4B

• 4.1 Compiling Raspberry Pi Linux Source Code

• 4.2 Running the Compiled Linux Firmware

• 5. Using the Root File System

• 5.1 Setting Boot Options in U-Boot

• 5.2 Inserting the SD Card into the Virtual Machine

• 5.3 Modifying File Scripts

• 6. Summary

1. Overview

This article will describe the thoughts behind developing an embedded Linux system through the boot process of Raspberry Pi 4. By following the boot process of Raspberry Pi 4B, we will observe a Linux boot process and gradually build a complete embedded Linux development environment on Raspberry Pi to guide the analysis of the development process of each part.

Through reading this article, one can grasp some methods for embedded Linux development and environment setup, and also gain a general understanding of the operating processes of Raspberry Pi 4 and Linux. After having a clear understanding of the entire running process from romboot to uboot to kernel to rootfs, it will be clearer to learn Linux and embedded low-level development.

2. Overview of the Development Environment

Embedded software is unique as it needs to compile machine code that can run on embedded platforms using a PC, which requires the use of a cross-compilation toolchain. During Linux development, the host machine is used for cross-compilation, and the generated target code is downloaded to the machine for execution.

Generally, the connection channels between the development board and the PC are serial and network cable. UART can display basic debug information, while the network cable can be used for file transfer between the board and the computer.

The serial connection is as follows:

Generally, the network cable connection allows Raspberry Pi and PC to be in the same network segment.

When developing in the same network segment, it is quite convenient. Using Linux for embedded Linux development is more convenient. Since most learning and work are done under Windows, it is suggested to install a virtual machine under Ubuntu for development work.

2.1 Installing the Virtual Machine Environment

On Windows, you can install VMware Workstation 16 Player or Oracle VM VirtualBox as the virtual machine environment.

2.1.1 Image Download

For example, download the software VMware Workstation 16 Player and then download the specific version from the official Ubuntu website.

You can download from domestic mirrors, which is faster.

http://mirrors.163.com/ubuntu-releases/20.04/

Select a specific version for installation.

Next, start installing Ubuntu 20.04.

Then set the username and password.

Select the size of the virtual machine hard disk space; for convenience, set it to 40G.

Then click Finish to start the installation.

After a while, you will enter the automatic installation interface without any intervention.

After the installation is complete, it will automatically start.

2.1.2 Installing Necessary Packages

The following is a list of packages that need to be installed:

git

First, install git.

sudo apt install git

To manage the project, first create a GitHub account and then configure git’s username and password.

git config --global user.name "YOUR_FIRST_NAME YOUR_LAST_NAME"
git config --global user.email "YOUR_GIT_ASSOCIATED_EMAIL"

net-tools

Input the following command to install:

sudo apt install net-tools

To ensure the network environment is in the same segment, you need to set the network card to bridge mode.

It is important to select your own network adapter when choosing.

On Ubuntu, input ifconfig, and on Windows, input ipconfig. As long as the front segments are the same, the last part can be different. This way, you can proceed to the next step.

tftp

TFTP (Trivial File Transfer Protocol) is a simplified version of FTP, suitable for simple scenarios, such as transferring files to the lower machine during embedded development.

The purpose of installing tftp is to facilitate development. On Raspberry Pi, the storage medium is an SD card. If you need to unplug and plug the SD card every time after compiling, and then install the Linux firmware, it is very cumbersome. Here, you can use uboot to load the Linux firmware via tftp.

The installation process is as follows:

First, install the reset program:

sudo apt install tftpd-hpa
sudo apt install tftp-hpa

Check the status of the server:

sudo systemctl status tftpd-hpa

Open the configuration file:

sudo vim /etc/default/tftpd-hpa

The edited content is as follows:

TFTP_USERNAME="tftp" # Account used by tftpd program

TFTP_DIRECTORY="/srv/tftp" # Directory

TFTP_ADDRESS=":69" # Port

TFTP_OPTIONS="--secure --create" # --secure not set will have cross-directory issues, --create is to add the right to write data to the client

Set the permission for the access directory:

sudo chown tftp:tftp /srv/tftp

Restart the tftp server:

sudo systemctl restart tftpd-hpa

Local test of the tftp server:

1. First, create a file abc.txt in the /srv/tftp directory.

sudo vim /srv/tftp/abc.txt

2. Input tftp 127.0.0.1

If the above phenomenon appears, it indicates that the test is successful.

Remote testing:

Ensure that the tftp server is functioning correctly. You can download a software called Tftp64 on Windows. Open it, select tftp client, and choose the file to transfer.

Select the put button, and the command below will pop up indicating success. If it fails, pay attention to the computer firewall settings.

Check the files in the tftp server; if you can see the files, it indicates that the tftp environment has been set up successfully.

nfs

The purpose of installing nfs is that when developing applications on Linux, you do not want to frequently transfer files. After compiling the application on the host machine, you can directly copy it to the local directory. The Linux on the embedded platform can access the freshly compiled program on the host machine via the nfs file system, making the development of Linux applications more convenient.

sudo apt install nfs-kernel-server

Then create a shared directory for nfs to store shared files:

sudo mkdir /opt/nfs

Modify the configuration file:

sudo vim  /etc/exports

/opt/nfs 10.1.1.* (rw,async,root_squash)

Edit the file in /etc/exports as follows:

Where 10.1.1.* is the address information of your network segment.

Finally, restart nfs:

sudo systemctl restart nfs-kernel-server

Test the installation of nfs:

After installation, you can install the nfs client:

sudo apt install nfs-common

Then create a local folder:

sudo mkdir -p /opt/nfs-client

Finally, mount it:

sudo mount -t nfs 10.1.1.160:/opt/nfs  /opt/nfs-client

When creating a file in /opt/nfs, you can also see it in the /opt/nfs-client directory, indicating success.

2.2 Setting Up the Raspberry Pi Development Environment

2.2.1 Hardware Connection

The actual hardware pin distribution on Raspberry Pi 4 is shown in the above figure, where the serial ports RX, TX, and GND need to be connected.

Prepare an SD card of 8G or more, then open Raspberry Pi Imager and choose the Raspberry Pi image to burn into it.

The purpose of this step is because the Raspberry Pi boot process needs to load the first stage boot file from the SD card.

By default, the burned firmware does not output when connected to the serial port; you need to modify the config.txt file on the SD card. Just add the following line at the end:

enable_uart=1

Then, by using the mobaxterm tool to open the serial port and connect the power supply, you can see the output as follows:

Then input the username and password as follows:

raspberrypi login:pi
Password:raspberry

Now you can use the default Raspberry Pi 4 serial debugging function.

2.2.2 Analysis of the Raspberry Pi 4B Boot Process

Briefly, the boot process of Raspberry Pi 4B is as follows: after powering on, the Raspberry Pi automatically loads the bootcode.bin file located in the SD card. This file is loaded into the GPU of the Raspberry Pi to run, initializing PLL, DDR, etc., and then reads the start4.elf file from the SD card to execute. During the execution of this file, it reads the config.txt file and processes the boot flow according to the configuration script.

When doing embedded development, the underlying boot logic needs to be very clear, so that one can sort out the problems in any situation, ensuring consistency between hardware and software layers.

The above figure basically shows a general embedded Linux boot process, with good descriptions of the characteristics and functions of each stage.

The boot process for the Broadcom BCM2711 on Raspberry Pi 4B follows these steps:

First Stage Bootloader:

• The first stage BootROM is generally hard-coded inside the chip and executed in the GPU, while the ARM core is in a reset state.
• The BootROM of Raspberry Pi 4B is loaded via EEPROM, while previous versions loaded from the bootcode.bin file on the SD card.

Second Stage Bootloader:

• This stage of the boot firmware can be loaded from SD card, network, USB, etc.
• On Raspberry Pi 4, the start.elf binary file in the SD card is used.
• The start.elf file finds the config.txt file in the SD’s file system and processes the boot flow according to the information inside.

This article mainly modifies the config.txt configuration file to perform the uboot boot process.

3. Installation of Cross-Compilation Tools and Compilation of U-Boot

3.1 Installing the ARM 64-bit Cross-Compilation Environment

Since we need to compile 64-bit programs, we need to install the ARM 64-bit cross-compilation environment.

https://www.linaro.org/downloads/

Enter the above website.

It is recommended to download gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz.

After downloading, place it in the specified directory.

# Create a folder
sudo mkdir -p /opt/linaro
# Unzip to the specified folder path
sudo tar -xvf gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar -C /opt/linaro

Update the environment variables:

 sudo vim ~/.bashrc

Add the following at the end:

alias crosscompiler='export KERNEL=kernel8;export ARCH=arm64;export CROSS_COMPILE=/opt/linaro/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-'

Update the environment:

source ~/.bashrc

3.2 Compiling U-Boot for Raspberry Pi

First, download the code:

 git clone https://github.com/u-boot/u-boot.git

The normal download completion image is as follows:

Next, switch branches and start compiling:

cd u-boot
git checkout v2020.04-rc3

Before compiling, you need to install the necessary programs:

sudo apt install u-boot-tools bison bc make flex libssl-dev ncurses-*

After installation, execute:

crosscompiler

This command is the one defined in the environment variable to set the environment variables.

 make rpi_4_defconfig

Directly use the default configuration to compile:

make -j $(nproc)

The compiled uboot.bin file is the program that can be executed directly on Raspberry Pi 4B.

3.3 Running U-Boot on Raspberry Pi

At this point, you can run the compiled U-Boot program on Raspberry Pi 4B.

Insert the Raspberry Pi SD card into the computer, rename the config.txt file in the SD card to config.txt.bak, and create a new config.txt.

arm_control=0x200
kernel=u-boot.bin
dtoverlay=disable-bt

The meanings of these three lines are:

arm_control=0x200 # Because Raspberry Pi 4B supports 64-bit architecture, this indicates running 64-bit programs

Using U-Boot:

kernel=u-boot.bin # kernel indicates the firmware to be run

Enable the serial port:

 dtoverlay=disable-bt # When the serial debugging is enabled, Bluetooth cannot be used

Insert the SD card into the computer, and you can see that U-Boot starts normally.

4. Compiling and Downloading Linux on Raspberry Pi 4B

4.1 Compiling Raspberry Pi Linux Source Code

Now that the U-Boot functionality for Raspberry Pi 4B has been completed, we will start compiling the Linux kernel for Raspberry Pi.

Raspberry Pi maintains a separate branch for Linux code, which can be downloaded using the following command:

git clone --branch rpi-5.6.y https://github.com/raspberrypi/linux

Enter the Linux directory:

Create a new directory to store the compiled firmware:

mkdir rpi_hw

Start compiling:

make O=rpi_hw bcm2711_defconfig

Remove the MMC/SD/SDIO driver:

make O=rpi_hw menuconfig

Enter Device Driver and select to remove MMC/SD/SDIO card support.

After saving the configuration, you can start compiling:

make O=rpi_hw -j $(nproc)

Why remove the MMC/SD/SDIO driver?

This is because we need to compile the driver for network boot, so there is no need to operate on the SD card of the Raspberry Pi.

After the compilation is complete, you can find the compiled files in rpi_hw/arch/arm64/boot.

Copy the compiled Linux kernel file to:

 sudo cp rpi_hw/arch/arm64/boot/Image /srv/tftp/

4.2 Running the Compiled Linux Firmware

During the period of loading the compiled firmware into RAM via U-Boot, you need to understand the memory distribution of Raspberry Pi 4B.

Using the bdinfo command in U-Boot, you can see that there are two banks on Raspberry Pi 4B, the first bank at 0x00000000 and the second at 0x40000000.

When loading the Image file from the SD card, it starts running at the address 0x8000 in the DRAM of Raspberry Pi 4B.

Of course, the address can also be set in U-Boot, and Linux will relocate the code.

At this point, you need to set the boot information in U-Boot.

First, set the static IP address of U-Boot:

setenv ipaddr 10.1.1.100 # Set the static address of the development board (customized)
setenv serverip 10.1.1.160 # Set the server address
setenv netmask 255.255.255.0
saveenv
reset

The above operations set the IP address, and next, set the boot:

setenv kernel_addr_r 0x8000
setenv kernel Image
setenv netboot 'tftp ${kernel_addr_r} ${kernel} && booti ${kernel_addr_r} - ${fdtcontroladdr}'
setenv bootcmd 'run netboot'
setenv bootargs 'console=ttyAMA0'
saveenv
reset

You can see the following:

After starting, an error will be reported:

This is normal; currently, there is no rootfs. However, the Linux kernel can be loaded and debugged normally.

Next, let’s mount the rootfs.

5. Using the Root File System

This article does not discuss the process of creating a general root file system; it mainly describes how to use it.

5.1 Setting Boot Options in U-Boot

First, set the path in uboot:

setenv nfsroot /opt/nfs/

Set the boot parameters:

setenv bootargs "console=ttyAMA0,115200 root=/dev/nfs rw nfsroot=${serverip}:${nfsroot},v3,tcp ip=$ipaddr:$serverip::$netmask::eth0:off"
saveenv

5.2 Inserting the SD Card into the Virtual Machine

First, mount the USB drive to the virtual machine:

You can see two disks appear:

Among them:

• rootfs is the Linux root file system
• boot is an ext32 file that can be accessed on Windows

You can copy all files from rootfs to /opt/nfs/:

sudo cp * /opt/nfs/ -R

In the /opt/nfs directory, create a folder named file and copy all files from boot into it:

sudo mkdir -p /opt/nfs/file
sudo cp * /opt/nfs/file/

5.3 Modifying File Scripts

You need to modify:

sudo vim etc/fstab

Add the following file:

10.1.1.160:/opt/nfs/file /boot nfs defaults,vers=4.1,proto=tcp 0 0

The purpose is to shield the default two items in the list, so that only the NFS file system needs to be mounted.

After the modification, insert the SD card, and you can normally obtain the Linux kernel firmware from TFTP and mount the root file system from the NFS file system.

If you find that you need a password or have forgotten the password, you can enter:

cd /opt/nfs/
sudo vim /etc/passwd

And delete the x in the middle.

In Linux, the default x indicates that a password is required to enter.

6. Summary

This article analyzes the entire environment setup of the Raspberry Pi Linux system and the booting of the Raspberry Pi. Experiments were conducted on the boot, U-Boot loading of Linux kernel, and mounting of the NFS file system.

For the part of creating the root file system, the default root file system of Raspberry Pi was used. If you need to customize and create it, you can perform tailored operations.

The entire process of embedded Linux development and environment setup can be well tested on Raspberry Pi 4B. No matter how it changes, mastering the process and tools of embedded development makes application and driver development very convenient and efficient.

Due to time constraints, there are still some experiments that have not been completed, such as application development on Linux, driver development, etc., and J-Link debugging, etc.

Learning Linux usage and booting on Raspberry Pi 4B is very convenient and reasonably priced, making it a good development platform.