Setting Up an ARM64 Linux Environment with QEMU

QEMU (Quick EMUlator) is a general-purpose open-source emulator and virtualization tool that can simulate most CPU platforms, such as x86, ARM, ARM64, RISC-V, PowerPC, MIPS, etc.

It can also simulate various hardware peripherals, such as memory, emmc, sdcard, usb, etc. Therefore, through QEMU, we can simulate a development board.

During the chip development process, chip companies usually use QEMU for preliminary software development and debugging before the chip returns. In this article, we will introduce how to simulate the ARM64 platform with QEMU and run the linux kernel, thus facilitating kernel development and debugging.

The tools and source code version information involved in this article are as follows:

Ubuntu: Ubuntu 22.04.2 LTS
qemu: 8.2.0
kernel: Linux 6.1.72
busybox: busybox-1.36.1

If you do not currently have a development environment for Ubuntu, you can refer to VMware Workstation Install Ubuntu-22.04 Virtual Machine tutorial to install the Ubuntu system.

Compile QEMU Emulator

Installation Tools

To install the tools needed to compile the QEMU source code, the commands and logs are as follows:

max@ubuntu2204:~$ sudo apt update
[sudo] password for max: 
Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy InRelease
Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-updates InRelease
……
……
Building dependency tree... Done
Reading state information... Done
407 packages can be upgraded. Run 'apt list --upgradable' to see them.
max@ubuntu2204:~$
max@ubuntu2204:~$ sudo apt -y install python3.10-venv python3-sphinx ninja-build libpixman-1-dev pkg-config flex bison libfdt-dev libglib2.0-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
……
……
Setting up python3-docutils (0.17.1+dfsg-2) ...
Setting up python3-sphinx (4.3.2-1) ...
max@ubuntu2204:~$

Download QEMU Source Code

The official website of the QEMU emulator is https://www.qemu.org/, the homepage is as follows:On the QEMU official website download page, you can view all the version source packages released by QEMU, as shown below:

Use wget to download the source code of the QEMU emulator, the command and logs are as follows:

max@ubuntu2204:~$ wget https://download.qemu.org/qemu-8.2.0.tar.xz
--2024-01-13 21:46:41--  https://download.qemu.org/qemu-8.2.0.tar.xz
Resolving download.qemu.org (download.qemu.org)... 143.244.51.238, 89.187.187.12, 185.180.13.211, ...
……
……
qemu-8.2.0.tar.xz                                  100%[===============================================================================================================>] 123.99M  2.54MB/s    in 58s     
2024-01-13 21:47:42 (2.14 MB/s) - ‘qemu-8.2.0.tar.xz’ saved [130008888/130008888]
max@ubuntu2204:~$

Compile QEMU Emulator

Unzip the downloaded qemu-8.2.0.tar.xz compressed package and compile the QEMU source code, the command and logs are as follows:

max@ubuntu2204:~$ tar xvJf qemu-8.2.0.tar.xz
qemu-8.2.0/
qemu-8.2.0/.patchew.yml
qemu-8.2.0/ebpf/
……
……
qemu-8.2.0/subprojects/libvhost-user/libvhost-user-glib.h
qemu-8.2.0/subprojects/keycodemapdb.wrap
qemu-8.2.0/qemu.nsi
max@ubuntu2204:~$ 
max@ubuntu2204:~$ cd qemu-8.2.0
max@ubuntu2204:~/qemu-8.2.0$ 
max@ubuntu2204:~/qemu-8.2.0$ ./configure
Using './build' as the directory for build output
python determined to be '/usr/bin/python3'
python version: Python 3.10.12
……
……
Found ninja-1.10.1 at /usr/bin/ninja
Running postconf script '/home/max/qemu-8.2.0/build/pyvenv/bin/python3 /home/max/qemu-8.2.0/scripts/symlink-install-tree.py'
max@ubuntu2204:~/qemu-8.2.0$
max@ubuntu2204:~/qemu-8.2.0$ make -j8
changing dir to build for make ""...
make[1]: Entering directory '/home/max/qemu-8.2.0/build'
ninja: no work to do.
/home/max/qemu-8.2.0/build/pyvenv/bin/meson introspect --targets --tests --benchmarks | /home/max/qemu-8.2.0/build/pyvenv/bin/python3 -B scripts/mtest2make.py > Makefile.mtest
……
……
[9342/9342] Linking target tests/qtest/dbus-display-test
make[1]: Leaving directory '/home/max/qemu-8.2.0/build'
max@ubuntu2204:~/qemu-8.2.0$

The above command will compile the emulator for all platforms supported by QEMU, which will take a long time. If you want to save compilation time, you can choose to compile only the aarch64 emulator by configuring the target-list parameter when executing the configure command, as follows:

./configure --target-list=aarch64-softmmu,aarch64-linux-user,aarch64_be-linux-user --enable-kvm --enable-debug

At this point, the QEMU emulator has been compiled, and you can check the generated emulators for each platform as follows:

max@ubuntu2204:~/qemu-8.2.0$ ls build/qemu-system-*
build/qemu-system-aarch64  build/qemu-system-hppa         build/qemu-system-microblazeel  build/qemu-system-nios2    build/qemu-system-riscv64  build/qemu-system-sparc    build/qemu-system-xtensaeb
build/qemu-system-alpha    build/qemu-system-i386         build/qemu-system-mips          build/qemu-system-or1k     build/qemu-system-rx       build/qemu-system-sparc64
build/qemu-system-arm      build/qemu-system-loongarch64  build/qemu-system-mips64        build/qemu-system-ppc      build/qemu-system-s390x    build/qemu-system-tricore
build/qemu-system-avr      build/qemu-system-m68k         build/qemu-system-mips64el      build/qemu-system-ppc64    build/qemu-system-sh4      build/qemu-system-x86_64
build/qemu-system-cris     build/qemu-system-microblaze   build/qemu-system-mipsel        build/qemu-system-riscv32  build/qemu-system-sh4eb    build/qemu-system-xtensa

Next, you can run the build/qemu-system-aarch64 to simulate the ARM64 platform processor to run the Linux kernel.

Install ARM64 Cross Compilation Toolchain

Currently, we choose to use the linaro cross-compilation toolchain to compile uboot, kernel, busybox, etc. The download URL of the linaro cross-compilation toolchain is: https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/, the webpage is as follows:

First, we create a ~/cross-compiler directory, then download the cross-compilation toolchain compressed package gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz and unzip it. The commands and logs are as follows:

max@ubuntu2204:~$ mkdir ~/cross-compiler && cd ~/cross-compiler
max@ubuntu2204:~/cross-compiler$ wget https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz
--2024-01-14 00:07:24--  https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz
Resolving releases.linaro.org (releases.linaro.org)... 52.215.200.125
……
……
gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu. 100%[===============================================================================================================>] 112.43M  1.13MB/s    in 4m 37s  

2024-01-14 00:12:04 (416 KB/s) - ‘gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz’ saved [117896452/117896452]
max@ubuntu2204:~/cross-compiler$
max@ubuntu2204:~/cross-compiler$ tar xf gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz
max@ubuntu2204:~/cross-compiler$

Add the bin directory of this cross-compilation toolchain to the PATH environment variable as follows:

PATH=/home/max/cross-compiler/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin:$PATH

This way, we can use this cross-compilation toolchain, as follows:

max@ubuntu2204:~/cross-compiler$ aarch64-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/home/max/cross-compiler/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/../libexec/gcc/aarch64-linux-gnu/7.5.0/lto-wrapper
……
……
Thread model: posix
gcc version 7.5.0 (Linaro GCC 7.5-2019.12) 
max@ubuntu2204:~/cross-compiler$

At this point, the ARM64 cross-compilation toolchain has been installed.

Compile Linux Kernel

Install Tools

On Ubuntu, install the tools needed to compile the kernel, the commands and logs are as follows:

max@ubuntu2204:~$ sudo apt install -y build-essential libncurses-dev bison flex libssl-dev libelf-dev git
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
……
……
Setting up build-essential (12.9ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
max@ubuntu2204:~$

Download Linux Kernel Source Code

We are downloading the official latest long-term version linux-6.1.y kernel from the official Linux website. You can refer to the article on obtaining the Linux kernel source code to view all versions officially released by Linux and various fast methods for downloading kernel source code. The download command and logs are as follows:

max@ubuntu2204:~$ git clone https://mirrors.tuna.tsinghua.edu.cn/git/linux-stable.git -b linux-6.1.y
Cloning into 'linux-stable'...
remote: Enumerating objects: 12104695, done.
Receiving objects: 100% (12104695/12104695), 2.58 GiB | 6.23 MiB/s, done.
remote: Total 12104695 (delta 0), reused 0 (delta 0), pack-reused 12104695
Resolving deltas: 100% (10399156/10399156), done.
Checking objects: 100% (33554432/33554432), done.
Updating files: 100% (78744/78744), done.
max@ubuntu2204:~$

Enter the linux-stable directory to check that the downloaded kernel version is Linux 6.1.72, the command and logs are as follows:

max@ubuntu2204:~$ cd linux-stable/
max@ubuntu2204:~/linux-stable$ git branch -vv
* linux-6.1.y 7c58bfa711cb [origin/linux-6.1.y] Linux 6.1.72
max@ubuntu2204:~/linux-stable$

Compile Linux Kernel Source Code

Enter the linux-stable kernel directory, and configure the kernel options through menuconfig, the command and logs are as follows:

max@ubuntu2204:~/linux-stable$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 O=build menuconfig
make[1]: Entering directory '/home/max/linux-stable/build'
  GEN     Makefile
#
# using defaults found in arch/arm64/configs/defconfig
#

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

make[1]: Leaving directory '/home/max/linux-stable/build'
max@ubuntu2204:~/linux-stable$

Executing the above command will enter the menuconfig menu page, as shown:

In the menu, select the following options:

Device Drivers
    >  Block devices
        > RAM block device support

And adjust the “Default RAM disk size (kbytes)” to 65536, the adjusted interface is as follows:

Then select save to save and exit the menuconfig menu. At this point, the kernel options are configured. Next, start compiling the kernel image, the command and logs are as follows:

max@ubuntu2204:~/linux-stable$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 O=build -j8
make[1]: Entering directory '/home/max/linux-stable/build'
  SYNC    include/config/auto.conf.cmd
  GEN     Makefile
……
……
  LD [M]  sound/soc/tegra/snd-soc-tegra210-sfc.ko
make[1]: Leaving directory '/home/max/linux-stable/build'
max@ubuntu2204:~/linux-stable$

Check the generated Image kernel image, the command and logs are as follows:

max@ubuntu2204:~/linux-stable$ file build/arch/arm64/boot/Image
build/arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
max@ubuntu2204:~/linux-stable$

At this point, the kernel image Image has been compiled.

Compile Busybox

The official website of Busybox is https://busybox.net/ We download the source code of BusyBox version 1.36.1 from the homepage using wget, the command and logs are as follows:

max@ubuntu2204:~$ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
--2024-01-14 11:50:58--  https://busybox.net/downloads/busybox-1.36.1.tar.bz2
Resolving busybox.net (busybox.net)... 140.211.167.122
……
……
busybox-1.36.1.tar.bz2                   100%[================================================================================>]   2.41M  1.11MB/s    in 2.2s    
2024-01-14 11:51:15 (1.11 MB/s) - ‘busybox-1.36.1.tar.bz2’ saved [2525473/2525473]
max@ubuntu2204:~$

Unzip the busybox source code and enter the source code directory, then select static compilation through the menuconfig menu, that is, select the “Build static binary (no shared libs)” option. The command and logs are as follows:

max@ubuntu2204:~$ tar xf busybox-1.36.1.tar.bz2
max@ubuntu2204:~$ cd busybox-1.36.1/
max@ubuntu2204:~/busybox-1.36.1$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 menuconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/basic/split-include
……
……
*** End of configuration.
*** Execute 'make' to build the project or try 'make help'.
max@ubuntu2204:~/busybox-1.36.1$

Executing the above command will display the busybox menu, as shown:

Select the following in the menu:

Settings
    > [*] Build static binary (no shared libs)

The interface is as follows:

Then save and exit. At this point, the options for busybox are configured. Next, start compiling busybox, the command and logs are as follows:

max@ubuntu2204:~/busybox-1.36.1$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 install -j8
  SPLIT   include/autoconf.h -> include/config/*
  GEN     include/bbconfigopts.h
  GEN     include/common_bufsiz.h
……
……
--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------

Check the generated busybox executable file, the command is as follows:

max@ubuntu2204:~/busybox-1.36.1$ file _install/bin/busybox
_install/bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=a6e3639090cbb9784bbdbfb939fa6ad48b25f244, stripped

Create initrd File System Image

To facilitate the creation of the initrd file system image, we use the mk_initrd.sh script to generate the initrd.ext4 image file, which will mount the file system in this image after the kernel is running. The script is as follows:

#!/bin/bash
set -x
CURR_DIR=`pwd`
MOUNT_DIR=$CURR_DIR/mount_point
BUSYBOX_PREFIX=/home/max/busybox-1.36.1
dd if=/dev/zero of=initrd.ext4 bs=1M count=32
mkfs.ext4 initrd.ext4
mkdir -p $MOUNT_DIR
sudo mount initrd.ext4 $MOUNT_DIR
cp -arf $BUSYBOX_PREFIX/_install/* $MOUNT_DIR
cd $MOUNT_DIR
mkdir -p etc dev mnt proc sys tmp mnt etc/init.d/
echo "proc /proc proc defaults 0 0" > etc/fstab
echo "tmpfs /tmp tmpfs defaults 0 0" >> etc/fstab
echo "sysfs /sys sysfs defaults 0 0" >> etc/fstab
echo "#!/bin/sh" > etc/init.d/rcS
echo "mount -a" >> etc/init.d/rcS
echo "mount -o remount,rw /" >> etc/init.d/rcS
echo "echo -e \"Welcome to ARM64 Linux\"" >> etc/init.d/rcS
chmod 755 etc/init.d/rcS
echo "::sysinit:/etc/init.d/rcS" > etc/inittab
echo "::respawn:-/bin/sh" >> etc/inittab
echo "::askfirst:-/bin/sh" >> etc/inittab
chmod 755 etc/inittab
cd dev
mknod console c 5 1
mknod null c 1 3
mknod tty1 c 4 1
cd $CURR_DIR
sudo umount $MOUNT_DIR
echo "make initrd ok!"

The user needs to adjust the BUSYBOX_PREFIX variable in the script according to the directory where they compiled busybox. After modifying the mk_initrd.sh script, we first create a make_initrd directory and copy the mk_initrd.sh script to that directory.

Since the mount and umount operations in Ubuntu require sudo permissions, you need to enter sudo permissions in the terminal before executing the mk_initrd.sh script, and then execute this mk_initrd.sh script to generate the initrd.ext4 image. The command and logs are as follows:

max@ubuntu2204:~$ mkdir make_initrd
max@ubuntu2204:~$ cd make_initrd/
max@ubuntu2204:~/make_initrd$ ls
mk_initrd.sh
max@ubuntu2204:~/make_initrd$ 
max@ubuntu2204:~/make_initrd$ sudo su
[sudo] password for max: 
root@ubuntu2204:/home/max/make_initrd# 
root@ubuntu2204:/home/max/make_initrd# sh mk_initrd.sh
+ pwd
+ CURR_DIR=/home/max/make_initrd
+ MOUNT_DIR=/home/max/make_initrd/mount_point
+ BUSYBOX_PREFIX=/home/max/busybox-1.36.1
……
……
+ echo make initrd ok!
make initrd ok!
root@ubuntu2204:/home/max/make_initrd# 
root@ubuntu2204:/home/max/make_initrd# exit
exit
max@ubuntu2204:~/make_initrd$ ls -lh initrd.ext4
-rw-r--r-- 1 root root 32M  1月 14 12:32 initrd.ext4

At this point, the file system image initrd.ext4 has been created.

Use QEMU AARCH64 Emulator to Run Linux Kernel

First, check which ARM 64 bit CPUs are supported by QEMU, the command and logs are as follows:

max@ubuntu2204:~/qemu-8.2.0/build/aarch64-softmmu$ ./qemu-system-aarch64 -cpu help
Available CPUs:
  a64fx
  arm1026
  arm1136
……
……
  sa1100
  sa1110
  ti925t
max@ubuntu2204:~/qemu-8.2.0/build/aarch64-softmmu$

Here we choose to use the ARMv8 64-bit CPU cortex-a57 to run the Linux kernel. Since the command parameters are too long, for convenience, we write the command to run the QEMU emulator into a script qemu.sh, the script is as follows:

#!/bin/bash
set -x
QEMU=~/qemu-8.2.0
KERNEL=~/linux-stable
INITRD=~/make_initrd
$QEMU/build/aarch64-softmmu/qemu-system-aarch64 \
-nographic \
-M virt \
-cpu cortex-a57 \
-smp 2 \
-m 4G \
-kernel $KERNEL/build/arch/arm64/boot/Image \
-append "nokaslr root=/dev/ram init=/linuxrc console=ttyAMA0 console=ttyS0" \
-initrd $INITRD/initrd.ext4

The user needs to adjust the QEMU, KERNEL, and INITRD variables in the qemu.sh script according to the directories where they compiled QEMU, kernel, and initrd.ext4 images. Then run the qemu.sh script to let the emulator run the Linux kernel. The corresponding command and logs are as follows:

max@ubuntu2204:~$ sh qemu.sh 
+ QEMU=/home/max/qemu-8.2.0
+ KERNEL=/home/max/linux-stable
+ INITRD=/home/max/make_initrd
+ /home/max/qemu-8.2.0/build/aarch64-softmmu/qemu-system-aarch64 -nographic -M virt -cpu cortex-a57 -smp 2 -m 4G -kernel /home/max/linux-stable/build/arch/arm64/boot/Image -append nokaslr root=/dev/ram init=/linuxrc console=ttyAMA0 console=ttyS0 -initrd /home/max/make_initrd/initrd.ext4
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[    0.000000] Linux version 6.1.72 (max@ubuntu2204) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #1 SMP PREEMPT Sun Jan 14 11:57:15 CST 2024
……
……
[    1.739471] Freeing unused kernel memory: 7552K
[    1.740835] Run /linuxrc as init process
[    1.865209] EXT4-fs (ram0): re-mounted. Quota mode: none.
Welcome to ARM64 Linux

Please press Enter to activate this console. 
~ #

At this point, we have successfully run the Linux kernel using the QEMU AARCH64 emulator and mounted the file system. At this point, we can enter commands to control the Linux system in the emulator, for example:

~ # cat /proc/version 
Linux version 6.1.72 (max@ubuntu2204) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #1 SMP PREEMPT Sun Jan 14 11:57:15 CST 2024
~ #

If you want to exit the QEMU emulator system, you need to press ctrl+a, release, and then press x. This way you can exit the QEMU emulator system.

At this point, we have successfully run the Linux kernel using the QEMU AARCH64 emulator!