Overview
QEMU (Quick EMUlator) is a general-purpose open-source emulator and virtualization tool that can simulate most CPU platforms, such as x86, ARM, ARM64, RISC-V, PowerPC, MIPS, etc. It can also simulate various hardware peripherals, such as memory, emmc, sdcard, usb, etc. Therefore, through qemu, we can simulate a development board. During the chip development process, chip companies usually use qemu for preliminary software development and debugging before the chip returns. In this article, we will introduce how to simulate the ARM64 platform with qemu and run the linux kernel, thus facilitating kernel development and debugging.
The tools and source code version information involved in this article are as follows:
Ubuntu: Ubuntu 22.04.2 LTSqemu: 8.2.0kernel: Linux 6.1.72busybox: busybox-1.36.1If you do not currently have a development environment for Ubuntu, you can refer to VMware Workstation Install Ubuntu-22.04 Virtual Machine tutorial to install the Ubuntu system.
Compile QEMU Emulator
Installation Tools
To install the tools needed to compile the QEMU source code, the commands and logs are as follows:
max@ubuntu2204:~$ sudo apt update[sudo] password for max: Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy InReleaseHit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-updates InRelease…………Building dependency tree... DoneReading state information... Done407 packages can be upgraded. Run 'apt list --upgradable' to see them.max@ubuntu2204:~$max@ubuntu2204:~$ sudo apt -y install python3.10-venv python3-sphinx ninja-build libpixman-1-dev pkg-config flex bison libfdt-dev libglib2.0-devReading package lists... DoneBuilding dependency tree... DoneReading state information... DoneThe following additional packages will be installed:…………Setting up python3-docutils (0.17.1+dfsg-2) ...Setting up python3-sphinx (4.3.2-1) ...max@ubuntu2204:~$Download QEMU Source Code
The official website of the QEMU emulator is https://www.qemu.org/, the homepage is as follows:
On the QEMU official website download page, you can view all the version source packages released by QEMU, as shown below:
Use wget to download the source code of the QEMU emulator, the command and logs are as follows:
max@ubuntu2204:~$ wget https://download.qemu.org/qemu-8.2.0.tar.xz--2024-01-13 21:46:41-- https://download.qemu.org/qemu-8.2.0.tar.xzResolving download.qemu.org (download.qemu.org)... 143.244.51.238, 89.187.187.12, 185.180.13.211, ...…………qemu-8.2.0.tar.xz 100%[===============================================================================================================>] 123.99M 2.54MB/s in 58s 2024-01-13 21:47:42 (2.14 MB/s) - ‘qemu-8.2.0.tar.xz’ saved [130008888/130008888]max@ubuntu2204:~$Compile QEMU Emulator
Unzip the downloaded qemu-8.2.0.tar.xz compressed package and compile the QEMU source code, the command and logs are as follows:
max@ubuntu2204:~$ tar xvJf qemu-8.2.0.tar.xzqemu-8.2.0/qemu-8.2.0/.patchew.ymlqemu-8.2.0/ebpf/…………qemu-8.2.0/subprojects/libvhost-user/libvhost-user-glib.hqemu-8.2.0/subprojects/keycodemapdb.wrapqemu-8.2.0/qemu.nsimax@ubuntu2204:~$ max@ubuntu2204:~$ cd qemu-8.2.0max@ubuntu2204:~/qemu-8.2.0$ max@ubuntu2204:~/qemu-8.2.0$ ./configureUsing './build' as the directory for build outputpython determined to be '/usr/bin/python3'python version: Python 3.10.12…………Found ninja-1.10.1 at /usr/bin/ninjaRunning postconf script '/home/max/qemu-8.2.0/build/pyvenv/bin/python3 /home/max/qemu-8.2.0/scripts/symlink-install-tree.py'max@ubuntu2204:~/qemu-8.2.0$max@ubuntu2204:~/qemu-8.2.0$ make -j8changing dir to build for make ""...make[1]: Entering directory '/home/max/qemu-8.2.0/build'ninja: no work to do./home/max/qemu-8.2.0/build/pyvenv/bin/meson introspect --targets --tests --benchmarks | /home/max/qemu-8.2.0/build/pyvenv/bin/python3 -B scripts/mtest2make.py > Makefile.mtest…………[9342/9342] Linking target tests/qtest/dbus-display-testmake[1]: Leaving directory '/home/max/qemu-8.2.0/build'max@ubuntu2204:~/qemu-8.2.0$The above command will compile the emulator for all platforms supported by QEMU, which will take a long time. If you want to save compilation time, you can choose to compile only the aarch64 emulator by configuring the target-list parameter when executing the configure command, as follows:
./configure --target-list=aarch64-softmmu,aarch64-linux-user,aarch64_be-linux-user --enable-kvm --enable-debugAt this point, the QEMU emulator has been compiled, and you can check the generated emulators for each platform as follows:
max@ubuntu2204:~/qemu-8.2.0$ ls build/qemu-system-*build/qemu-system-aarch64 build/qemu-system-hppa build/qemu-system-microblazeel build/qemu-system-nios2 build/qemu-system-riscv64 build/qemu-system-sparc build/qemu-system-xtensaebbuild/qemu-system-alpha build/qemu-system-i386 build/qemu-system-mips build/qemu-system-or1k build/qemu-system-rx build/qemu-system-sparc64build/qemu-system-arm build/qemu-system-loongarch64 build/qemu-system-mips64 build/qemu-system-ppc build/qemu-system-s390x build/qemu-system-tricorebuild/qemu-system-avr build/qemu-system-m68k build/qemu-system-mips64el build/qemu-system-ppc64 build/qemu-system-sh4 build/qemu-system-x86_64build/qemu-system-cris build/qemu-system-microblaze build/qemu-system-mipsel build/qemu-system-riscv32 build/qemu-system-sh4eb build/qemu-system-xtensaNext, you can run the build/qemu-system-aarch64 to simulate the ARM64 platform processor to run the Linux kernel.
Install ARM64 Cross Compilation Toolchain
Currently, we choose to use the linaro cross-compilation toolchain to compile uboot, kernel, busybox, etc. The download URL of the linaro cross-compilation toolchain is: https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/, the webpage is as follows:
First, we create a ~/cross-compiler directory, then download the cross-compilation toolchain compressed package gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz and unzip it. The commands and logs are as follows:
max@ubuntu2204:~$ mkdir ~/cross-compiler && cd ~/cross-compilermax@ubuntu2204:~/cross-compiler$ wget https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz--2024-01-14 00:07:24-- https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xzResolving releases.linaro.org (releases.linaro.org)... 52.215.200.125…………gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu. 100%[===============================================================================================================>] 112.43M 1.13MB/s in 4m 37s 2024-01-14 00:12:04 (416 KB/s) - ‘gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz’ saved [117896452/117896452]max@ubuntu2204:~/cross-compiler$max@ubuntu2204:~/cross-compiler$ tar xf gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xzmax@ubuntu2204:~/cross-compiler$Add the bin directory of this cross-compilation toolchain to the PATH environment variable as follows:
PATH=/home/max/cross-compiler/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin:$PATHThis way, we can use this cross-compilation toolchain, as follows:
max@ubuntu2204:~/cross-compiler$ aarch64-linux-gnu-gcc -vUsing built-in specs.COLLECT_GCC=aarch64-linux-gnu-gccCOLLECT_LTO_WRAPPER=/home/max/cross-compiler/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/../libexec/gcc/aarch64-linux-gnu/7.5.0/lto-wrapper…………Thread model: posixgcc version 7.5.0 (Linaro GCC 7.5-2019.12) max@ubuntu2204:~/cross-compiler$At this point, the ARM64 cross-compilation toolchain has been installed.
Compile Linux Kernel
Install Tools
On Ubuntu, install the tools needed to compile the kernel, the commands and logs are as follows:
max@ubuntu2204:~$ sudo apt install -y build-essential libncurses-dev bison flex libssl-dev libelf-dev gitReading package lists... DoneBuilding dependency tree... DoneReading state information... Done…………Setting up build-essential (12.9ubuntu3) ...Processing triggers for man-db (2.10.2-1) ...Processing triggers for libc-bin (2.35-0ubuntu3.1) ...max@ubuntu2204:~$Download Linux Kernel Source Code
We are downloading the official latest long-term version linux-6.1.y kernel from the official Linux website, which takes a long time to download from the official Linux website, so we choose to download from the Tsinghua repository. You can refer to the article on obtaining the Linux kernel source code to view all versions officially released by Linux and various fast methods for downloading kernel source code. The download command and logs are as follows:
max@ubuntu2204:~$ git clone https://mirrors.tuna.tsinghua.edu.cn/git/linux-stable.git -b linux-6.1.yCloning into 'linux-stable'...remote: Enumerating objects: 12104695, done.Receiving objects: 100% (12104695/12104695), 2.58 GiB | 6.23 MiB/s, done.remote: Total 12104695 (delta 0), reused 0 (delta 0), pack-reused 12104695Resolving deltas: 100% (10399156/10399156), done.Checking objects: 100% (33554432/33554432), done.Updating files: 100% (78744/78744), done.max@ubuntu2204:~$Enter the linux-stable directory to check that the downloaded kernel version is Linux 6.1.72, the command and logs are as follows:
max@ubuntu2204:~$ cd linux-stable/max@ubuntu2204:~/linux-stable$ git branch -vv* linux-6.1.y 7c58bfa711cb [origin/linux-6.1.y] Linux 6.1.72max@ubuntu2204:~/linux-stable$Compile Linux Kernel Source Code
Enter the linux-stable kernel directory, and configure the kernel options through menuconfig, the command and logs are as follows:
max@ubuntu2204:~/linux-stable$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 O=build menuconfigmake[1]: Entering directory '/home/max/linux-stable/build' GEN Makefile## using defaults found in arch/arm64/configs/defconfig#*** End of the configuration.*** Execute 'make' to start the build or try 'make help'.make[1]: Leaving directory '/home/max/linux-stable/build'max@ubuntu2204:~/linux-stable$Executing the above command will enter the menuconfig menu page, as shown:
In the menu, select the following options:
Device Drivers > Block devices > RAM block device supportAnd adjust the “Default RAM disk size (kbytes)” to 65536, the adjusted interface is as follows:
Then select save to save and exit the menuconfig menu. At this point, the kernel options are configured. Next, start compiling the kernel image, the command and logs are as follows:
max@ubuntu2204:~/linux-stable$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 O=build -j8make[1]: Entering directory '/home/max/linux-stable/build' SYNC include/config/auto.conf.cmd GEN Makefile………… LD [M] sound/soc/tegra/snd-soc-tegra210-sfc.komake[1]: Leaving directory '/home/max/linux-stable/build'max@ubuntu2204:~/linux-stable$Check the generated Image kernel image, the command and logs are as follows:
max@ubuntu2204:~/linux-stable$ file build/arch/arm64/boot/Imagebuild/arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pagesmax@ubuntu2204:~/linux-stable$At this point, the kernel image Image has been compiled.
Compile Busybox
The official website of Busybox is https://busybox.net/ We download the source code of BusyBox version 1.36.1 from the homepage using wget, the command and logs are as follows:
max@ubuntu2204:~$ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2--2024-01-14 11:50:58-- https://busybox.net/downloads/busybox-1.36.1.tar.bz2Resolving busybox.net (busybox.net)... 140.211.167.122…………busybox-1.36.1.tar.bz2 100%[================================================================================>] 2.41M 1.11MB/s in 2.2s 2024-01-14 11:51:15 (1.11 MB/s) - ‘busybox-1.36.1.tar.bz2’ saved [2525473/2525473]max@ubuntu2204:~$Unzip the busybox source code and enter the source code directory, then select static compilation through the menuconfig menu, that is, select the “Build static binary (no shared libs)” option. The command and logs are as follows:
max@ubuntu2204:~$ tar xf busybox-1.36.1.tar.bz2max@ubuntu2204:~$ cd busybox-1.36.1/max@ubuntu2204:~/busybox-1.36.1$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 menuconfig HOSTCC scripts/basic/fixdep HOSTCC scripts/basic/split-include…………*** End of configuration.*** Execute 'make' to build the project or try 'make help'.max@ubuntu2204:~/busybox-1.36.1$Executing the above command will display the busybox menu, as shown:
Select the following in the menu:
Settings > [*] Build static binary (no shared libs)The interface is as follows:
Then save and exit. At this point, the options for busybox are configured. Next, start compiling busybox, the command and logs are as follows:
max@ubuntu2204:~/busybox-1.36.1$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 install -j8 SPLIT include/autoconf.h -> include/config/* GEN include/bbconfigopts.h GEN include/common_bufsiz.h…………--------------------------------------------------You will probably need to make your busybox binarysetuid root to ensure all configured applets willwork properly.--------------------------------------------------Check the generated busybox executable file, the command is as follows:
max@ubuntu2204:~/busybox-1.36.1$ file _install/bin/busybox_install/bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=a6e3639090cbb9784bbdbfb939fa6ad48b25f244, strippedCreate initrd File System Image
To facilitate the creation of the initrd file system image, we use the mk_initrd.sh script to generate the initrd.ext4 image file, which will mount the file system in this image after the kernel is running. The script is as follows:
#!/bin/bashset -xCURR_DIR=`pwd`MOUNT_DIR=$CURR_DIR/mount_pointBUSYBOX_PREFIX=/home/max/busybox-1.36.1dd if=/dev/zero of=initrd.ext4 bs=1M count=32mkfs.ext4 initrd.ext4mkdir -p $MOUNT_DIRsudo mount initrd.ext4 $MOUNT_DIRcp -arf $BUSYBOX_PREFIX/_install/* $MOUNT_DIRcd $MOUNT_DIRmkdir -p etc dev mnt proc sys tmp mnt etc/init.d/echo "proc /proc proc defaults 0 0" > etc/fstabecho "tmpfs /tmp tmpfs defaults 0 0" >> etc/fstabecho "sysfs /sys sysfs defaults 0 0" >> etc/fstabecho "#!/bin/sh" > etc/init.d/rcSecho "mount -a" >> etc/init.d/rcSecho "mount -o remount,rw /" >> etc/init.d/rcSecho "echo -e \"Welcome to ARM64 Linux\"" >> etc/init.d/rcSchmod 755 etc/init.d/rcSecho "::sysinit:/etc/init.d/rcS" > etc/inittabecho "::respawn:-/bin/sh" >> etc/inittabecho "::askfirst:-/bin/sh" >> etc/inittabchmod 755 etc/inittabcd devmknod console c 5 1mknod null c 1 3mknod tty1 c 4 1cd $CURR_DIRsudo umount $MOUNT_DIRecho "make initrd ok!"The user needs to adjust the BUSYBOX_PREFIX variable in the script according to the directory where they compiled busybox. After modifying the mk_initrd.sh script, we first create a make_initrd directory and copy the mk_initrd.sh script to that directory. Since the mount and umount operations in Ubuntu require sudo permissions, you need to enter sudo permissions in the terminal before executing the mk_initrd.sh script, and then execute this mk_initrd.sh script to generate the initrd.ext4 image. The command and logs are as follows:
max@ubuntu2204:~$ mkdir make_initrdmax@ubuntu2204:~$ cd make_initrd/max@ubuntu2204:~/make_initrd$ lsmk_initrd.shmax@ubuntu2204:~/make_initrd$ max@ubuntu2204:~/make_initrd$ sudo su[sudo] password for max: root@ubuntu2204:/home/max/make_initrd# root@ubuntu2204:/home/max/make_initrd# sh mk_initrd.sh+ pwd+ CURR_DIR=/home/max/make_initrd+ MOUNT_DIR=/home/max/make_initrd/mount_point+ BUSYBOX_PREFIX=/home/max/busybox-1.36.1…………+ echo make initrd ok!make initrd ok!root@ubuntu2204:/home/max/make_initrd# root@ubuntu2204:/home/max/make_initrd# exitexitmax@ubuntu2204:~/make_initrd$ ls -lh initrd.ext4-rw-r--r-- 1 root root 32M 1月 14 12:32 initrd.ext4At this point, the file system image initrd.ext4 has been created.
Use QEMU AARCH64 Emulator to Run Linux Kernel
First, check which ARM 64 bit CPUs are supported by QEMU, the command and logs are as follows:
max@ubuntu2204:~/qemu-8.2.0/build/aarch64-softmmu$ ./qemu-system-aarch64 -cpu helpAvailable CPUs: a64fx arm1026 arm1136………… sa1100 sa1110 ti925tmax@ubuntu2204:~/qemu-8.2.0/build/aarch64-softmmu$Here we choose to use the ARMv8 64-bit CPU cortex-a57 to run the Linux kernel. Since the command parameters are too long, for convenience, we write the command to run the QEMU emulator into a script qemu.sh, the script is as follows:
#!/bin/bashset -xQEMU=~/qemu-8.2.0KERNEL=~/linux-stableINITRD=~/make_initrd$QEMU/build/aarch64-softmmu/qemu-system-aarch64
-nographic
-M virt
-cpu cortex-a57
-smp 2
-m 4G
-kernel $KERNEL/build/arch/arm64/boot/Image
-append "nokaslr root=/dev/ram init=/linuxrc console=ttyAMA0 console=ttyS0"
-initrd $INITRD/initrd.ext4The user needs to adjust the QEMU, KERNEL, and INITRD variables in the qemu.sh script according to the directories where they compiled QEMU, kernel, and initrd.ext4 images. Then run the qemu.sh script to let the emulator run the Linux kernel. The corresponding command and logs are as follows:
max@ubuntu2204:~$ sh qemu.sh + QEMU=/home/max/qemu-8.2.0+ KERNEL=/home/max/linux-stable+ INITRD=/home/max/make_initrd+ /home/max/qemu-8.2.0/build/aarch64-softmmu/qemu-system-aarch64 -nographic -M virt -cpu cortex-a57 -smp 2 -m 4G -kernel /home/max/linux-stable/build/arch/arm64/boot/Image -append nokaslr root=/dev/ram init=/linuxrc console=ttyAMA0 console=ttyS0 -initrd /home/max/make_initrd/initrd.ext4[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070][ 0.000000] Linux version 6.1.72 (max@ubuntu2204) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #1 SMP PREEMPT Sun Jan 14 11:57:15 CST 2024…………[ 1.739471] Freeing unused kernel memory: 7552K[ 1.740835] Run /linuxrc as init process[ 1.865209] EXT4-fs (ram0): re-mounted. Quota mode: none.Welcome to ARM64 Linux
Please press Enter to activate this console. ~ #At this point, we have successfully run the Linux kernel using the QEMU AARCH64 emulator and mounted the file system. At this point, we can enter commands to control the Linux system in the emulator, for example:
~ # cat /proc/version Linux version 6.1.72 (max@ubuntu2204) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #1 SMP PREEMPT Sun Jan 14 11:57:15 CST 2024~ #If you want to exit the QEMU emulator system, you need to press ctrl+a, release, and then press x. This way you can exit the QEMU emulator system.
At this point, we have successfully run the Linux kernel using the QEMU AARCH64 emulator!