Version Information

Host Machine: ubuntu 20.04.6 LTS (Focal Fossa)

Virtual Machine: ubuntu 20.04.6 LTS (Focal Fossa)

The steps for installing the host machine are omitted, as they are no different from installing a virtual machine in vmware.

It is important to ensure that Intel VT-x is enabled.

If the virtual machine reports This platform does not support virtualization Intel VT-x/EPT. Do you want to continue without using virtualization Intel VT-x/EPT?, refer to the article below for a solution.

https://blog.csdn.net/2301_77695535/article/details/146309899

Installing QEMU/KVM and Virsh on the Host Machine

Virsh is short for Virtual Shell, a command-line tool for managing virtual machines. You can use Virsh to create, edit, start, stop, shut down, and delete VMs. Virsh currently supports KVM, LXC, Xen, QEMU, OpenVZ, VirtualBox, and VMware ESX. Here we will use Virsh to manage QEMU/KVM virtual machines.

Before installation, first confirm whether your CPU supports virtualization technology. Use grep to check cpuinfo for the presence of “vmx” (Intel-VT technology) or “svm” (AMD-V support) output:

egrep "(svm|vmx)" /proc/cpuinfo

Some CPU models may have VT support disabled in the BIOS by default. We need to check the BIOS settings to ensure that VT support is enabled. Use the kvm-ok command to check:

sudo apt install cpu-checker
kvm-ok

If the output is:

INFO: /dev/kvm exists
KVM acceleration can be used

This indicates that the CPU virtualization support has been enabled in the BIOS.

Run the following command to install QEMU/KVM and Virsh:

sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager

Check if the libvirt daemon is active:

sudo systemctl is-active libvirtd
active

If there is no output of active, run the following command to start the libvertd service:

sudo systemctl enable libvirtd
sudo systemctl start libvirtd

Installing QEMU Virtual Machine on the Host Machine

Create a virtual machine image with a size of 40G, in qcow2 format with dynamically allocated disk space.

qemu-img create -f qcow2 ubuntutest.img 40G

Create a virtual machine system and install the operating system:

qemu-system-x86_64 \
-name ubuntutest \
-smp 2 \
-m 4096 \
-hda ubuntutest.img \
-cdrom ubuntu-20.04.6-live-server-amd64.iso \
-boot d

Follow the steps to configure the installation; this step is no different from a normal virtual machine installation.

Note that I did not add -enable-kvm, which may affect gdb software breakpoints.

Configuring Internet Access

Option 1: Build a TAP Network Card for NAT Internet Access

Create TAP Device on Host Machine

sudo ip tuntap add dev tap0 mode tap
sudo ip addr add 192.168.100.1/24 dev tap0
sudo ip link set tap0 up

Configure IP Forwarding and NAT Rules on Host Machine

sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -s 192.168.100.0/24 -o ens33 -j MASQUERADE
sudo iptables -A FORWARD -i tap0 -j ACCEPT

Start QEMU with TAP

qemu-system-x86_64 \
-enable-kvm \
-m 4096 \
-drive file=./ubuntutest.img \
-boot d \
-net nic -net tap,ifname=tap0,script=no,downscript=no

Configure the virtual machine’s network to any address in the 192.168.100.0/24 subnet (for example, 192.168.100.10).

Open the netplan configuration file, modify the address, and use sudo netplan apply to restart the network.

# This is the network config written by 'subiquity'
network:
  ethernets:
    ens3:
      dhcp4: no
      addresses:
        - 192.168.100.10/24
      routes:
        - to: default
          via: 192.168.100.1
      nameservers:
        addresses: [8.8.8.8]
  version: 2

Using virbr0 for NAT Internet Access

Create and configure bridge.conf

File Path: Usually located at /etc/qemu/ or /usr/local/etc/qemu/ (depending on the installation method). Create it using the following command:

sudo mkdir -p /etc/qemu
sudo vim /etc/qemu/bridge.conf

Example content:

allow virbr0  # If using the default bridge interface of libvirt

Next, specify to use the virbr0 bridge for internet access when starting.

qemu-system-x86_64 \
-enable-kvm \
-m 4096 \
-drive file=./ubuntutest.img -boot d \
-netdev bridge,id=net0,br=virbr0 \
-device virtio-net-pci,netdev=net0

When configuring the virtual machine network, set it to be in the same subnet as virbr0, which is 192.168.122.0/24 (for example, 192.168.122.10).

Downloading and Compiling the Linux Kernel

Installing Compilation Dependencies

First, we need to install the dependency packages required for compiling the kernel. I am compiling the Linux kernel code on the host machine, so the following commands are executed directly on the host machine.

sudo apt install libncurses5-dev libssl-dev bison flex libelf-dev gcc make openssl libc6-dev dwarves

Downloading Linux Kernel Code and Building

Download the Linux kernel code; here I am using version 5.10.209.

https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.10.209.tar.gz

Use the following command to compile:

sudo make menuconfig

To build a debuggable kernel, we need to configure the following parameters.

CONFIG_DEBUG_INFO includes debugging information in the kernel and kernel modules; this option adds the -g option to the compiler parameters used by gcc.

The menu path for this option is:

Kernel hacking  --->
Compile-time checks and compiler options  ---> 
 [*] Compile the kernel with debug info

You can also directly set CONFIG_DEBUG_INFO in the .config file.

CONFIG_DEBUG_INFO=y

CONFIG_FRAME_POINTER saves the call frame information in different locations in registers or on the stack, allowing gdb to construct stack back traces more accurately when debugging the kernel.

You can also set this in the .config file:

CONFIG_FRAME_POINTER=y

Enable CONFIG_GDB_SCRIPTS, but disable CONFIG_DEBUG_INFO_REDUCED.

CONFIG_GDB_SCRIPTS=y
CONFIG_DEBUG_INFO_REDUCED=n

CONFIG_KGDB enables the built-in kernel debugger, which allows for remote debugging.

Kernel hacking  --->
Generic Kernel Debugging Instruments  ---> 
 [*] KGDB: kernel debugger

CONFIG_KGDB=y

Disable CONFIG_RANDOMIZE_BASE setting.

CONFIG_RANDOMIZE_BASE can be found at the following location:

Processor type and features --->
Randomize the address of the kernel image (KASLR)

Or directly add the following statement in the .config file:

CONFIG_RANDOMIZE_BASE=n

KASLR changes the base address where the kernel code is placed at boot time. If you enable KASLR in the kernel configuration (CONFIG_RANDOMIZE_BASE=y), you will not be able to set breakpoints from gdb. After setting the necessary kernel parameters, we can start compiling the kernel:

sudo make -j8 
sudo make INSTALL_MOD_STRIP=1 modules_install
sudo make install

make modules_install will install the module files to /lib/modules/5.10.209, and it is best to add INSTALL_MOD_STRIP=1, otherwise the initrd.img will be very large.

The compilation requires approximately 30G of space, so ensure you have at least >=30G of disk space prepared.

After completing these steps, we will have the required Linux kernel image bzImage (vmlinuz) and initrd.img

xu@xu-dev:~/work/linux-5.10.209$ ls /boot/ -alh |grep 5.10.209
-rw-r--r--  1 root root 243K 4月  20 19:34 config-5.10.209
lrwxrwxrwx  1 root root   19 4月  20 19:34 initrd.img -> initrd.img-5.10.209
-rw-r--r--  1 root root  60M 4月  20 19:34 initrd.img-5.10.209
-rw-r--r--  1 root root 5.5M 4月  20 19:34 System.map-5.10.209
lrwxrwxrwx  1 root root   16 4月  20 19:34 vmlinuz -> vmlinuz-5.10.209
-rw-r--r--  1 root root  14M 4月  20 19:34 vmlinuz-5.10.209

Introduction to Compiled Products

During the Linux kernel compilation process, the generated files can be categorized based on functionality. Below is a detailed introduction to the corresponding file roles and sources:

Core Executable Files

vmlinux

Description: The original, uncompressed kernel executable file, containing a complete symbol table and debugging information, and is relatively large.
Generation Path: Located in the kernel source root directory or in the arch/<architecture>/boot/compressed/ directory.
Usage: Used for debugging and analyzing kernel crash issues, not directly used to boot the system.
Source: Generated during the compilation process by linking target files such as head-y, init-y, and core-y.

zImage ** and ** bzImage

Description: Compressed kernel image files, where bzImage (Big zImage) supports larger kernel sizes (used when exceeding 512KB).
Generation Path: Located in the arch/<architecture>/boot/ directory (e.g., arch/x86/boot/bzImage).
Structure: Composed of setup.bin (bootloader) and the compressed vmlinux, with additional decompression header information.
Usage: Directly used for system booting, and is the default kernel file for most Linux distributions.

uImage

Description: An image file designed for the U-Boot bootloader, adding U-Boot header information to the zImage.
Generation Path: Must be processed using the mkimage tool from the zImage.
Usage: Used in embedded systems in conjunction with U-Boot.

Boot-Related Files

System.map

Description: Kernel symbol mapping file, records the memory addresses of all functions and variables.
Generation Path: Located in the kernel source root directory.
Usage: Used to associate addresses with symbols during debugging, such as analyzing kernel crash logs.

initrd.img ** or ** initramfs

Description: Initial memory file system, containing temporary drivers and tools needed for early boot (such as disk drivers, filesystem modules).
Generation Path: Generated by mkinitramfs or dracut, stored in the /boot/ directory.
Usage: Resolves dependencies before mounting the root filesystem.

Kernel Module Files

.ko ** files (Kernel Object) **

Description: Dynamically loadable kernel modules, inserted into the kernel as needed.
Generation Path: Located in the directories of various drivers or functional modules (e.g., drivers/net/ethernet/).
Installation Path: Installed to /lib/modules/<kernel version>/ directory via make modules_install.
Usage: Flexibly extend kernel functionality (e.g., adding new hardware drivers).

Configuration Files and Logs

.config

Description: Kernel compilation configuration file, records all selected functional options (e.g., CONFIG_DEBUG_INFO=y).
Generation Path: Located in the kernel source root directory.
Source: Generated via make menuconfig or by copying the default configuration (e.g., arch/arm/configs/s5pv210_defconfig).

vmlinux.lds

Description: Linker script file that defines the memory layout of various segments (code, data, stack) in the kernel.
Generation Path: Located in the arch/<architecture>/kernel/ directory.
Usage: Guides the linker in generating vmlinux.

Intermediate Files and Tool Generated Files

.o ** and ** built-in.o

Description: Intermediate object files generated during the compilation process, where built-in.o is the merge of all .o files in the same directory.
Generation Path: Located in the directories of various submodules (e.g., init/built-in.o).
Usage: Gradually builds the final kernel image.

Device Tree Files (.dtb)

Description: Binary files describing the hardware topology, used in embedded systems (e.g., ARM architecture).
Generation Path: Generated from .dts files using the device tree compiler (DTC), located in arch/<architecture>/boot/dts/.
Usage: Adapts to different hardware platforms.

Boot Management and Version Identification

vmlinuz-<version number>

Description: Compressed kernel image installed in the /boot/ directory, usually a symbolic link to bzImage.
Usage: Loaded by bootloaders like Grub to start the kernel.

config-<version number>

Description: A backup of the .config file, preserving the complete configuration used during compilation.
Generation Path: Located in the /boot/ directory.

Summary Comparison Table

File Type	Key Files	Function	Generation Command
Core Image	`<span>vmlinux</span>`, `<span>bzImage</span>`	Kernel execution and booting	`<span>make</span>`, `<span>make bzImage</span>`
Modules	`<span>.ko</span>`	Dynamically extend kernel functionality	`<span>make modules</span>`
Configuration	`<span>.config</span>`	Records compilation options	`<span>make menuconfig</span>`
Boot Support	`<span>initrd.img</span>`	Load early boot dependencies	`<span>mkinitramfs</span>`
Symbol Mapping	`<span>System.map</span>`	Debug symbol address mapping	Automatically generated
Device Tree	`<span>.dtb</span>`	Embedded hardware description	`<span>make dtbs</span>`

Kernel Debugging

After obtaining the kernel image bzImage and initrd.img, you can use them to boot the qemu virtual machine.

Note that my debugging uses QEMU’s -kernel and -initrd to directly load the kernel, rather than using grub to load the kernel.

My qemu startup script is as follows:

qemu-system-x86_64 \
-smp 2 \
-m 4096 \
-S -s \
-drive file=/home/xu/work/ubuntutest.img \
-netdev bridge,id=net0,br=virbr0 \
-device virtio-net-pci,netdev=net0  \
-kernel /home/xu/work/kernel-with-rwx/bzImage \
-initrd /home/xu/work/kernel-with-rwx/initrd.img-5.10.209 \
-append "root=/dev/mapper/ubuntu--vg-ubuntu--lv ro maybe-ubiquity console=ttyS0\
-nographic

-smp 2 allocates 2 virtual CPU cores (vCPU), with each core being single-threaded and single-socket by default. For finer control (e.g., multi-socket, multi-thread), it can be expanded to -smp 2,sockets=1,cores=2,threads=1
-m 4096 allocates 4096MB (4GB) of memory to the virtual machine. QEMU defaults to allocating 128MB, and this parameter should be adjusted based on host resources and virtual machine needs.
-S pauses CPU execution at startup, waiting for an external debugger (like GDB) to connect before continuing, commonly used for kernel debugging.
-s enables the GDB debugging server, which listens on the local port 1234 by default. Combined with -S, it allows debugging the kernel from the startup phase.
-drive file=/home/xu/work/ubuntutest.img loads the disk image named ubuntutest.img as the primary hard drive. The default interface type is if=virtio (high-performance virtualization driver), and if the format is not specified, QEMU automatically detects it (e.g., qcow2 or raw).
-netdev bridge,id=net0,br=virbr0 creates a bridged network backend, connecting to the host’s virbr0 bridge, allowing the virtual machine to access the external network through the host’s network interface.
-device virtio-net-pci,netdev=net0 adds a virtual network card to the virtual machine, using the virtio-net-pci driver (high-performance paravirtualized network card), bound to the above network backend.
-kernel /home/xu/work/kernel-with-rwx/bzImage specifies the Linux kernel image file bzImage, bypassing the virtual machine’s BIOS boot to directly load the kernel.
-initrd /home/xu/work/kernel-with-rwx/initrd.img-5.10.209 uses initrd.img-5.10.209 as the initial memory file system, containing the modules and tools needed for early boot.
-append "root=/dev/mapper/ubuntu--vg-ubuntu--lv ro maybe-ubiquity console=ttyS0"<code>: specifies the location of the root filesystem (LVM logical volume). ro: mounts the root filesystem in read-only mode, which usually needs to be switched to read-write mode later.console=ttyS0: redirects console output to the serial port ttyS0, used in conjunction with -nographic. (The root filesystem path can be checked using df -h on Ubuntu systems).

As mentioned above, there is also a method to use grub to boot the kernel, which is relatively more complicated. It requires copying the kernel from the host machine to the virtual machine and executing make modules_install and make install, then starting with -boot d to load the kernel from the hard drive.

qemu-system-x86_64 \
-enable-kvm \
-smp 2 \
-m 4096 \
-drive file=./ubuntutest.img -boot d \
-netdev bridge,id=net0,br=virbr0 \
-device virtio-net-pci,netdev=net0

However, I find using -kernel and -initrd to be more convenient and recommend it.

Next, start gdb for debugging. Use another terminal to open gdb:

xu@xu-dev:~/work/linux-5.10.209$ gdb vmlinux  -q
Reading symbols from vmlinux...
warning: File "/home/xu/work/linux-5.10.209/scripts/gdb/vmlinux-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
    add-auto-load-safe-path /home/xu/work/linux-5.10.209/scripts/gdb/vmlinux-gdb.py
line to your configuration file "/home/xu/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/home/xu/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"

Use gdb for remote connection:

(gdb) target remote :1234
Remote debugging using :1234
0x000000000000fff0 in exception_stacks ()

Set a breakpoint:

(gdb) b start_kernel
Breakpoint 1 at 0xffffffff82daad61: file init/main.c, line 847.

Continue execution to hit the breakpoint:

(gdb) c
Continuing.

Thread 1 hit Breakpoint 1, start_kernel () at init/main.c:847

Step through debugging:

847	{
(gdb) n
851		set_task_stack_end_magic(&amp;init_task);
(gdb) n
852		smp_setup_processor_id();
(gdb) n
855		cgroup_init_early();
(gdb)

Issues

Certificate Not Found

  CC [M]  kernel/kheaders.o
  CC      certs/system_keyring.o
make[1]: *** No rule to make target 'debian/canonical-certs.pem', needed by 'certs/x509_certificate_list'.  Stop.
make: *** [Makefile:1832: certs] Error 2

Solution:

1. Modify CONFIG_SYSTEM_TRUSTED_KEYS to empty value:

CONFIG_SYSTEM_TRUSTED_KEYS="" 

2. Modify CONFIG_SYSTEM_REVOCATION_KEYS (optional) to empty value if it is not already empty:

Before modification: original variable has value CONFIG_SYSTEM_REVOCATION_KEYS="debian/canonical-revoked-certs.pem" After modification: assign an empty valueCONFIG_SYSTEM_REVOCATION_KEYS="" 

Kernel Stuck on “loading initial ramdisk” for a Long Time and Kernel Crash Report on Boot with Error: Kernel panic not syncing: System is deadlocked on memory

During the kernel compilation process, if the kernel modules are installed without using INSTALL_MOD_STRIP=1, it can lead to an excessively large initrd file, causing the boot process to get stuck decompressing the initrd for too long.

For example, my initrd.img is 1.2G.

To resolve this issue, you need to add INSTALL_MOD_STRIP=1 when installing the modules:

sudo make INSTALL_MOD_STRIP=1 modules_install
sudo make install

kernel crash report on boot with error Kernel panic not syncing: System is deadlocked on memory is related to the initrd.img. One way to resolve it is by reducing the size of initrd.img, or alternatively, increasing the virtual machine’s memory.

GDB Software Breakpoints Not Triggering, Hardware Breakpoints Can Trigger

qemu may cause this issue if started with the -enable-kvm option.

If you specify -enable-kvm at startup, you need to use hardware breakpoints when setting breakpoints, for example:

hb start_kernel

PS

How to Clean Up Kernel Compilation Artifacts

If you wish to uninstall after compiling and installing the Linux kernel, here are the specific steps and precautions:

Step 1: Confirm Kernel Version and Installation Path

Check Installed Kernel Versions:

ls /boot | grep vmlinuz    # List all kernel files
uname -r                  # Check the currently running kernel version (to avoid accidental deletion)

Confirm the Manually Compiled Kernel Version: Assume the target kernel version to uninstall is 5.18.8, ensure it is not in the output of uname -r, otherwise deleting it may cause the system to fail to boot.

Step 2: Manually Delete Kernel-Related Files

Manually compiled kernel files are scattered across multiple directories and need to be deleted one by one by path:

Delete Kernel Files in the /boot Directory:

sudo rm -rf /boot/vmlinuz-5.18.8
sudo rm -rf /boot/initrd.img-5.18.8
sudo rm -rf /boot/config-5.18.8
sudo rm -rf /boot/System.map-5.18.8

Delete Kernel Module Directory:

sudo rm -rf /lib/modules/5.18.8

Delete Kernel Source Directory (Optional):

sudo rm -rf /usr/src/linux-5.18.8

Clean Up initramfs Residue:

sudo rm -rf /var/lib/initramfs-tools/5.18.8

Step 3: Update GRUB Boot Configuration

After deleting the kernel, update GRUB to avoid residual invalid boot entries:

sudo update-grub

Step 4: Reboot the System

sudo reboot

After rebooting, use uname -r to confirm whether the current kernel version has switched to another available version.

Reference Articles

https://github.com/mz1999/blog/blob/master/docs/gdb-kernel-debugging.md

Setting Up a QEMU Linux Kernel (5.10.209) Development Environment

Version Information

Installing QEMU/KVM and Virsh on the Host Machine

Installing QEMU Virtual Machine on the Host Machine

Configuring Internet Access

Option 1: Build a TAP Network Card for NAT Internet Access

Create TAP Device on Host Machine

Configure IP Forwarding and NAT Rules on Host Machine

Start QEMU with TAP

Using virbr0 for NAT Internet Access

Downloading and Compiling the Linux Kernel

Installing Compilation Dependencies

Downloading Linux Kernel Code and Building

Introduction to Compiled Products

Core Executable Files

Boot-Related Files

Kernel Module Files

Configuration Files and Logs

Intermediate Files and Tool Generated Files

Boot Management and Version Identification

Summary Comparison Table

Kernel Debugging

Issues

Certificate Not Found

Kernel Stuck on “loading initial ramdisk” for a Long Time and Kernel Crash Report on Boot with Error: Kernel panic not syncing: System is deadlocked on memory

GDB Software Breakpoints Not Triggering, Hardware Breakpoints Can Trigger

PS

How to Clean Up Kernel Compilation Artifacts

Step 1: Confirm Kernel Version and Installation Path

Step 2: Manually Delete Kernel-Related Files

Step 3: Update GRUB Boot Configuration

Step 4: Reboot the System

Reference Articles

Leave a Comment Cancel reply

Version Information

Installing QEMU/KVM and Virsh on the Host Machine

Installing QEMU Virtual Machine on the Host Machine

Configuring Internet Access

Option 1: Build a TAP Network Card for NAT Internet Access

Create TAP Device on Host Machine

Configure IP Forwarding and NAT Rules on Host Machine

Start QEMU with TAP

Using virbr0 for NAT Internet Access

Downloading and Compiling the Linux Kernel

Installing Compilation Dependencies

Downloading Linux Kernel Code and Building

Introduction to Compiled Products

Core Executable Files

Boot-Related Files

Kernel Module Files

Configuration Files and Logs

Intermediate Files and Tool Generated Files

Boot Management and Version Identification

Summary Comparison Table

Kernel Debugging

Issues

Certificate Not Found

Kernel Stuck on “loading initial ramdisk” for a Long Time and Kernel Crash Report on Boot with Error: Kernel panic not syncing: System is deadlocked on memory

GDB Software Breakpoints Not Triggering, Hardware Breakpoints Can Trigger

PS

How to Clean Up Kernel Compilation Artifacts

Step 1: Confirm Kernel Version and Installation Path

Step 2: Manually Delete Kernel-Related Files

Step 3: Update GRUB Boot Configuration

Step 4: Reboot the System

Reference Articles

Related posts

Leave a Comment Cancel reply