This article is excerpted from Chapter 16.6 of the second edition of the “Experiment Guide Manual”. The experiment guide is the accompanying experimental book for the second edition of “Run Linux Kernel Introduction”. The PDF version has been released and can be downloaded and printed freely! Download method: Log in to the “Run Linux Community” WeChat public account and enter “Run Linux 2” to get the download link.
This article is Experiment 16-4: Interrupt Experiment in Chapter 16 of the second edition of “Run Linux Kernel Introduction”. Before implementing the basic functions of the operating system, we first implement the interrupt handling function. We use the generic timer in the ARM Core on the Raspberry Pi as the interrupt source to clear the interrupt path first. This will also be needed when creating and switching processes and can be used as a clock interrupt.
1. Experiment Objectives
1) Understand and familiarize with ARM64 assembly language. 2) Understand and familiarize with ARM64 exception level handling. 3) Understand and familiarize with ARM64 interrupt handling process. 4) Understand and familiarize with the usage of system timers on Raspberry Pi.
2. Experiment Requirements
1) Implement support for the ARM64 exception vector table in boot.s. 2) Use the system timer on the Raspberry Pi as the interrupt source and write an interrupt handler that outputs “Timer interrupt occurred” whenever a timer interrupt occurs.
3. Experiment Details
This experiment has a certain level of difficulty for beginners, and readers need to be familiar with the following aspects.
-
The interrupt handling process in the ARMv8 architecture.
-
The interrupt controller on Raspberry Pi 3B and Raspberry Pi 4B.
-
The timer peripherals on Raspberry Pi.
-
The generic timer on ARMv8.
-
Saving and restoring interrupt context.
3.1 Interrupt Handling Process
ARM64 processor cores generally have two pins related to interrupts: nIRQ and nFIQ, and each processor core has a pair of such pins.
Additionally, the PSTATE status has two bits related to interrupts:
-
I: Used to mask IRQ interrupts.
-
F: Used to mask FIQ interrupts.
The following diagram shows the processing flow when an interrupt occurs in the ARM64 processor.
3.2 Interrupt Controller on Raspberry Pi
The ARM company provides a standard GIC controller, for example, the Raspberry Pi 4B supports GIC-400, while the Raspberry Pi 3B supports traditional interrupt methods (legacy interrupts).
The Raspberry Pi 4B provides and supports two types of interrupt controllers, while the Raspberry Pi 3B only supports the traditional interrupt controller.
-
GIC-400 (default)
-
Traditional interrupt controller (legacy interrupt controller)
Both Raspberry Pi 3B and 4B support multiple interrupt sources, as shown in the figure.
-
ARM Core N: Interrupt sources from the ARM Core itself, such as the generic timer inside the core.
-
ARMC: Interrupt sources accessible by VPU and CPU, such as mailbox, etc.
-
ARM_LOCAL: Interrupt sources accessible only by the CPU, such as local timers, etc.
The following diagram shows the routing situation of the traditional interrupt controller on the Raspberry Pi.
From the diagram, we can see that interrupts from ARM Core N and ARM_LOCAL will route to the hardware unit of ARM_LOCAL routing, while ARMC interrupts will route to the hardware unit of ARMC routing. The following diagram shows the routing situation of the interrupt status.
The routing process of the interrupt status register:
-
First read the SOURCEn interrupt status register.
-
If bit 8 of the SOURCEn register is set, read the PENDING2 register.
-
If bit 24 of PENDING2 is set, read the PENDING0 register.
-
If bit 25 of PENDING2 is set, read the PENDING1 register.
The Raspberry Pi 4B also supports the GIC-400 interrupt controller, which is left for the reader to learn by themselves.
3.3 Using the Generic Timer on ARM Core We use the generic timer on the ARM core as an example to explain how to use the timer interrupt on the Raspberry Pi. The Cortex-A72 supports four ARM Core generic timers:
-
CNT_PS_IRQ: Secure EL1 Physical Timer Event interrupt
-
CNT_PNS_IRQ: Nonsecure EL 1 Physical Timer Event interrupt
-
CNT_HP_IRQ: Hypervisor Physical Timer Event interrupt, EL2
-
CNT_V_IRQ: Virtual Timer Event interrupt EL3. Here, P means physical, S means secure, indicating whether it is in the secure world or non-secure world. ARMv8 supports TrustZone, where software running in the TrustZone operates in the secure world, while software outside the TrustZone operates in the non-secure world. The second one, PNS, means non-secure, which refers to the timer in the non-secure world. The third one is HP timer, where HP refers to the hypervisor, which is the non-secure EL2 physical timer mentioned in the A72 manual. The fourth is the virtual timer. The interrupt-related settings of these four generic timers are in the ARM LOCAL interrupt group registers. There are four IRQ_SOURCE registers, one for each CPU, representing the interrupt source status, four FIQ_SOURCE registers, and one for each CPU. There are four TIMER_CNTRL registers, one for each CPU, used to enable the corresponding interrupt source.
In this experiment, we take the Nonsecure EL 1 Physical Timer as an example, which is the PNS timer mentioned in the Raspberry Pi manual. The related initialization registers are in the ARM Core, as seen in the armv8.6 manual. There are only 2-3 registers related to it, so the timer is a very simple device.
-
The first register: CNTP_CTL_EL0, Counter-timer Physical Timer Control register. (Chapter D13.8.16). The relevant part is bit 0, the enable bit, which indicates whether to turn on this timer. Imask indicates whether the interrupt is masked. ISTATES is an interrupt status bit.
-
The second register: CNTP_TVAL_EL0, Counter-timer Physical Timer TimerValue register (Chapter D13.8.18). This is the timer value, and the timer value method is to initialize a value, and when it decrements to 0, the timer interrupt is triggered.
-
The simplest way of using the 64-bit compare value is to use the timer value method, which is to initialize a value and let it decrement. When it decrements to 0, the timer interrupt is triggered. We also use this method in our experiment.
3.4 Example
Below is the interrupt handling process of the Nonsecure generic timer at EL1: a) Initialize the timer, set the enable field of the cntp_ctl_el0 register to 1 b) Give the timer’s TimeValue an initial value, set the cntp_tval_el0 register c) Open the interrupt related to the timer in the Raspberry Pi interrupt controller, set CNT_PNS_IRQ in the TIMER_CNTRL0 register to 1 d) Open the overall IRQ interrupt switch in the PSTATE register
e) Timer interrupt occurs f) Jump to el1_irq assembly function g) Save interrupt context (using kernel_entry macro) h) Jump to interrupt handling function i) Read the IRQ_SOURCE0 interrupt status register in ARM_LOCAL j) Determine if the CNT_PNS_IRQ interrupt occurred k) If so, reset the TimeValue
l) Return to el1_irq assembly function m) Restore interrupt context n) Return to the interrupt scene
3.5 Interrupt Context
The interrupt context is the state at the moment when the interrupt occurs, including some states of the CPU processor. For ARM64 processors, the interrupt context includes the following contents:
-
PSTATE register
-
PC value
-
SP value
-
X0~x30 registers
Using a stack frame data structure to describe the interrupt context that needs to be saved (struct pt_regs).
When an interrupt occurs, the interrupt context is saved to the kernel stack of the current process, using a stack frame to help save the contents of the interrupt context.
The reference code for saving the interrupt context in this experiment is in the arch/arm64/kernel/entry.S file.
/*
Save the context at the time of the exception
Save x0~x29, x30 (lr), sp, elr, spsr to the stack
*/
.macro kernel_entry el
/*
SP points to the bottom of the stack, S_FRAME_SIZE indicates the size of a stack frame.
Define a struct pt_regs to describe a stack frame,
used to save context when an exception occurs.
*/
sub sp, sp, #S_FRAME_SIZE
/*
Save general registers x0~x29 to the stack frame pt_regs->x0~x29
*/
stp x0, x1, [sp, #16 *0]
stp x2, x3, [sp, #16 *1]
stp x4, x5, [sp, #16 *2]
stp x6, x7, [sp, #16 *3]
stp x8, x9, [sp, #16 *4]
stp x10, x11, [sp, #16 *5]
stp x12, x13, [sp, #16 *6]
stp x14, x15, [sp, #16 *7]
stp x16, x17, [sp, #16 *8]
stp x18, x19, [sp, #16 *9]
stp x20, x21, [sp, #16 *10]
stp x22, x23, [sp, #16 *11]
stp x24, x25, [sp, #16 *12]
stp x26, x27, [sp, #16 *13]
stp x28, x29, [sp, #16 *14]
/* x21: position of the stack top*/
add x21, sp, #S_FRAME_SIZE
mrs x22, elr_el1
mrs x23, spsr_el1
/* Save lr to pt_regs->lr, save sp to pt_regs->sp position*/
stp lr, x21, [sp, #S_LR]
/* Save elr_el1 to pt_regs->pc
Save spsr_elr to pt_regs->pstate*/
stp x22, x23, [sp, #S_PC]
.endm
When the interrupt returns, the interrupt scene is restored from the kernel stack of the current process.
The reference code for restoring the interrupt context in this experiment is in the arch/arm64/kernel/entry.S file.
/*
Restore the context saved when the exception occurred
*/
.macro kernel_exit el
/* Restore elr_el1 from pt_regs->pc,
Restore spsr_el1 from pt_regs->pstate
*/
ldp x21, x22, [sp, #S_PC] // load ELR, SPSR
msr elr_el1, x21 // set up the return data
msr spsr_el1, x22
ldp x0, x1, [sp, #16 * 0]
ldp x2, x3, [sp, #16 * 1]
ldp x4, x5, [sp, #16 * 2]
ldp x6, x7, [sp, #16 * 3]
ldp x8, x9, [sp, #16 * 4]
ldp x10, x11, [sp, #16 * 5]
ldp x12, x13, [sp, #16 * 6]
ldp x14, x15, [sp, #16 * 7]
ldp x16, x17, [sp, #16 * 8]
ldp x18, x19, [sp, #16 * 9]
ldp x20, x21, [sp, #16 * 10]
ldp x22, x23, [sp, #16 * 11]
ldp x24, x25, [sp, #16 * 12]
ldp x26, x27, [sp, #16 * 13]
ldp x28, x29, [sp, #16 * 14]
/* Restore lr from pt_regs->lr*/
ldr lr, [sp, #S_LR]
add sp, sp, #S_FRAME_SIZE // restore sp
eret // return to kernel
.endm
4. Experiment Steps
The reference code for this experiment can run on QEMU and Raspberry Pi 3B. In addition to the generic timer in the ARM core, Raspberry Pi 3B and 4B also support two additional peripherals: System Timer and ARM side Timer. However, since the QEMU emulator does not support these two peripherals, this experiment uses the generic timer in the ARM core.
4.1 Simulation on QEMU
We will first simulate on QEMU. Since the reference code for this experiment has not yet implemented support for the GIC-400 interrupt controller, we can only simulate the Raspberry Pi 3B in QEMU. In the Ubuntu Linux host, enter the directory of the reference experiment code.
rlk@rlk :$ cd /home/rlk/rlk/runninglinuxkernel_5.0/kmodules/rlk_lab/rlk_basic/chapter_16_benos/lab04_generic_timer/
Enter the make menuconfig menu.
rlk@master:lab04_generic_timer $ make menuconfig
Where:
Board selection (Raspberry 3B) -> Select Raspberry Pi 3B
Uart for Pi (pl_uart) -> Select PL serial port
Compile and run.
rlk@master:lab04_generic_timer $ make
rlk@master:lab04_generic_timer $ make run
4.2 Running on Raspberry Pi 3B
Copy the benos.bin file compiled in Chapter 4.1 to the MicroSD (you can refer to the method introduced in Chapter 16.1 regarding sharing between VMware Player and Windows host to copy to the host, and then copy to the MicroSD card), and rename it to benos3.bin. Insert the MicroSD back into Raspberry Pi 3B, then power it on. You can see the following log information in the serial port.
5. Running on Raspberry Pi 4B
We want to run the reference code for this experiment on Raspberry Pi 4B. If you have already compiled it, first do a make clean.
rlk@master:lab04_generic_timer $ make clean
Enter the make menuconfig menu.
rlk@master:lab04_generic_timer $ make menuconfig
Where:
Board selection (Raspberry 4B) -> Select Raspberry Pi 4B
Uart for Pi (pl_uart) -> Select PL serial port
Then recompile.
rlk@master:lab04_generic_timer $ make
Using the previous method, copy the compiled benos.bin file to the MicroSD. Then, insert the MicroSD card back into Raspberry Pi 4B. Start the Raspberry Pi. We find that the startup information is printed from the serial port, but “Core0 Timer interrupt received” is not printed.
What could be the reason? Why can Raspberry Pi 3B run while Raspberry Pi 4B cannot?
As mentioned earlier, the interrupt controllers of Raspberry Pi 3B and 4B are slightly different. The Raspberry Pi 3B only supports the traditional interrupt controller by default, while the Raspberry Pi 4B supports two: the traditional interrupt controller and the GIC-400 interrupt controller. Moreover, the Raspberry Pi 4B defaults to support GIC-400. Our experimental reference code only supports the traditional interrupt controller and does not yet support GIC-400. Therefore, there are two solutions here.
Solution 1: Modify the configuration so that Raspberry Pi 4B defaults to use the traditional interrupt controller.
Solution 2: Modify the code to support GIC-400.
Below is a hack for Solution 1. 1. You need to modify the bcm2711-rpi-4-b-nogic.dtb file in the boot partition of the MicroSD card. Use the hexedit tool to modify it, violently changing the string “arm,gic-400” to “not,gic-400”. In fact, just modifying this string is enough. This way, the firmware of Raspberry Pi 4B cannot find the description of GIC on the device tree, and it will automatically switch to the traditional interrupt controller. 2. Modify the config.txt file in the boot partition of the MicroSD card, adding an option “enable_gic=0”.
[pi4]
kernel=benos4.bin
enable_gic=0
The above two files are located in: /home/rlk/rlk/runninglinuxkernel_5.0/kmodules/rlk_lab/rlk_basic/chapter_16_benos/lab04_generic_timer/hack_rpi4 directory.
Reinsert the MicroSD card into Raspberry Pi 4B, power it on again, and you will see that the Raspberry Pi 4B can print “Core0 Timer interrupt received”.
As for Solution 2, we leave it for the readers who wish to explore further.
Experiment Code
The reference code for the experiment can be found in the runninglinuxkenrel_5.0 git repo.
Access from China:
https://benshushu.coding.net/public/runninglinuxkernel_5.0/runninglinuxkernel_5.0/git/files
github:
https://github.com/figozhang/runninglinuxkernel_5.0
The reference code corresponding to this article is in the following directory: kmodules/rlk_lab/rlk_basic/chapter_16_benos/lab04_generic_timer.
We provide a configured experimental environment based on the VMware/Vbox image of Ubuntu 20.04, which can be obtained by logging into the “Run Linux Community” WeChat public account and entering “Run Linux 2” to get the download link.
