Detailed Explanation of ARMv8/ARMv9 Interrupts: Software Aspects – An Introduction to Linux Kernel Interrupts

Table of Contents

• 1 Definition of the Linux Kernel ARM64 Interrupt Vector Table

• 2 Setting the Base Address of the Interrupt Vector Table in Linux Kernel ARM64

• 3 Introduction to the kernel_ventry Macro

• 4 Unimplemented Exception Vectors: elx_yyy_invalid

• 5 Introduction to el1_irq – Jumping to the Registered Handler Function

• 6 handle_domain_irq

• 7 Introduction to Interrupt Cascading

1 Definition of the Linux Kernel ARM64 Interrupt Vector Table

(linux/arch/arm64/kernel/entry.S)

/*
 * Exception vectors.
 */
.pushsection ".entry.text","ax"

.align 11
SYM_CODE_START(vectors)
	kernel_ventry 1, sync_invalid // Synchronous EL1t
	kernel_ventry 1, irq_invalid // IRQ EL1t
	kernel_ventry 1, fiq_invalid // FIQ EL1t
	kernel_ventry 1, error_invalid // Error EL1t

	kernel_ventry 1, sync // Synchronous EL1h
	kernel_ventry 1, irq // IRQ EL1h
	kernel_ventry 1, fiq // FIQ EL1h
	kernel_ventry 1, error // Error EL1h

	kernel_ventry 0, sync // Synchronous 64-bit EL0
	kernel_ventry 0, irq // IRQ 64-bit EL0
	kernel_ventry 0, fiq // FIQ 64-bit EL0
	kernel_ventry 0, error // Error 64-bit EL0

#ifdef CONFIG_COMPAT
	kernel_ventry 0, sync_compat,32 // Synchronous 32-bit EL0
	kernel_ventry 0, irq_compat,32 // IRQ 32-bit EL0
	kernel_ventry 0, fiq_compat,32 // FIQ 32-bit EL0
	kernel_ventry 0, error_compat,32 // Error 32-bit EL0
#else
	kernel_ventry 0, sync_invalid,32 // Synchronous 32-bit EL0
	kernel_ventry 0, irq_invalid,32 // IRQ 32-bit EL0
	kernel_ventry 0, fiq_invalid,32 // FIQ 32-bit EL0
	kernel_ventry 0, error_invalid,32 // Error 32-bit EL0
#endif
SYM_CODE_END(vectors)

Thoughts:1. Is the exception vector table arranged according to the ARMv8 definition? Isn’t each offset limited to 128 bytes of address space? How is this achieved?2. The Linux Kernel ARM64 architecture does not implement FIQ; why is it implemented here?3. Why are the first group of exception vectors not implemented?

2 Setting the Base Address of the Interrupt Vector Table in Linux Kernel ARM64

(linux/arch/arm64/kernel/head.S)

SYM_FUNC_START_LOCAL(__primary_switched)
	adrp x4, init_thread_union
	add sp, x4, #THREAD_SIZE
	adr_l x5, init_task
	msr sp_el0, x5 // Save thread_info

	adr_l x8, vectors // load VBAR_EL1 with virtual
	msr vbar_el1, x8 // vector table address
	isb

......
	b start_kernel
SYM_FUNC_END(__primary_switched)

Thoughts:1. When setting VBAR_EL1, if there are 8 ARM cores in the system, do all 8 cores need to be set? How are they set individually?

3 Introduction to the kernel_ventry Macro

(linux/arch/arm64/kernel/entry.S)

	.macro kernel_ventry, el, label, regsize = 64
	.align 7
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
	.if \\el == 0
	alternative_if ARM64_UNMAP_KERNEL_AT_EL0
	.if \\regsize == 64
	mrs x30, tpidrro_el0
	msr tpidrro_el0, xzr
	.else
	mov x30, xzr
	.endif
	alternative_else_nop_endif
	.endif
#endif

	sub sp, sp, #PT_REGS_SIZE
#ifdef CONFIG_VMAP_STACK
	/*
	 * Test whether the SP has overflowed, without corrupting a GPR.
	 * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)
	 * should always be zero.
	 */
	add sp, sp, x0 // sp' = sp + x0
	sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp
	tbnz x0, #THREAD_SHIFT, 0f
	sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0
	sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp
	b el\(\)el\(_\label

0:
	/*
	 * Either we've just detected an overflow, or we've taken an exception
	 * while on the overflow stack. Either way, we won't return to
	 * userspace, and can clobber EL0 registers to free up GPRs.
	 */

	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
	msr tpidr_el0, x0

	/* Recover the original x0 value and stash it in tpidrro_el0 */
	sub x0, sp, x0
	msr tpidrro_el0, x0

	/* Switch to the overflow stack */
	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0

	/*
	 * Check whether we were already on the overflow stack. This may happen
	 * after panic() re-enables interrupts.
	 */
	mrs x0, tpidr_el0 // sp of interrupted context
	sub x0, sp, x0 // delta with top of overflow stack
	tst x0, #~(OVERFLOW_STACK_SIZE - 1) // within range?
	b.ne __bad_stack // no? -> bad stack pointer

	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
	sub sp, sp, x0
	mrs x0, tpidrro_el0
#endif
	b el\(\)el\(_\label
	.endm

Note: .align=7 indicates that this code segment is aligned to 2^7=128 bytes, which is consistent with the size of each offset in the vector table. Although the code appears complex, it ultimately jumps to b el\(\)el\(_\label, which translates to jumping to functions like the following:

el1_sync_invalid
el1_irq_invalid
el1_fiq_invalid
el1_error_invalid

el1_sync
el1_irq
el1_fiq
el1_error

el0_sync
el0_irq
el0_fiq
el0_error

4 Unimplemented Exception Vectors: elx_yyy_invalid

The unimplemented vectors are defined as elx_yyy_invalid functions, which are also a form of implementation as they ultimately call the panic function.For example, the flow of el1_irq_invalid: el1_irq_invalid –> <span><span>bl bad_mode</span></span> –> panic(“bad mode”)

SYM_CODE_START_LOCAL(el1_irq_invalid)
	inv_entry 1, BAD_IRQ
SYM_CODE_END(el1_irq_invalid)

/*
 * Bad Abort numbers
 *-----------------
 */
#define BAD_SYNC0
#define BAD_IRQ1
#define BAD_FIQ2
#define BAD_ERROR3

/*
 * Invalid mode handlers
 */
.macro inv_entry, el, reason, regsize = 64
	kernel_entry \el, \regsize
	mov x0, sp
	mov x1, #\reason
	mrs x2, esr_el1
	bl bad_mode
	ASM_BUG()
.endm

/*  * bad_mode handles the impossible case in the exception vector. This is always
  * fatal.
  */
 asmlinkage void notrace bad_mode(struct pt_regs* regs, int reason, unsigned int esr)
{
	arm64_enter_nmi(regs);
	console_verbose();
	pr_crit("Bad mode in %s handler detected on CPU%d, code 0x%08x -- %s\n",
		handler[reason], smp_processor_id(), esr,
		esr_get_class_string(esr));
	__show_regs(regs);
	local_daif_mask();
	panic("bad mode");
}

5 Introduction to el1_irq – Jumping to the Registered Handler Function

Stripping away the transactions to see the essence,<span><span>el1_interrupt_handler handle_arch_irq</span></span> essentially callshandle_arch_irq, which points to the handler function defined inirq-gic-v3.c.

.align 6
SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
	kernel_entry 1
	el1_interrupt_handler handle_arch_irq
	kernel_exit 1
SYM_CODE_END(el1_irq)

We will not delve deeper into kernel_entry and kernel_exit, as they perform many tasks. Currently, we need to understand that one saves general-purpose registers, and the other restores them.

.macro kernel_entry, el, regsize = 64
	.if \regsize == 32
	mov w0, w0 // zero upper 32 bits of x0
	.endif
	stp x0, x1, [sp, #16*0]
	stp x2, x3, [sp, #16*1]
	stp x4, x5, [sp, #16*2]
	stp x6, x7, [sp, #16*3]
	stp x8, x9, [sp, #16*4]
	stp x10, x11, [sp, #16*5]
	stp x12, x13, [sp, #16*6]
	stp x14, x15, [sp, #16*7]
	stp x16, x17, [sp, #16*8]
	stp x18, x19, [sp, #16*9]
	stp x20, x21, [sp, #16*10]
	stp x22, x23, [sp, #16*11]
	stp x24, x25, [sp, #16*12]
	stp x26, x27, [sp, #16*13]
	stp x28, x29, [sp, #16*14]
......

.macro kernel_exit, el......
	msr elr_el1, x21 // set up the return data
	msr spsr_el1, x22
	ldp x0, x1, [sp, #16*0]
	ldp x2, x3, [sp, #16*1]
	ldp x4, x5, [sp, #16*2]
	ldp x6, x7, [sp, #16*3]
	ldp x8, x9, [sp, #16*4]
	ldp x10, x11, [sp, #16*5]
	ldp x12, x13, [sp, #16*6]
	ldp x14, x15, [sp, #16*7]
	ldp x16, x17, [sp, #16*8]
	ldp x18, x19, [sp, #16*9]
	ldp x20, x21, [sp, #16*10]
	ldp x22, x23, [sp, #16*11]
	ldp x24, x25, [sp, #16*12]
	ldp x26, x27, [sp, #16*13]
	ldp x28, x29, [sp, #16*14]
	ldr lr, [sp, #S_LR]
	add sp, sp, #PT_REGS_SIZE // restore sp
......

Next, we analyze the gic_handle_irq() function, which involves reading and writing to the GIC, retrieving the hardware interrupt number, then calling the handle_domain_irq function to find the matching interrupt handler function and then callback.

(linux/drivers/irqchip/irq-gic-v3.c)

static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs* regs)
{
	u32 irqnr;

	irqnr = do_read_iar(regs);

	/* Check for special IDs first */
	if ((irqnr >= 1020 && irqnr <= 1023))
		return;

	if (gic_supports_nmi() &&
		unlikely(gic_read_rpr() == GICD_INT_NMI_PRI)) {
		gic_handle_nmi(irqnr, regs);
		return;
	}

	if (gic_prio_masking_enabled()) {
		gic_pmr_mask_irqs();
		gic_arch_enable_irqs();
	}

	if (static_branch_likely(&& supports_deactivate_key))
		gic_write_eoir(irqnr);
	else
		isb();

	if (handle_domain_irq(gic_data.domain, irqnr, regs)) {
		WARN_ONCE(true, "Unexpected interrupt received!\n");
		gic_deactivate_unhandled(irqnr);
	}
}

Additionally, note that after Linux Kernel 5.0, there have been subtle changes in the handler functions in the GIC, as shown below:

6 handle_domain_irq

Supplement on IRQ Domain IntroductionIn the Linux kernel, we use the following two IDs to identify an interrupt from a peripheral:1. IRQ number. The CPU assigns a number to each peripheral interrupt, referred to as the IRQ Number. This IRQ number is a virtual interrupt ID, independent of hardware, and is used by the CPU to identify a peripheral interrupt.2. HW interrupt ID. For the interrupt controller, it collects multiple peripheral interrupt request lines and passes them up, thus the interrupt controller needs to encode the peripheral interrupts. The interrupt controller uses the HW interrupt ID to identify the peripheral interrupt.In the case of cascading interrupt controllers, merely using the HW interrupt ID cannot uniquely identify a peripheral interrupt; it is also necessary to know which interrupt controller the HW interrupt ID belongs to (the HW interrupt ID may be encoded differently on different interrupt controllers).Thus, the CPU and interrupt controller have different concepts in identifying interrupts, but from the perspective of driver engineers, we align with the CPU’s view; we only want to obtain an IRQ number without concern for which HW interrupt ID on which interrupt controller. The advantage of this is that when hardware related to interrupts changes, the driver software does not need to be modified. Therefore, the interrupt subsystem in the Linux kernel needs to provide a mechanism to map HW interrupt IDs to IRQ numbers…(This section is adapted from: http://www.wowotech.net/linux_kenrel/irq-domain.html)

Thoughts:1. The previous text mentions “in the case of cascading interrupt controllers”; why is there interrupt cascading? Isn’t it sufficient for a GIC controller to connect several thousand interrupts?

The processing flow of handle_domain_irq is as follows, ultimately calling the interrupt handler function registered with request_irq.

7 Introduction to Interrupt Cascading

This is also something I cannot understand; isn’t it sufficient for a GIC controller to connect several thousand interrupts? Perhaps it is for the convenience of SOC design. For example, a certain platform uses cascading methods.

Recommended Courses	“From Beginner to Mastering Armv8/Armv9 Architecture” – Three Sessions
“Trustzone/TEE/Security from Beginner to Master” – Standard Edition
Arm Selected – Platinum VIP Courses💋 All Courses
🌍Consult via WeChat: sami01_2023