The previous article introduced the basic concepts of virtualization and hypervisors. To facilitate virtualization, ARM has done a lot of work, primarily defining four Exception Levels (EL). As mentioned in earlier articles about exceptions and privileges, I will elaborate a bit more here. Each exception level is numbered, ranging from EL0 to EL3, with higher numbers corresponding to higher privilege levels. User programs run at EL0, operating systems run at EL1, hypervisors run at EL2, and firmware runs at EL3. By the way, in Intel’s architecture, a similar concept is referred to as ring 0-3.
In the ARM architecture, system registers are independent across different exception levels, encoded within the instruction set, and implemented separately in hardware. These system registers can be distinguished by their suffixes indicating the accessible levels. For example, TTBR0_EL1 is a register that holds the translation table base address. If EL0 attempts to access this register, an exception will be triggered.
Registers at lower exception levels can be accessed from higher exception levels, although this is generally not done. However, it is necessary in certain scenarios. For instance, during context switching or power management operations, more privileged levels may sometimes access registers associated with lower exception levels to achieve virtualization functions or read/write register sets as part of save and restore operations.
Besides exception levels, there are two other concepts: execution state and security state.
ARMv8-A has two available execution states:
-
AArch32: 32-bit execution state. Operations in this state are compatible with Armv7-A. There are two instruction sets available: T32 and A32. The standard register width is 32 bits.
-
AArch64: 64-bit execution state. There is one instruction set available: A64. The standard register width is 64 bits.
The ARMv8-A architecture implements two security states, allowing further partitioning of software to isolate and segregate trusted software:
-
Secure State: In this state, the Processing Element (PE) can access both secure and non-secure physical address spaces. Software running in this state can only respond to secure interrupts.
-
Non-Secure State: In this state, the PE can only access non-secure physical address space. Software running in this state can only respond to non-secure interrupts.
With the above concepts in mind, let’s take a look at virtualization in AArch64, as shown in the diagram below. The EL2 in secure state is shown in gray because secure state EL2 is not always available.
When it comes to virtualization, it can be divided into several major components: CPU virtualization, memory virtualization, IO virtualization, and interrupt virtualization. There isn’t much to discuss regarding CPU virtualization; it’s sufficient to differentiate between virtual machines (VM) and virtual CPUs (vCPU) for easier reading. A virtual machine may contain one or more virtual CPUs. A VM is not equivalent to a vCPU; for instance, a memory page can be allocated to a VM, allowing all vCPUs belonging to that VM to access it. A virtual interrupt, however, is specific to a vCPU, meaning only that vCPU can receive it. In ARM documentation, a more rigorous definition of a processor is the Processing Element (PE), as a CPU may contain multiple cores or threads, while PE refers to a single execution unit.
Next, let’s look at memory virtualization. According to Baidu Baike, “Virtual memory is a technology in computer system memory management. It makes applications believe they have a contiguous block of available memory (a continuous and complete address space), while in reality, it is often fragmented into multiple physical memory segments, with some temporarily stored on external disk storage for data swapping when needed.” Before discussing memory virtualization, it’s essential to understand virtual addresses (VA). In the earliest computers, programs accessed physical addresses (PA) directly. This method works fine when only one program runs at a time, but running multiple programs introduces many issues, such as how to prevent malicious programs from accessing physical memory allocated to other programs or how to avoid memory fragmentation. To solve the problems associated with direct access to physical addresses, a method was devised to add an indirect address before accessing the physical address directly. In this way, the memory address accessed in the program is no longer the actual physical address but a virtual address, which the operating system maps to the appropriate physical address. As long as the operating system manages the mapping from virtual addresses to physical addresses correctly, it can ensure that different programs access memory addresses in different areas and implement address protection strategies to isolate access between programs.
To support virtualization, ARMv8-A supports two-step address translation, commonly referred to as Stage 2 translation. Stage 2 translation allows the hypervisor to control the memory view of virtual machines (VMs). Specifically, it allows the hypervisor to control which system resources a VM can access and their positions in the VM’s address space. Stage 2 translation ensures that a VM can only see the resources allocated to it, while it cannot see resources allocated to other VMs or the hypervisor.
The operating system (OS) controls a set of address translation tables responsible for mapping from the virtual address space to what it considers the physical address space. The hypervisor is responsible for controlling the second step of address translation to obtain the actual physical address.
The translation controlled by the operating system is called stage 1 translation, while the translation managed by the program is called stage 2 translation. The address space that the operating system considers to be physical memory is referred to as Intermediate Physical Address (IPA) space. Therefore, the address translation process is VA->IPA->PA.
Each VM is assigned a virtual machine identifier (VMID). The VMID is used to tag TLB entries to identify which VM each entry belongs to. This tagging allows multiple translations from different VMs to coexist in the TLB. TLB entries can also be tagged with an address space identifier (ASID). The operating system assigns an ASID to applications, and all TLB entries in that application are tagged with that ASID. This means that TLB entries from different applications can coexist in the TLB without the possibility of one application using TLB entries belonging to another application. Each VM has its own ASID namespace. For example, two virtual machines may both use ASID 5, but they are for different purposes.
Both stage 1 and stage 2 mappings include attributes such as type and access permissions. The Memory Management Unit (MMU) combines the attributes from both stages to yield the final effective value. The MMU achieves this by selecting the more restrictive stage. In the example below, the device type is more restrictive than the normal type. Therefore, the final type is device. Conversely, if stage 1 = normal and stage 2 = device, the result is the same.
This method of combining attributes applies to most use cases, but sometimes the hypervisor may wish to override this behavior, for example, during the early boot phase of a VM.
The IPA address space of a VM is also divided into regions for memory and peripheral devices. VMs can access actual physical peripherals (commonly referred to as directly assigned devices) and virtual peripherals. The hypervisor fully simulates virtual peripherals in software.
Recommended Reading
Technical Sharing | ARM Series - Virtualization (Part 1)
END
About Anxin Education
Anxin Education is an innovative education platform focusing on AIoT (Artificial Intelligence + Internet of Things), providing a comprehensive AIoT education solution from primary and secondary schools to higher education institutions. Anxin Education relies on Arm technology to develop the ASC (Arm Smart Connectivity) curriculum and talent training system, which has been widely applied in industry-academia cooperation in higher education and STEM education in primary and secondary schools, dedicated to cultivating talents in the field of smart connectivity that meet the demands of the times.
