With the widespread use of ARM architecture processors in the field of cloud computing, the demand for application developers to be familiar with ARM architecture-related technologies is becoming increasingly urgent. It is well known that providing virtual machine services is a typical application scenario of cloud computing, and the performance of devices within virtual machines is crucial to service quality.
When talking about devices in virtual machines, one cannot ignore device virtualization. Initially, devices in virtual machines were entirely realized through software simulation, known as full virtualization devices. The advantage of this approach is that the OS running on the physical machine can operate in the virtual machine without any modifications, but the device performance is poor and does not meet people’s needs. Later, the idea of paravirtualization was proposed, hoping to optimize software, such as modifying device models and reducing unnecessary memory copies, to meet the performance demands for devices. A typical representative is the virtio device. Although this method significantly improves performance, there remains a large gap compared to real devices. To achieve performance equivalent to real devices, the method of device passthrough has been proposed and gradually become mainstream.
Using passthrough devices in virtual machines can indeed bring the maximum performance improvement. However, it exposes a series of system security issues. For example, devices that provide DMA usually have the ability to write to any page of memory, meaning that the Guest OS in the virtual machine having the ability to create DMA is equivalent to user space having root privileges. Malicious devices may exploit this to launch DMA attacks, posing security risks to other virtual machines and even the entire physical machine. Therefore, it is necessary to provide DMA remapping functionality, which restricts passthrough device access to memory to specific domains using page tables, achieving isolation for passthrough devices while improving IO performance and ensuring the security of passthrough device DMA. Additionally, the interrupts from passthrough devices are by default sent to the physical CPU, and to notify the vCPU in the virtual machine, interrupt redirection is required.
The implementation of DMA remapping and interrupt redirection was initially entirely the responsibility of the hypervisor or emulator (such as QEMU). This is a purely software approach, which inevitably greatly reduces device performance, hence the urgent need for hardware collaboration. Consequently, ARM’s SMMU and GIC specifications have added support for DMA remapping and interrupt redirection. The focus of this article is to introduce how to support device passthrough in virtualization environments around these two features.
DMA Remapping
SMMU stands for System MMU, a component in the ARM architecture that provides address translation services for non-CPU devices, functioning similarly to the CPU’s MMU component. Here, non-CPU devices primarily refer to peripheral subsystems that initiate DMA operations. As shown in the figure below:
Note: PE is the abbreviation for Processing Element, which can be understood as CPU.
In virtualization application scenarios, address translation often occurs in two stages. The first stage is the conversion from virtual address (VA) to the physical address (IPA) perceived by the Guest OS; the second stage is the conversion from IPA to the real physical address (PA). The SMMU can support both stages of conversion simultaneously or support each stage individually. This makes DMA remapping for passthrough devices possible. The DMA of passthrough devices only needs to obtain the IPA, and the subsequent work is handled by the SMMU for stage two processing, achieving safe and efficient access to PA. As shown in the figure below:
For the above application scenario, we introduce the internal structure of the SMMU to explain the specific implementation principles. The SMMU contains two core units: the Translation Control Unit (TCU) and the Translation Buffer Unit (TBU). The TCU mainly contains the Stream Mapping Table (SMT) and the Context Bank Table (CBT). The SMT indexes internal entries via StreamID, which corresponds one-to-one with specific passthrough devices, allowing access to a pointer to the specific entry in the CBT. The content of the CBT entry contains the pointer to the stage two translation table, and once this pointer is obtained, the translation process can proceed step-by-step like a normal MMU. The role of the TBU is akin to that of a TLB, leveraging cache principles to enhance SMMU efficiency, which will not be elaborated further. The above description is summarized in the block diagram below:
Interrupt Redirection
GIC stands for Generic Interrupt Controller. The main function of the GIC is to route various interrupt information to the CPU. The interrupt handling situation in a virtualization environment is much more complex; some interrupts generated by instruction traps need to be handled by the hypervisor itself; some interrupts generated by passthrough devices assigned to a virtual machine need to be processed by software within the virtual machine; and there are also cases where interrupts are received while the corresponding virtual machine is not in a running state. This requires the GIC to provide mechanisms to redirect interrupts to meet these complex demands.
Since the focus of this article is on explaining the support for passthrough device virtualization, this section will introduce how the GIC supports interrupt remapping for passthrough devices without delving into the internal composition and implementation principles of the GIC. Taking GICv3 as an example, the illustration is as follows:
Accompanied by the following steps:
1. The interrupt IRQ initiated by the passthrough device reaches the GIC; 2. The GIC sends the physical IRQ to the CPU; 3. The hypervisor reads the interrupt information from the Physical CPU Interface; 4. The hypervisor discovers that this interrupt was triggered by a passthrough device in a virtual machine, so it re-injects a virtual IRQ into the GIC by writing to the GIC List Register; 5. The GIC generates a vIRQ interrupt and sends it to the vCPU; 6. Finally, the vCPU receives the vIRQ interrupt, which is handled by the Guest OS running on it, and processes subsequent interrupt tasks through interaction with the Virtual CPU interface.
Thus, a complete interrupt remapping process concludes.
After the GIC participates in the interrupt remapping for passthrough devices, the virtual machine has a significant improvement in interrupt processing efficiency. However, compared to ordinary physical interrupts on the host, there is always an additional step of involvement from the software hypervisor. Therefore, we refer to the above method as indirect interrupt injection. To further enhance interrupt processing efficiency, GICv4 introduces direct interrupt injection. The specific principle is that when the passthrough device receives an interrupt, the GIC maps the physical interrupt assigned to the device with the virtual interrupt. Thus, when the device triggers an interrupt, the GIC directly signals the vCPU. In cases where the device triggers an interrupt while the vCPU is not running, i.e., the virtual machine is in a sleep state, if the vIRQ is sent directly to the vCPU, there will be no response. In this case, a physical door-bell interrupt is generated instead, which is handled by the hypervisor, which will reschedule the vCPU to ensure timely response to the interrupt.
This article explains how to support device passthrough from the perspective of virtualization, detailing the principles of ARM architecture supporting DMA remapping and interrupt redirection, aiming to popularize some virtualization-supporting technologies under the ARM architecture. Of course, ARM’s support for virtualization is not limited to this. With the continuous improvement of the ARM ecosystem, these technologies will become more widely known.
END
关于安芯教育
安芯教育是聚焦AIoT（人工智能+物联网）的创新教育平台，提供从中小学到高等院校的贯通式AIoT教育解决方案。
安芯教育依托Arm技术，开发了ASC（Arm智能互联）课程及人才培养体系。已广泛应用于高等院校产学研合作及中小学STEM教育，致力于为学校和企业培养适应时代需求的智能互联领域人才。