Advancements in RV Virtualization Technology

✦

✧

RV Virtualization Progress Special

✦

✦

Editor’s Note

✦

This month, the RISC-V ecosystem presents three main lines: enhanced security, breakthroughs in virtualization performance, and improvements and innovations in hardware architecture. The iteration of the IOPMP and IOMMU specifications marks the systematic construction of hardware-level security capabilities, while the advancement of ACPI RIMT fills the gap in standardized system configuration for RISC-V. In terms of virtualization, from PMU snapshots to Steal Time statistics, from nested acceleration proposals to IOMMU passthrough support, the technical solutions have shifted from functional implementation to deep performance analysis, providing critical infrastructure for cloud computing scenarios. Notably, the CoVE architecture challenges the x86 monopoly on confidential computing with its open design, highlighting RISC-V’s differentiated advantages in heterogeneous security tracks. As core specifications like Debug are implemented, RISC-V is accelerating its evolution from embedded to data center-level applications, with the collaborative standardization process among multiple vendors becoming a key driving force for ecosystem prosperity.

At the same time, we are soliciting recent advancements in RISC-V technology. We welcome progress on various topics, including but not limited to RISC-V architecture design, virtualization, trusted computing, system software, application software, AI large models, and application ecosystems. For specific matters, you can send an email to: [email protected]

✦

Content Guide

✦

Security and Memory Management

• The IOPMP specification v0.9.2-RC3 adds definitions for non-priority error types, handling of stalled transaction faults, standardization of register default values, and fixes the MSI write error logging issue. A patch for Qemu virt simulation has been released (link to be fixed).

• The IOMMU ATC expansion has open-sourced Rivos support for invalidating page table non-leaf level cache entries and QEMU/Linux driver patches for NAPOT address ranges, optimizing address translation efficiency in virtualization scenarios.

• The internal review of the ACPI RIMT specification defines a RISC-V specific ACPI table for IOMMU configuration, supporting PCIe device DMA security and address translation, comparable to x86/ARM virtualization capabilities.

Virtualization and Performance Optimization

• The Supervisor software event standardization SBI v3.0-rc1 introduces an event delivery framework, supporting RAS faults, performance events, and asynchronous page fault handling in virtualization, optimizing the paravirtualization lock mechanism.

• PMU snapshots and virtualization acceleration Rivos achieves shared memory counter delivery, reducing VM trap overhead, with KVM supporting Perf sampling. The Smcdeleg/Sscofpmf extension proposal reduces trap occurrences by 60%.

• The Steal Time patch updates the Linux kernel to add RISC-V paravirtualization support, extending SBI STA to account for VM CPU resource preemption time, improving the KVM testing framework.

Hardware Architecture and Debugging

• The RISC-V Debug 1.0 specification approval clarifies external/internal debugging scenarios, covering hardware boot, operating system debugging, and process-level issue diagnosis, supporting low-level debugging without software dependencies.

• The nested virtualization acceleration proposal reconstructs the scope of HSTATUS.VTVM, proposing a CSR access transformation mechanism that reduces the trigger frequency from VS mode to HS mode via the HNESTED register.

System-Level Innovations

• The IOMMU passthrough support patch adds a RISC-V IOMMU driver to the Linux kernel, supporting single-level address translation and paging domains, providing a foundation for PCIe device passthrough.

• The CoVE confidential computing architecture provides scalable TEE based on RISC-V hardware isolation and dynamic memory management, reducing physical key dependencies compared to Intel TDX solutions, while being compatible with KVM virtualization.

✦

Virtualization Security

✦

The IOPMP specification v0.9.2-RC3 has been publicly released and passed internal review

On January 6, IOPMP community chair Paul Ku publicly announced the new version of the IOPMP specification, and on January 30, announced that this version has passed internal review and will enter the “stable” phase. Additionally, Andes has provided a new patch to implement IOPMP simulation on Qemu virt, but it seems the patch link is currently inactive.

Changes include:

1. Clarification of error types triggered by non-priority rules.

2. Support for fault handling of stalled transactions.

3. Clarification of register definitions: default values and operation types.

4. Logging errors caused by MSI write failures.

5. Fixing typos and inconsistencies.

Document download link:

https://github.com/riscv-non-isa/iopmp-spec/releases/download/v0.9.2-RC3/2025-0106-iopmp-v0.9.2-RC3.pdf

Author: Huang Zhibai

The IOMMU ATC invalid command enhancement implementation on QEMU has been open-sourced

Rivos proposed two extensions for IOMMU ATC in June 2024: the first extension supports invalidating cache entries from non-leaf levels of the page table during address-directed invalidation operations; the second extension supports using NAPOT (naturally aligned power of two range) address ranges as parameters in address-directed invalidation commands. They have now publicly released the implementation of these two extensions based on the RV-IOMMU Qemu model for reference.

Community email address

https://lists.riscv.org/g/tech-iommu/topic/

iommu_atc_invalidation/105374497

Code modification Patch

QEMU model:

hw/riscv/riscv-iommu:

https://github.com/tjeznach/qemu/commit

/d37ec2d255c8391168fd4913443ec94a2

cd43717

Linux iommu/riscv driver:

iommu/riscv:

https://github.com/tjeznach/linux/commit

/3a601d356568c0929229d56474a97212

781b7c4d

https://github.com/tjeznach/linux/commit

/ef51d4be750d2d7c449747ab53fd5c12d

5a4a175

Author: Huang Zhibai

Discussion on sPMP

Recently, the RISC-V community has engaged in in-depth discussions regarding the updates to the SPMP (Supervisor Physical Memory Protection) specification. This specification aims to provide a lightweight memory protection mechanism for scenarios such as the Internet of Things, and is nearing its final version, although some technical details remain controversial. Community members are collaborating via emails and meetings to advance resolutions.

Link:

https://lists.riscv.org/g/tech-spmp/topic/

update_the_spmp_specification/110628384

Author: Xu Kailiang

Internal review of the ACPI RISC-V IO Mapping Table (RIMT) specification

The RIMT specification defines a RISC-V specific static ACPI table for RISC-V platforms, conveying IOMMU configuration information to the operating system, thereby supporting IOMMU functionality. This enables hardware virtualization support similar to x86/ARM in RISC-V systems, enhancing the DMA security and address translation capabilities of peripherals (such as PCIe devices).

Link:

https://lists.riscv.org/g/tech-iommu/topic/

internal_review_of_the_acpi/108281618

The PDF file of this specification:

https://github.com/riscv-non-isa/riscv-acpi-rimt/releases/download/v1.0.0-rc1/rimt-spec.pdf

The source code of this specification:

https://github.com/riscv-non-isa/riscv-acpi-rimt

Author: Xue Songtao

Scalable Confidential Compute Architecture (CoVE/AP-TEE)

Provides **scalable confidential computing support** for the RISC-V platform, protecting the data usage security of virtual machines (TVM) through a hardware-verified Trusted Execution Environment (TEE), while minimizing the size of the Trusted Computing Base (TCB) and being compatible with existing virtualization software stacks (such as KVM). Key technologies include:

Hardware Isolation Mechanism

• Confidential Qualifier: Each hardware thread (hart) status identifier, combined with MMU/PMP extensions, controls access to confidential memory (accessible only when hart is in confidential mode).

• Memory Tracking Table (MTT): A dynamic physical memory attribute (PMA) mechanism that isolates host and TVM memory through page-level metadata (such as confidentiality identifier C-bit), prohibiting cross-domain access.

• Trusted Execution Environment Security Manager (TSM): As the core of the TCB, responsible for TVM lifecycle management, memory allocation, and secure context switching. Interacts with the SoC Root of Trust (RoT) through TSM drivers (M-mode component) to support hardware remote attestation.

ISA Extensions

• TEECALL/TEERET instructions: Secure context switching, isolating host and TSM execution flows.

• Confidential Interrupt Isolation: Based on the RISC-V AIA architecture, preventing the host from leaking TVM data through interrupts.

Link:

https://arxiv.org/pdf/2304.06167

https://github.com/rivosinc/salus

Author: James Yen

✦

Virtualization Performance Optimization

✦

Supervisor Software Events

Added in SBI Version 3.0-rc1

Some of the typical use cases for SSE are to deliver:

1. RAS events in Firmware first approach.

2. Perf events.

3. Paravirtualized async page faults in virtualization.

Link:

https://lists.riscv.org/g/techprs/

message/798

Author: Xu Kailiang

Paravirt Locking

Application in Linux (RFC PATCH)

https://lore.kernel.org/linux-riscv/

20241227011011.2331381-1-

[email protected]/

Reduce spinlock waiting in the guest using paravirtualization.

SBI patch

https://lists.riscv.org/g/tech-prs/topic/

patch_v2_introduce_pvlk/110344730

Author: Xu Kailiang

SBI PMU Extension Specific Improvements and Clarifications

Improvements: The original patch required 2 SBI calls for hardware and cache events, while this version only requires one call.

New: Support for raw/firmware events

Link:

https://lists.riscv.org/g/tech-prs/message/1008

Author: Zhang Ziyang

Add SBI Message Proxy (MPXY) Extension

SBI M-Mode can support various messaging protocols. Adding the SBI Message Proxy (MPXY) extension allows hypervisor clients to use new messaging protocols.

Link:

https://lists.riscv.org/g/tech-prs/

message/1079

Author: Zhang Ziyang

PMU Snapshot

The RISC-V community has recently welcomed a key advancement, with Rivos developer Atish Patra submitting 24 patches for the Linux kernel that implement improvements to the PMU (Performance Monitoring Unit) in the RISC-V SBI v2.0 specification and support Perf performance sampling functionality for the first time in KVM virtualization environments. This breakthrough will significantly enhance RISC-V’s performance analysis capabilities in cloud computing and virtualization scenarios.

Core Improvements

• SBI PMU snapshot functionality: Passes counter information through shared memory, reducing trap occurrences between the hypervisor and guest, lowering performance overhead.

• Firmware high read (fw_read_hi): Supports complete reading of 64-bit counters on a 32-bit RISC-V architecture, ensuring accuracy of performance data.

• KVM guest Perf sampling: For the first time, implements hardware event sampling (such as CPU cycles, instruction counts) in KVM virtual machines, helping developers pinpoint performance bottlenecks accurately.

Link:

https://lwn.net/Articles/969482/

Author: Wei Zhixiang

PMU Enhancements on Passthrough

The RISC-V community recently engaged in technical discussions around optimizing the performance monitoring unit (PMU) passthrough in virtualization environments, proposing innovative solutions based on hardware extensions Smcdeleg and Sscofpmf, aimed at significantly reducing performance overhead in virtualization scenarios, attracting widespread attention. In virtualization environments, accessing PMU registers and handling interrupts typically requires proxying through the hypervisor, leading to frequent context switch traps and SBI (Supervisor Binary Interface) calls, creating performance bottlenecks. For example, currently, each context switch triggers 4-9 traps, and additional 2+n calls during sampling collection.

Core of the Technical Solution

• Smcdeleg: Through a delegation mechanism, grants PMU register access directly to the virtual machine (VS mode), eliminating traps. VS mode should be limited to configuring the VUINH/VSINH flags to ensure PMU automatically stops counting upon VM exit, avoiding security risks.

• Sscofpmf: Introduces fine-grained PMU overflow interrupt delegation, allowing virtual machines to independently handle certain counter interrupts while retaining the host’s monitoring capability for critical events.

• Significantly reduced trap occurrences: After enabling snapshots, context switch traps decrease from 9x to 4x; combined with the delegation mechanism, it can further reduce to 5x, and sampling traps decrease from (2+n)*2 to 2.

• Counter splitting and lazy saving: Proposes passing some PMU registers directly to the virtual machine, with the host using the remaining part, combined with a lazy save strategy to reduce context switch overhead during multi-vCPU sharing.

• Security and compatibility: Discussions referenced similar solutions in x86 architecture (such as selector traps) as a reference, emphasizing the need for RISC-V to balance standardization and flexibility.

Link:

https://lists.riscv.org/g/sig-hypervisors/

message/528?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C%2CPMU%2C20%2C2%2C0%2C109318369

Author: Wei Zhixiang

Steal Time Accounting

This is a series of Linux kernel patches for the RISC-V architecture aimed at implementing “steal-time” support. These patches are divided into three main parts:

• RISC-V paravirt support (patches 01-03): Adds paravirt (paravirtualization) support for RISC-V and implements steal-time statistics using the SBI (Supervisor Binary Interface) STA extension.

• KVM SBI STA implementation (patches 04-09): Implements the SBI STA extension in KVM (Kernel-based Virtual Machine) to allow KVM guests (for example, Linux guests with paravirt steal-time support enabled) to understand steal-time.

• KVM self-testing (patches 10-13): Adds support for RISC-V in KVM self-testing, including steal_time tests and SBI extended registers in get-reg-list tests.

Core Concepts:

• Steal-time: In virtualization environments, a virtual CPU (VCPU) may be unable to run due to the host scheduling other tasks (such as other VCPUs or host processes). Steal-time refers to the time during which the VCPU cannot run for this reason.

• SBI (Supervisor Binary Interface): A standard interface defined by RISC-V for communication between S-mode software (such as operating systems or hypervisors) and underlying platform firmware (commonly referred to as M-mode firmware).

• SBI STA extension: An extension of the SBI that allows guests (virtual machines) to query their VCPU’s steal time.

• Paravirt (paravirtualization): A virtualization technology in which the guest operating system is aware that it is running in a virtual machine and uses specific interfaces (such as SBI) to communicate with the hypervisor to enhance performance.

• KVM (Kernel-based Virtual Machine): A full virtualization solution provided in the Linux kernel.

Overview of the Patch Content:

The purpose of these patches is to allow the Linux kernel guest operating system on RISC-V to perceive the runtime loss caused by host resource scheduling. This is particularly important in resource-overcommitted virtualization environments as it helps the guest operating system understand why its performance is below expectations. The specific implementation is through the addition of support for the SBI STA extension, which provides a standardized interface for the guest to query its VCPU’s “steal time.” The patches also include the implementation of this extension in KVM, along with corresponding kernel configuration options and self-test.

Link:

https://lore.kernel.org/all/[email protected]/T/

Author: James Yen

Accelerating Nested Virtualization

Recently, the scope of the HSTATUS.VTVM bit in the RISC-V H extension specification needs further clarification. According to existing specifications, when HSTATUS.VTVM is 1 and V is 1, all virtual memory management instructions (including SFENCE and HFENCE) will trigger entry into HS mode. However, in reality, only the SFENCE instruction needs to trigger entry into HS mode in this case, while the HFENCE instruction is only defined in HS mode (i.e., when V is 0), so it does not need to trigger when V is 1.

To better describe nested virtualization in RISC-V, the following virtualization privilege modes are defined:

• Host HS-mode: The hypervisor kernel of the host runs in this mode, and software runs in hardware HS mode.

• Host U-mode: The user space of the host hypervisor runs in this mode, and software runs in hardware U mode.

• Guest HS-mode: The hypervisor kernel of the guest runs in this mode, and software runs in hardware VS mode.

• Guest U-mode → HW VU-mode: The user space of the guest hypervisor runs in this mode, and software runs in hardware VU mode.

• Guest VS-mode → HW VS-mode: Software runs in hardware VS mode.

• Guest VU-mode → HW VU-mode: Software runs in hardware VU mode.

High-Level Software Implementation Methods for Nested Virtualization:

• Host HS-mode (host hypervisor) enables HSTATUS.VTSR to simulate the Guest’s SRET instruction. This simulation process will include performing CSR world switching when switching from Guest HS/U mode to Guest VS/VU mode.

• After performing CSR world switching, virtual interrupts will be injected into the Guest VS/VU mode.

• For Guest’s “h” and “vs” register accesses, it will trigger entry into Host HS-mode (host hypervisor), where: a) these CSRs will be simulated for Guest HS mode. b) For Guest U mode and Guest VS/VU mode, the trigger will be forwarded to Guest HS mode.

• Host HS-mode (host hypervisor) will manage two Stage2 page tables: a) A regular Stage2 page table for Guest HS/U mode. b) An image Stage2 page table for Guest VS/VU mode. The host hypervisor needs to traverse the HGATP page table under Guest HS mode when filling in the image Stage2 page table.

• All HFENCE instructions will trigger entry into Host HS-mode, where the host hypervisor will: a) Simulate HFENCE.VVMA and HFENCE.GVMA instructions for Guest HS mode. b) Forward triggers from Guest VS mode for HFENCE.VVMA and HFENCE.GVMA to Guest HS mode.

• All HLV/HSV instructions from Guest HS/U mode and Guest VS/VU mode will trigger entry into Host HS-mode (host hypervisor), where: a) HLV/HSV instructions from Guest HS/U mode will be simulated by the host hypervisor. b) HLV/HSV instructions from Guest VS/VU mode will be forwarded to Guest HS mode by the host hypervisor.

Proposal for Accelerating Nested Virtualization Performance:

Based on the above high-level software implementation methods, a method to accelerate nested virtualization performance is proposed, aiming to reduce the triggers for CSR access from VS mode to HS mode (as described in point 3). The goal of this proposal is to enhance the performance of nested virtualization, allowing RISC-V platforms to operate more efficiently in multi-layer virtualization environments.

Link:

https://lists.riscv.org/g/tech-privileged/topic/proposal_for_accelerating/72021208

Author: Huang Minglang

✦

Virtualization Hardware Architecture

✦

Discussion on AIA Support for Multiple MSI Vectors

Differences Between MSI-X and Multi-MSI

MSI-X Multi-Vectors:

• Each MSI-X vector has independent MSI address and data.

• AIA and Linux IMSIC drivers have supported this (by allocating independent descriptors for each MSI-X vector through the PCI framework).

• Verified scenario: When PCIe devices use multiple MSI-X vectors, interrupt distribution to different CPUs works normally.

Multi-MSI:

• Multiple interrupts share the same MSI address, distinguished only by contiguous MSI data values.

• AIA v1.0 does not support: Modifying the MSI address of a single interrupt affects all interrupts, making it impossible to route independently to different CPUs.

• Current status in Linux: IMSIC driver does not support multi-MSI. Earlier attempts (v11 driver version) were abandoned due to technical issues.

Reasons for Not Supporting Multi-MSI

• Architectural limitations: AIA’s design did not consider flexible routing support for shared address multi-MSI interrupts.

• Industry practice: Mainstream PCIe devices tend to use MSI-X (for example, Linux on x86 architecture also does not support multi-MSI).

• Development priorities: Lack of sufficient demand to drive AIA specifications or Linux driver adaptation for multi-MSI.

Author: Xue Songtao

RISC-V Debug, Load/Store Pair for RV32, and Four Other Specifications Ratified at February RISC-V Board of Directors Meeting

This specification mainly involves debugging capabilities in the RISC-V architecture, divided into two scenarios: internal debugging (or self-hosted debugging) and external debugging.

(1) Accessing hardware on the hardware platform (external debugging): External debugging allows access to and debugging of hardware modules when the CPU on the hardware platform is not functioning properly, helping diagnose and fix issues.

(2) Booting the hardware platform for testing, configuration, and programming (external debugging): External debugging can guide the hardware platform for initialization settings, testing hardware components, and configuring required operations when there is no executable code path available.

(3) Debugging low-level software without operating system or other software support (external debugging): External debugging supports debugging and testing low-level software without operating system or other software support, ensuring its correctness and functionality.

(4) Debugging issues with the operating system itself (external or internal debugging): External or internal debugging can diagnose and fix issues with the operating system itself, ensuring stability and reliability during runtime.

(5) Debugging processes running on the operating system (internal or external debugging): Processes running in an operating system environment can be monitored, analyzed, and debugged through internal or external debugging to resolve runtime issues and exceptions.

Specific content can be referred to:

https://lf-riscv.atlassian.net/browse/RVS-981

Author: Wang Yuan

IOMMU for Device Pass-Through

A series of patches have been released for the Linux kernel, adding support for RISC-V architecture IOMMU hardware. The release of this patch series marks a new phase in the implementation of the RISC-V IOMMU specification.

The RISC-V IOMMU specification has been officially approved and can be reviewed on GitHub (link: riscv-iommu). The design goal of this specification is to provide an efficient input-output memory management unit for the RISC-V platform, capable of supporting more flexible device resource management and virtualization.

Overview of the RISC-V IOMMU Specification:

Data Structures:

1. Device Context: Associates devices with address spaces, storing address translation parameters for each device.

2. Process Context: Associates different virtual address spaces based on the process identifier provided by the device.

3. MSI Page Table Configuration: Directs MSI interrupt messages to the guest interrupt file.

Memory Queue Interfaces:

1. Command Queue: Sends commands to the IOMMU.

2. Fault/Event Queue: Reports faults and events.

3. Page Request Queue: Reports “page request” messages from PCIe devices.

4. Message signal interrupts and linear signal interrupt mechanisms.

Memory Mapping Programming Interfaces:

1. Mandatory and optional register layouts and descriptions.

2. Software guidelines for device initialization and functionality discovery.

Main Improvements of This Patch Series:

This patch series introduces hardware initialization for RISC-V IOMMU for the first time, supporting complete single-level address translation and paging domain functionality. Specific updates include:

• Patch 1: Adds device tree bindings required for the RISC-V IOMMU driver.

• Patch 2: Defines data structures, hardware programming interface register layouts for RISC-V IOMMU, and provides minimal initialization code to enable global passthrough mode.

• Patch 3: Implements the RISC-V IOMMU hardware platform driver for PCIe devices.

• Patch 4: Introduces IOMMU interface for kernel subsystems.

• Patch 5: Implements device directory management, including I/O mapping or memory device directory table location discovery, hardware capability discovery, and additional device to domain implementations.

• Patch 6: Implements command queue and fault queue, and introduces directory cache invalidation sequences.

• Patch 7: Introduces paging domains, supporting the highest page table modes declared by hardware. Currently, only 4K page mappings are supported, and subsequent patches will support large page mappings.

Subsequent patch series will further improve large page support, update cache management mechanisms, and provide complete ATS, PRI, and SVA support.

Link:

https://patchwork.kernel.org/project/linuxriscv/cover/cover.1717612298.git.

[email protected]/

Author: Huang Minglang

✦

END

✦