With the popularity of cloud computing and artificial intelligence applications, GPU virtualization technology plays an increasingly important role in improving hardware utilization and reducing costs.NVIDIA vGPU technology allows multiple virtual machines (VMs) to share a single physical GPU, enabling flexible resource allocation and high-performance computing support.This article systematically analyzes the software architecture, implementation details, and future roadmap of the latest vGPU upstreaming solution proposed by NVIDIA at the KVM Forum 2025.
1 Introduction
GPU virtualization has wide applications in scenarios such as cloud gaming, AI inference, and graphic workstations. Traditional GPU virtualization solutions face issues such as high performance overhead and complex management.
NVIDIA vGPU is based on hardware-assisted virtualization technology, exposing GPU resources to virtual machines through SR-IOV virtual functions (VF), achieving near-native performance.
In recent years, NVIDIA has promoted the upstreaming of vGPU drivers, aiming to integrate core functionalities into the Linux kernel and KVM/QEMU mainline, simplifying deployment and maintenance.
2 System Architecture
2.1 Overall Architecture
The NVIDIA vGPU architecture mainly includes the following components:
PCIe SR-IOV Virtual Functions (VF): The physical GPU exposes multiple VFs, each of which can be assigned to a VM.
NVIDIA vGPU VFIO Driver: Runs in the host kernel space, responsible for vGPU lifecycle management, resource allocation, and VFIO device operations.
nova-core: A core driver implemented in Rust, handling PF driver event dispatch and GPU resource management.
GSP (GPU System Processor): Responsible for executing management tasks related to GPU virtualization, including vGPU instance creation, resource binding, and isolation.
QEMU/VFIO: Provides the interface for virtual machines to access vGPU, supporting advanced features such as live migration.
2.2 Software Stack Interaction Process
vGPU Type Selection: Administrators select the vGPU type through the sysfs interface (defining GPU resource quotas).
Resource Allocation: Allocating framebuffer (FB) memory for vGPU, managing heap memory, command memory, and work submission channels.
vGPU Bootstrapping: Creating vGPU instances in GSP and registering resources.
Configuration and Isolation: Programming hardware units to ensure resource isolation between VMs.
Operation and Management: Supporting applications within VMs to access the GPU via Guest Driver, with the VFIO Driver handling device IOCTL and interrupts.
2.3 Key Data Structures
vGPU Metadata File: Defines available vGPU types and their attributes (such as memory size, display capabilities, etc.).
Management RPC Channel: Communication between the VFIO Driver and GSP for lifecycle management and configuration.
3 Core Implementation
3.1 vGPU Lifecycle Management
Creation: Allocating resources, initializing RPC, and submitting FB memory cleanup tasks.
Destruction: Releasing resources and notifying GSP to destroy the instance.
Type Switching: Must be set through sysfs before vGPU creation.
3.2 Resource Management
Each vGPU instance requires the following resources:
Guest GPU Memory: Framebuffer visible to the virtual machine.
Management Heap Memory: Buffer for communication between the VFIO Driver and GSP.
Command Memory: Buffer for storing GPU commands.
Work Submission Channels: Divided into Guest-visible channels and VFIO Driver exclusive channels.
3.3 Live Migration Support
Through the VFIO device live migration interface, combined with GSP state saving and recovery mechanisms, online migration of vGPU is achieved.
4 Performance and Functional Features
Performance is close to native GPU performance, supporting 3D and CUDA workloads. Supports dynamic creation/destroying of vGPU, type switching, and live migration. Per-vGPU logs and status information can be exposed through debugfs.
5 Upstreaming Progress and Roadmap
5.1 Current Status (as of September 2025)
Release of RFC v2 (based on nova-core and GSP microcode 570). Supports multi-vGPU homogeneous deployment and FB memory cleanup, achieving vGPU creation/destroying/selection interfaces.
Support for RTX A6000 Ada workstation vGPU type.
The future plan is to expand support for more GPU models and vGPU types, enhance the stability and performance of live migration, improve the flexibility and security of management interfaces, and continue to promote driver upstreaming into the Linux mainline.
6 Conclusion
The NVIDIA vGPU upstreaming solution achieves high-performance, manageable GPU virtualization through hardware-assisted virtualization and the VFIO framework.
Combining the designs of nova-core and GSP not only enhances performance and security but also simplifies deployment and maintenance.
Research results indicate that this solution can efficiently support 3D graphics and CUDA computing workloads, achieving complete lifecycle management and live migration capabilities.
In the future, as more features enter the upstream kernel, this solution is expected to become the mainstream choice for GPU virtualization in KVM environments.
Written by: ZevornReviewed by: ZevornImages and References:
1. Wang, Z. (2025). Upstreaming NVIDIA vGPU Support: Architecture, Implementation, and Roadmap. KVM Forum 2025.
2. RFC v2 patches: https://lore.kernel.org/all/[email protected]/
3. Kernel repository: https://github.com/zhiwang-nvidia/linux/tree/zhi/vgpu-rfc-v2
4. vGPU metadata file: https://github.com/zhiwang-nvidia/vgpu-tools/blob/metadata/metadata/18.1/vgpu-570.144.bin
5. KVM Forum 2025 Live stream – Day 1 – 3.0.2. Bilibili. https://b23.tv/JHZOsPv
–
Follow our public account for free access to the community-provided ima knowledge base.
Now launched:QEMU/GEM5/Compiler/Linux