Summary With the rise of large models, the demand for GPU computing power is increasing. However, the current reality often limits enterprises to a limited number of GPU cards. Even with virtualization, it is often difficult to fully utilize GPU card resources or to maintain resource usage. To address issues like uneven GPU computing power resources and to support domestic alternatives for GPU computing power while improving the utilization of GPU resources, the need for GPU computing power pooling is urgent. This article shares several routes for GPU device virtualization, GPU virtualization and sharing solutions, and the cloud-native implementation of GPU computing power pooling.
Author Wang Zhaohui, Architect at China Galaxy Securities, focuses on container cloud, microservices, DevOps, data governance, digital transformation, and has unique insights into related technologies. He is skilled in software planning and design, and his “platform integration” viewpoint is increasingly recognized and proven. He has published numerous technical articles discussing container platform construction, microservices technology, DevOps, digital transformation, data governance, and middle platform construction, receiving widespread attention and recognition.
Intelligent applications such as facial recognition, voice recognition, text recognition, intelligent recommendation, intelligent customer service, and intelligent risk control are widely used across various industries. These applications fall under the category of judgment-based AI, which are typically bound to specific business scenarios. Therefore, when using GPU (Graphics Processing Unit) cards, they are usually independent of each other, without considering the GPU sharing capabilities between businesses. At most, vGPU virtualization segmentation is achieved, allowing a single physical GPU card to virtualize multiple vGPUs, which can run multiple judgment-based AI applications. With the rise of large models, the demand for GPU computing power is increasing, while the current reality often limits enterprises to a limited number of GPU cards, making it difficult to support numerous business needs. Additionally, due to business characteristics, even with virtualization, it is often difficult to fully utilize GPU card resources or maintain resource usage, resulting in limited card resources being effectively utilized.
From GPU Virtualization Demand to Pooling Demand
The increase in the number of intelligent applications has led to a growing demand for GPU computing power resources. Although NVIDIA provides GPU virtualization and multi-GPU instance segmentation solutions, it still cannot meet the demand for freely defined virtual GPUs and the sharing and reuse of the entire enterprise’s GPU resources. Applications developed using intelligent application frameworks like TensorFlow and Pytorch often monopolize a complete GPU card (the AntMan framework is designed for sharing), leading to GPU card shortages. On the other hand, most applications only use a small portion of the card’s resources, such as ID card recognition, invoice recognition, voice recognition, and research analysis. The utilization rate of GPU cards in these inference scenarios is relatively low, even dropping to 0% when there are no business requests, with computing power limited by the finite number of cards.The occupation of a single inference scenario on one card causes significant waste, creating a contradiction with the insufficient number of cards. Therefore, computing power segmentation is a basic requirement in many scenarios.Furthermore, often limited by organizational structures, GPUs are procured and used independently by various teams, leading to isolated computing resources that are unevenly distributed. Some teams have idle GPU resources, while others have no cards available.
To solve the problems of uneven GPU computing power resources and to support domestic alternatives for GPU computing power, it is essential to coordinate online and offline resource demands, business peak and off-peak resource demands, training and inference, as well as the different resource demands of development, testing, and production environments. This can achieve unified management and scheduling reuse of computing power, enabling the segmentation, aggregation, super-dividing, remote invocation, application hot migration, and other capabilities of GPU resources, thereby increasing the utilization of GPU resources. Thus, the need for GPU computing power pooling is urgent.
GPU Device Virtualization Routes
There are several feasible solutions for GPU device virtualization.
First is the PCIe pass-through mode (PCIe Pass-through technology, pGPU), which directly mounts a physical GPU card from the host to a virtual machine for use. However, this method is exclusive, and the GPU card is not virtually segmented, which does not solve the problem of multiple applications running on one card, thus its significance is limited.
The second method uses SR-IOV technology, allowing a PCIe device to be shared among multiple virtual machines while maintaining high performance. This is achieved by creating multiple virtual vGPUs on the physical GPU device through SR-IOV, with each virtual vGPU allocated to a virtual machine, allowing the virtual machine to directly access and control these virtual functions, thus achieving efficient I/O virtualization. NVIDIA’s early vGPU is such an implementation, but NVIDIA vGPU requires an additional license, which increases costs. Although SR-IOV achieves a 1:N capability, its flexibility is relatively poor, making finer granularity segmentation and scheduling difficult.
The third method is the MPT (Mediated Pass-Through) mode. MPT is essentially a general PCIe device virtualization solution. It balances 1:N flexibility, high performance, and functional integrity, but logically corresponds to a device model implemented in kernel mode, and vendors usually do not disclose hardware programming interfaces, so adopting MPT may lead to vendor dependency.
The most commonly used mode is the API forwarding mode. Based on the calling hierarchy of AI applications (as shown in the figure below), API forwarding has multiple layers, including CUDA API forwarding (as shown in Figure 1), GPU Driver API forwarding (as shown in Figure 2), and device hardware layer API forwarding (as shown in Figure 3). Device hardware layer APIs are typically difficult to obtain, so the market usually adopts the CUDA API forwarding mode (intercepting and forwarding CUDA requests, also known as user mode) and GPU card driver API forwarding mode (intercepting and forwarding driver layer requests, also known as kernel mode).
Moreover, AI development frameworks are often bound to GPU cards (for example, Huawei supports the CANN framework, Haiguang supports the DTK framework, and NVIDIA supports TensorFlow, Pytorch, and other frameworks). When AI applications use AI frameworks, they can also forward at the AI framework level, which is useful when migrating AI applications.
AI Application Calling Hierarchy
GPU Virtualization and Sharing Solutions
Having understood the methods of GPU device virtualization, let’s explore the implementation methods of GPU virtualization and sharing based on device virtualization technology. There are various solutions for GPU virtualization and sharing, with NVIDIA officially providing vGPU, MIG, MPS, and various unofficial solutions such as vCUDA, rCUDA, and kernel hijacking.
NVIDIA VGPU Solution
NVIDIA vGPU is a virtualization solution provided by NVIDIA, known for its high reliability and security, but it does not support containers and can only virtualize several vGPUs with limited flexibility; it cannot dynamically adjust resource ratios; there is a certain amount of sharing loss; it does not support custom development and requires additional license fees.
MIG Solution
MIG is the Multi-Instance GPU solution. It only supports Linux operating systems and requires CUDA 11/R450 or higher; supported MIG cards include high-end cards like A100 and H100; it supports bare metal and containers, and the vGPU mode allows dynamic management of instances once the GPU card is set up with MIG. MIG settings are persistent, meaning they are not affected by reboots until the user explicitly switches them. With MIG, users can obtain up to 7 times the GPU resources on a single GPU card, providing more resources and higher flexibility for developers. It optimizes GPU utilization and supports running inference, training, and high-performance computing (HPC) tasks simultaneously on a single GPU. Each MIG instance operates as an independent GPU for applications, ensuring the programming model remains unchanged and is user-friendly for developers.
MPS (Multi-Process Scheduling)
MPS multi-process scheduling is an alternative binary-compatible implementation of the CUDA application programming interface. Starting with the Kepler GP10 architecture, NVIDIA introduced MPS, allowing multiple streams or CPU processes to simultaneously launch kernel functions to the GPU, combining them into a single application context running on the GPU, thus achieving better GPU utilization. When using MPS, the MPS Server manages GPU hardware resources through a CUDA Context, while multiple MPS Clients pass their tasks to the GPU through the MPS Server, bypassing the limitations of hardware time-slicing scheduling, enabling true parallel execution of their CUDA kernels. However, MPS also brings a fatal flaw due to shared CUDA Contexts, leading to poor fault isolation; if a task executing a kernel exits, any tasks sharing IPC and UVM with that task will also exit with errors.
rCUDA
rCUDA refers to remote CUDA, a remote GPU invocation solution that supports concurrent remote use of CUDA devices in a transparent manner. rCUDA provides a way for non-GPU nodes to access and use GPUs, allowing AI applications to run on non-GPU nodes. rCUDA is a client/server architecture where the client uses the CUDA runtime library to remotely invoke GPU interfaces on the server, which monitors requests and executes them using the GPU, returning the execution results. In practical scenarios, there is no need to configure GPU resources for local nodes; remote invocation of GPU resources can be achieved without concern for the GPU’s location, which is a crucial capability that isolates applications from GPU resource layers.
vCUDA
vCUDA adopts a method of intercepting and redirecting CUDA APIs at the user layer, establishing a logical image of pGPU, i.e., vGPU, in the VM to achieve fine-grained division, recombination, and reuse of GPU resources, supporting multi-machine concurrency, suspension recovery, and other advanced VM features. The vCUDA library is a wrapper around the nvidia-ml and libcuda libraries, intercepting CUDA calls from user programs within the container to limit the current container’s processes’ usage of GPU computing power and memory. The advantages of vCUDA include open-source APIs that are easy to implement; however, its drawbacks include rapid CUDA library upgrades requiring constant adaptation, leading to high costs; additionally, inaccurate isolation cannot provide precise limits on computing power, and low security allows users to bypass restrictions. Currently, most vendors in the market implement GPU computing power pooling using the vCUDA API forwarding method.
Cloud-Native Implementation of GPU Computing Power Pooling
GPU pooling (GPU-Pooling) is achieved through software-defined physical GPUs, integrating capabilities such as GPU virtualization, multi-card aggregation, remote invocation, and dynamic release, addressing issues of low GPU usage efficiency and poor elastic scaling. The ideal solution for GPU resource pooling is to shield the underlying heterogeneous resource details (supporting both NVIDIA and domestic GPU vendors), separating the coupling between upper-level AI framework applications and lower-level GPU types. However, currently, AI frameworks and GPU types are tightly coupled, and there is no implemented solution that abstracts a layer capable of shielding heterogeneous GPUs. Applications developed based on different frameworks must be rebuilt when migrating to other types of GPUs, often requiring re-adaptation and debugging.
Computing power isolation and fault isolation are key to GPU virtualization and pooling. Computing power isolation can be achieved through hardware isolation, which is a physical separation, shared CUDA Context methods, and time-sharing methods. The closer to the bottom layer, the better the isolation effect. For instance, the MIG hardware computing power isolation scheme is a form of hardware resource isolation and fault isolation with the best effect. However, hardware device programming interfaces and driver interfaces are often not publicly available, leading to significant vendor dependency, making implementation very difficult and lacking flexibility. For example, high-end cards such as the A100 supporting the Ampere architecture can only be segmented into a maximum of 7 MIG instances. NVIDIA MPS is the best option for computing power isolation aside from MIG. It merges multiple CUDA contexts into a single CUDA context, eliminating the overhead of context switching and achieving computing power isolation within the context, but it also incurs additional fault propagation. Both MIG and MPS have very obvious advantages and disadvantages and are not widely used in practical engineering. The implementation model of multi-task GPU time-slicing using API forwarding is relatively easy to implement and widely applicable.
According to the calling hierarchy of AI applications using GPUs, different levels of resource pooling capabilities can be achieved. For example, at the CUDA layer, Driver layer, hardware device layer, etc. At different abstraction levels, applications that require acceleration can be forwarded to the GPU resource pool. Overall, the closer to the bottom layer the forwarding is performed, the smaller the performance loss and the larger the operable range; however, this also increases the programming workload and difficulty.
With the large-scale deployment of cloud-native applications, GPU computing resource pooling needs to support cloud-native deployment capabilities, such as supporting Kubernetes and Docker services, allowing K8s Pods to bind to vGPUs virtualized on demand from the GPU resource pool to execute applications within the Pods. Regardless of whether it is NVIDIA GPU cards or domestic GPUs, all cards are in the computing power resource pool. Currently, different cards can be classified, and applications from different frameworks can be scheduled to suitable classified GPU computing power pools as needed, thereby improving resource management efficiency. The computing power resource pool also needs to implement corresponding resource, asset management, and operational monitoring and observability, optimizing scheduling capabilities and reducing GPU resource fragmentation. With the rapid growth of AI application demands, GPU computing resource pooling will be an important aspect of enterprise focus in the near future.
If you have any questions, click “Read Original” to leave a message under the community original text
If you find this article useful, please share, like or click “Looking”, so more peers can see it
Recommended materials/articles:
-
Understand RAG (Retrieval-Augmented Generation) in one article【Introduction to Large Model Industry Applications】
-
How can the financial industry improve GPU resource utilization efficiency and accelerate AI intelligent applications?
-
How to quickly build local knowledge base applications based on open-source large models
-
How to build efficient resource allocation strategies to support large model projects?【Large Model Industry Application Implementation Series】
-
Where is AIOps headed next?【Large Models and Automated Operations】
Welcome to follow the community “GPU” technology theme , which will continuously update high-quality materials and articles. Address:https://www.talkwithtrend.com/Topic/111987
Download the twt community client APP
Long press to identify the QR code to download
Or search for “twt” in the app store
Long press the QR code to follow the public account
*The content published by this public account only represents the author’s views and does not represent the community’s position