Dual Engine GPU Container Virtualization: Technical Analysis and Practical Sharing of User and Kernel Modes

Source | Reprinted with permission from Baidu Intelligent Cloud Technology Station Official Account

How to maximize the efficiency of hardware computing power is a concern for all resource operators and users. As a leading AI company, Baidu possesses perhaps the most comprehensive AI application scenarios in the industry.

In this article, we will share and discuss the solutions and best practices of GPU container virtualization in complex AI scenarios.

The left and right parts of the image below have been showcased multiple times in different contexts, but here we mainly want to emphasize the demand for computing power — the exponential growth of hardware computing power versus the contradiction of low utilization and resource waste in real application scenarios.

The left part shows statistics from OpenAI, indicating that since 2012, the computing power required for model training has doubled every 3.4 months. By the time of AlphaGo Zero and similar large models, the training computing power has increased by 300,000 times, and this trend continues. On one hand, as the demand for computing power grows, the computing performance of mainstream AI acceleration units is also increasing at a rate of doubling every two years. On the other hand, resource utilization efficiency restricts the full performance of hardware.

The right part shows the results of Facebook’s analysis of machine learning workloads in data centers in 2021. A large amount of AI computing power is lost due to failures, scheduling, time slice waste, and space unit waste, with actual computing power utilization being less than 30%. We believe this is also the current situation faced by major infrastructure operators in China.

As mentioned earlier, a utilization rate of less than 30% in online clusters may not align with many people’s perceptions. Many online colleagues may be developers of models and algorithms. Our general perception is that during the training and testing processes, utilization can remain at a very high level, even reaching 100% utilization.

However, when models go live in production environments, they are subject to many constraints, which lead to utilization falling far short of our expectations.

Below, we summarize the main limiting factors in a limited space:

• Model characteristics: Each model network is different, and the combinations of underlying operators vary, which significantly affects GPU utilization.
• Service SLA: Different scenarios require different SLAs; some services have high real-time requirements and must be strictly controlled within 10ms, meaning these services cannot improve utilization by increasing batch size, and the batch size may only be 1.
• Traffic patterns: Different model algorithm services cater to different application scenarios. For example, OCR recognition may be frequently called during work hours, while speech recognition is often called during commuting or leisure time, leading to fluctuations in GPU utilization throughout the day.
• Optimization effects: Depending on the iteration frequency of models and the different coverage scenarios, the granularity of model optimization varies. It is easy to imagine that a model that has not been fully optimized will also struggle to achieve high levels of utilization.
• Capacity redundancy: Before models go live, detailed capacity planning must be conducted, including maximum traffic and whether multi-region support is needed. During this process, significant capacity redundancy is reserved, which also causes computing power waste during regular operations.

Under the constraints of the above factors, the actual utilization rate in production environments may be what we are about to demonstrate. We abstracted several utilization patterns from the complex and variable online production environment.

• Low Mean Type: As shown in the upper left image, for a real online inference business, due to model characteristics and service SLA constraints, the peak GPU utilization is only 10%, and the average utilization is even lower.
• Fluctuating Peak Valley Type: As shown in the lower left image, this is a typical utilization pattern for online inference business, where services peak during the day and reach a low valley from midnight to the next morning, with an average utilization of only about 20% throughout the day, and low valley utilization being less than 10%.
• Short-term Surge Type: As shown in the upper right image, the utilization curve is basically consistent with the lower left image, but during the golden hours at night, there are two distinct peaks in utilization, reaching as high as 80%. To meet the service quality during peak periods, this service reserves a significant buffer during deployment, with average resource utilization just exceeding 30%.
• Cyclic Trigger Type: As shown in the lower right image, this is a typical utilization pattern for online training scenarios, where online training tasks lie between offline training and online inference, representing a periodic batch processing task. For example, a batch of data arrives every 15 minutes, but this batch of data only requires 2-3 minutes for training, leaving the GPU idle for a significant amount of time.

AI application scenarios are complex and variable; the above only lists four typical scenarios. How to balance business performance and resource efficiency in complex scenarios is the first challenge we encounter in GPU virtualization.The second challenge we face in the GPU virtualization process is the lack of a complete GPU isolation and mixing mechanism. Taking the currently mainstream NVIDIA GPU as an example, the typical AI software and hardware ecosystem is divided into several layers — application & framework layer, runtime layer, driver layer, and hardware layer. At the top layer are user applications, which include various common frameworks such as PaddlePaddle, TensorFlow, PyTorch, etc. Below the application layer is the API interface layer encapsulated by hardware providers, including various commonly used operator libraries and hardware runtime access interfaces. Below this API interface layer is the driver layer, which operates in kernel mode and serves as the software interface layer that directly communicates with the device. At the bottom layer is the actual AI acceleration hardware responsible for executing operators. Traditional virtualization solutions combine driver kernel mode and hardware virtualization logic. These two layers are the core IP of hardware providers and are generally closed source. It will be mentioned later that the current native isolation mechanism of GPUs cannot meet the usage requirements in cloud-native scenarios in terms of flexibility and allocation strength. In addition to the isolation mechanism, the existing mixing mechanism also struggles to meet the demands of complex scenarios. We see that many shared scheduling open-source solutions in the industry simply schedule two tasks onto one card from a resource perspective. In actual scenarios, simple sharing can cause mutual impacts between services, leading to long-tail latency and even throughput degradation, making simple sharing unsuitable for production environments. In the previous section on utilization pattern analysis, we saw that different businesses and scenarios exhibit different utilization patterns. How to abstract business scenarios and customize mixing schemes is key to implementing them in production environments.To give everyone a more comprehensive understanding of the development of GPUs and the history of virtualization, we will present a diagram showcasing the history of GPU virtualization development. The application of GPUs in general computing can be traced back to the Tesla architecture of the G80 era, which was the first generation to implement unified shaders, replacing the original separate vertex and pixel pipelines of graphics processors with a general-purpose processor SM. The first GPUs introduced by Baidu can be traced back to the Fermi architecture. From this point, a series of virtualization solutions emerged in the industry, most of which were based on API hijacking. A typical representative is rCUDA, which was initially maintained by academic groups and has maintained a certain frequency of updates and iterations until recently, but it seems to be primarily focused on academic research and has not been widely used in production environments. Baidu’s large-scale introduction of GPUs began with the Kepler architecture, which ushered in the era of Baidu’s self-developed super AI computer X-MAN. X-MAN 1.0 was the first to implement a single machine with a 16-card configuration, allowing for dynamic binding and flexible allocation of CPUs and GPUs at the PCIe hardware level. Limited by the performance of a single card, the focus at that time was more on expansion rather than partitioning. The subsequent Pascal, Volta, and Turing architectures saw rapid performance improvements, and the demand for virtualization became increasingly prominent. We saw that starting from the Kepler architecture, NV officially provided the GRID vGPU virtualization solution, which was initially aimed at graphics rendering and remote desktop scenarios. Around 2019, solutions were also provided for AI and high-performance computing scenarios. However, these solutions were based on virtual machines and were rarely used in AI scenarios. In the Ampere generation, NV launched the MIG instance partitioning scheme, which achieved the partitioning of various hardware resources such as SM, MEM, and L2 Cache at the hardware level, providing good hardware isolation performance. However, this scheme only supports the Ampere architecture and has certain restrictions on card models. Only a few models like A100 and A30 can support it. Moreover, even after partitioning, the performance of a single instance exceeds that of T4 computing power, which does not effectively solve the current efficiency issues in production environments.After gaining some understanding of the GPU architecture and virtualization history, let’s detail the main levels or technical routes for implementing GPU virtualization. To achieve resource virtualization isolation, resources must be separable in time or space. From the user’s perspective, this means that multiple tasks can be executed concurrently or in parallel. Here, we will discuss the parallel or concurrent space at multiple levels: user mode, kernel mode, and hardware. Since NV’s software and hardware ecosystem is closed source, the diagram here is drawn based on our comprehensive architecture white paper, reverse engineering papers, and our own understanding. We hope to be corrected if there are inaccuracies.User Mode SolutionLooking at this diagram from top to bottom, multiple processes on the GPU are inherently concurrent, meaning they are time-sliced. The driver and hardware are responsible for switching tasks in a time-sliced manner. Utilizing this mechanism, we can achieve virtualization effects by limiting computing resources and video memory resources at the API level. The APIs can be divided into two layers: one is the driver API, which is closely tied to the driver and is the only way for all upper-level calls to access the GPU. Controlling this layer of API essentially means controlling user resource access. It is worth mentioning that NV’s MPS technology can achieve space multiplexing, which also provides possibilities for further optimization of business performance, which we will elaborate on in the subsequent practical implementation section.

Kernel Mode Solution

The next level down is kernel mode. Whether it is full virtualization or semi-virtualization at the virtual machine level, or the container solutions of major cloud vendors in recent years, they all implement system call interception and MMIO hijacking at the kernel level. The biggest challenge in kernel mode is that many registers and MMIO behaviors do not have good documentation, requiring complex reverse engineering.

Hardware Solution

Below kernel mode is the hardware layer, where true parallelism is guaranteed. Both NV’s MIG technology and Baidu Kunlun’s SR-IOV technology have achieved computational resource partitioning at the hardware logic level, enabling true parallelism and space multiplexing. For instance, Kunlun can achieve hardware partitioning of 1/3 and 1/2, while A100 can achieve resource partitioning down to a minimum granularity of 1/7.We have spent considerable effort introducing the challenges and current status of GPU virtualization. Next, let’s look at how Baidu internally responds to these challenges.This diagram illustrates the Baidu Intelligent Cloud — Dual Engine GPU Container Virtualization Architecture.Here, we emphasize containers because we believe that future AI end-to-end applications will gradually converge onto cloud-native platforms, achieving fully containerized development, training, and inference. According to Gartner’s research, by 2023, 70% of AI tasks will be containerized. Baidu began its containerization efforts in 2011, and we now have over 10 years of deployment and optimization experience. We are also committed to contributing the product capabilities and optimization experience honed through this process to the community and a wide range of users. We also emphasize the dual engine. In the overall architecture, we adopt two isolation engines in user mode and kernel mode to meet users’ varying needs for isolation, performance, and efficiency.Above the isolation engine is the resource pooling layer, which is based on our deep understanding of the software and hardware system, gradually decoupling, extending, and pooling AI acceleration resources, creating a pooling abstraction layer for future infrastructure.Above the resource pooling layer is the unified resource scheduling layer of Matrix/k8s (here Matrix is Baidu’s internal container scheduling system). Based on the scheduling mechanism, we abstract various mixing strategies according to different business scenarios, including shared mixing, preemptive mixing, time-sharing mixing, and tidal mixing. These mixing strategies will be elaborated on in the subsequent practical implementation section.Relying on resource isolation and resource scheduling is the end-to-end scenario of AI business, including model development, model training, and online inference.Next, we will share the implementation of the user mode and kernel mode isolation engines.The following diagram is a core architectural diagram of the user mode isolation engine. At the top of the architecture diagram are user applications, which include various commonly used frameworks such as PaddlePaddle, TensorFlow, PyTorch, etc.Below the user applications are a series of API Hook interfaces, based on which we can achieve local use of GPU resources and remote mounting. By replacing the underlying dynamic libraries that the frameworks depend on, we can achieve resource control and isolation. It is important to note that this solution is completely transparent to applications; necessary library replacement operations have been automatically completed by the container engine and scheduling parts. After the CUDA API is hooked, it will reach the executor through two channels. Here, the vast majority of APIs, such as device management APIs, pass through to the executor without any operations. A few APIs related to resource requests will be intercepted through one layer, implementing a series of user mode virtualization functions. The logic implementation of this layer is efficient enough that its impact on performance is almost negligible.Currently, the user mode isolation engine can provide rich isolation and control features, including basic video memory isolation and computational power isolation. We have also extended many advanced features: encoder isolation, high-priority preemption, video memory over-subscription, video memory pooling, etc.The advantage of the user mode solution is its good performance and low long-tail latency, making it suitable for business scenarios that prioritize mechanism performance and extreme efficiency, such as latency-sensitive online inference tasks.On the basis of isolation, we provide remote functionality. The introduction of remote capabilities will greatly enhance the flexibility and efficiency of resource allocation, which we will elaborate on at the end of this article.This sharing is a technical sharing, and we will briefly outline the key points and challenges of remote technology, hoping to stimulate everyone’s business thinking and technical discussions. Based on the software and hardware technology stack discussed in the previous section on virtualization challenges, GPU remote access can roughly be implemented at the hardware link layer, driver layer, runtime layer, and user layer. However, through in-depth technical analysis and understanding of business scenarios, we believe that the runtime layer is currently the most suitable.How to implement the runtime layer technical route? What are the key points of the technology? We believe the main issue is semantic consistency. Based on runtime remote access, it is necessary to split the original local process into client and server processes. The CUDA runtime is closed source, and its internal implementation logic cannot be explored. How to ensure that the original program logic and API semantics remain intact after splitting the processes is achieved by using a one-to-one thread model to ensure alignment of internal logic and semantics of the API.The difficulty of remote implementation lies in the numerous APIs involved. In addition to the dynamic library libcudart.so, the runtime involves a series of dynamic libraries and APIs such as cuDNN, cuBLAS, cuFFT, etc., encompassing thousands of different API interfaces. We have implemented automatic header file parsing and code generation through compilation technology, and completed the parsing of hidden APIs through reverse engineering.After solving the 0-1 adaptation of the remote solution, subsequent backward compatibility is relatively easy to resolve. Currently, the CUDA API appears to be relatively stable, and only minor incremental adaptations are needed for new versions.We have mentioned space multiplexing and time-slicing multiple times. Here is a detailed explanation:

• Time-slicing: As the name implies, it is the reuse of time slices. This is similar to CPU process scheduling; within a single time slice, only one GPU process is running. Multiple GPU processes operate in an alternating manner at a microscopic level, which can only be considered concurrent.

• Space multiplexing: Unlike time-slicing, during space multiplexing, multiple processes can run simultaneously on one GPU at a microscopic moment, as long as the GPU’s resources are not fully utilized. The kernels of other processes can be launched, and the kernels of the two processes run intertwined at a microscopic level, truly achieving parallelism and further utilizing GPU resources.

As mentioned in the overview section, current common virtualization methods, including kernel mode virtualization and NVIDIA vGPU virtualization, are fundamentally based on time-slicing reuse schemes.NV has launched MPS (Multi-Process Service) for multi-process concurrent scenarios, which can achieve space multiplexing and is currently the only solution that balances efficiency and performance.Here’s a brief introduction to MPS: MPS essentially merges the contexts of two processes into one process, and the merged process interleaves the kernels of the previous two processes for launching. This approach has two benefits:

• Processes do not require context switching, reducing the overhead associated with context switching.
• At the same time, the kernels of different processes are interwoven, improving resource space utilization.

When discussing MPS, we must mention a commonly criticized drawback — the fault isolation issue.

How do we solve the stability issues of MPS? Baidu Intelligent Cloud has proposed a comprehensive process merging and sharing solution that combines scheduling, container engines, and business persistence.

• Achieve graceful exit of business processes through kill command redirection.

• Implement health checks and deadlock detection through the MPS status detection mechanism.

• Achieve automatic restart of user processes through service persistence.

This solution has covered over 90% of commercial (latency-sensitive critical business) resources and has been running for over two years, providing extreme performance while meeting the stability needs of the vast majority of users.As the acceptance of MPS increases, NV continues to enhance its stability. Here’s some good news: NV will significantly enhance MPS stability in the second half of this year, including features like deadlock state detection and graceful process exit, which will become part of the MPS product, further improving its stability and usability.Before introducing the high-priority preemption feature, let me share the business scenarios for high-priority preemption. Based on discussions with various users both inside and outside the company, most AI applications in production environments can be categorized into online, near-line, and offline tasks based on their latency sensitivity.

• Online tasks have the highest latency sensitivity and generally involve real-time responses to user requests.
• Near-line tasks are typically batch processing tasks that do not have strict latency requirements for individual logs but have completion time requirements ranging from hours to minutes for a batch of data.
• Offline tasks have no latency requirements and focus solely on throughput, typically involving model training tasks.

If we define latency-sensitive tasks as high-priority tasks and latency-insensitive near-line and offline tasks as low-priority tasks, then when mixing these two types of tasks, we define different kernel launch priorities based on the priority of the tasks, which is what we mentioned above as the high-priority preemption feature.The implementation principle is illustrated in the following diagram, where the user mode isolation engine maintains a logical kernel queue for both high-priority and low-priority tasks. When overall load is low, both queues are allowed to launch kernels simultaneously, interleaving the kernels of both queues. Once the load increases, the hierarchical launch module will promptly pending the launch of the low-priority queue, ensuring the execution latency of high-priority tasks.The advantage of this feature is that it guarantees offline throughput while minimizing or even avoiding the impact on online tasks.Similarly, let’s first introduce the definition and scenarios of time-sharing mixing.Time-sharing mixing is somewhat similar to time-sliced shared mixing in its mixing pattern. The difference is that time-sharing mixing proposes a video memory swap scheme for video memory, allowing scenarios where video memory is long-term occupied but computational power is intermittently used or occasionally triggered. When a process requires computational power, it obtains access to video memory, and once the process completes its computation, it releases access to video memory, allowing other waiting processes to gain running opportunities, thus fully utilizing intermittently idle GPU resources.The core technology of time-sharing mixing is video memory swapping. We can draw an analogy to CPU memory swapping; when a process runs out of memory, the system swaps out a portion of the system memory to disk based on certain strategies, freeing up space for running processes.The implementation principle of video memory swapping is illustrated in the following diagram. We maintain a video memory pool at the physical address of the video memory, with the upper layer determining which process has access to the GPU through resource locks. When a process obtains the lock, the video memory is moved from memory or disk to the physical video memory pool, further mapped to the virtual address space for process use. When the process releases the lock, the virtual video memory space of the process is retained, and the physical video memory is moved back to memory or disk. This lock is mutually exclusive, allowing only one process to obtain the lock, while other processes are pending in a waiting queue, obtaining the resource lock in a FIFO manner.The above describes the functional implementation of the user mode isolation engine. In practical applications, how does performance affect users? Here, we will directly present test data.The following diagram shows a comparison of data for the typical model ResNet-50 Server scenario on the public test set MLPerf. The bars in the diagram represent performance under exclusive, bare mixing, user mode isolation, and kernel mode isolation from left to right.The left graph shows average throughput comparisons; in inference scenarios, requests are triggered intermittently, and we can see that under any scheme, throughput can directly reach the pressure value. Here, we want to emphasize that in inference scenarios, throughput does not effectively showcase virtualization performance; more attention should be paid to latency in production environments.The right graph shows the comparison of P99 percentile latency. It can be seen that under low pressure (QPS = 40), the impact of user mode and bare mixing on long-tail latency is basically consistent, while kernel mode, due to its time-slicing reuse, has a slightly greater impact on long-tail latency. As we continue to increase pressure to QPS = 60, the advantages of user mode become apparent, as space multiplexing greatly reduces the impact on long-tail latency. With further increasing pressure, the user mode process merging solution even shows an order of magnitude improvement over other mixing methods.Although long-tail latency control is not as good as user mode, kernel mode has advantages in isolation, making it more suitable for scenarios with strong isolation demands.Next, let’s understand the technical implementation of the kernel mode isolation engine. First, let’s look at the characteristics of kernel mode virtualization implementation, which include: Kernel mode implementation; good isolation: supports video memory, computational power, and fault isolation; MB-level video memory isolation; 1% level computational power allocation; supports mainstream GPUs such as P4, V100, T4, A100/A10/A30; supports GPU driver versions from 410 to 510; user mode operating environment requires no changes; supports containerized deployment. Unlike the user mode implementation, the isolation functions of kernel mode virtualization are all implemented in kernel mode. The left half of the diagram below shows an architecture diagram of our kernel mode virtualization implementation, with GPU hardware at the bottom, kernel layer in the middle, and user layer at the top. The hardware layer refers to our GPU, which can be a bare GPU or a pass-through GPU. At the kernel layer, the original GPU driver controls the GPU’s functions; the actual operations on the GPU are all managed by this driver. Above the GPU driver is our implemented kernel module for GPU virtualization, which is the GPU interception driver, depicted in yellow. It comprises three functional parts, including video memory interception, computational power interception, and computational power scheduling, which implement video memory isolation and computational power isolation. In the user layer, the first is the interception interface provided by the interception module, divided into two parts: one part is the device file interface, and the other is the interface for configuring the interception module. The device file is provided to the container; let’s first look at the container. Above the container are applications, and below it is the CUDA runtime, which includes the CUDA underlying libraries, such as driver API/nvml API, etc. By providing our device file to the container as a fake device file, the upper-layer CUDA accesses our device file, thus completing the interception of the CUDA underlying library’s access to the GPU driver. In our kernel interception module, we intercept all system calls related to access, intercept and parse them, and redirect the actual access to the real GPU underlying driver. After the GPU underlying driver processes it, the result is returned to our interception module, which processes it again and finally returns the result to the underlying library in the container.In simple terms, we intercept the access of the underlying library to the GPU driver by simulating the device file, completing the interception of video memory and computational power through interception, parsing, and injection.Currently, video memory isolation is achieved by intercepting all system calls related to video memory, including video memory information, allocation, and release. Moreover, current video memory isolation can only be statically set and cannot be dynamically changed. Compared to user mode, kernel mode cannot support video memory over-subscription. In terms of computational power isolation, we obtain relevant information by intercepting the CUDA Context of the process. The scheduling object is the CUDA Context related to the process. Each GPU has a kernel thread that schedules all CUDA Contexts on that GPU.We have implemented four kernel mode computational power scheduling algorithms:

• Fixed Share: Each POD is allocated fixed computational power resources, meaning the total GPU computational power is divided into n parts, with each POD receiving 1/n of the computational power.
• Equal Share: All active PODs share computational power resources equally, meaning if the number of active PODs is n, each POD receives 1/n of the computational power.
• Weight Share: Each POD is allocated computational power resources based on weight, meaning the total GPU computational power is distributed to each POD according to their weight values. Regardless of whether a POD has business load, resources are allocated according to weight.
• Burst Weight Share: Active PODs are allocated computational power resources based on weight, meaning each POD is given a weight value, and active PODs share computational power according to the ratio of their weights.

Because kernel mode uses time-slicing for computational power scheduling, it is not very friendly for latency-sensitive businesses. We have specifically developed offline mixing technology to mix online and offline businesses, greatly improving the response speed of online businesses while allowing offline businesses to share GPU computational power resources, achieving the goal of improving GPU resource utilization. The characteristics of our offline mixing are:

• Online POD: Inference tasks that usually occupy a small amount of computational power.
• Offline POD: Training tasks that usually occupy most of the computational power.

When an online POD has task loads, it immediately preempts the offline POD, occupying all computational power to provide inference services. When the task load ends, it releases the computational power back to the offline POD.The following are the evaluation results of kernel mode computational power isolation: The test environment is a single card V100 SXM2 16G, testing throughput under training scenarios using the horovod framework with the model being ResNet50. The weight ratio between POD 1 and POD 2 is 1:2. The results shown in the diagram indicate that the throughput ratio between POD 1 and POD 2 is around 45-50%, roughly 1/2, which aligns with our preset values. Meanwhile, POD SUM shows a 2-4% loss compared to Native, as computational power isolation requires switching between CUDA Contexts, which inevitably incurs some loss, but our loss is within 5%, which can be considered tolerable.Let’s compare the characteristics of kernel mode and user mode. In terms of fault isolation, kernel mode has advantages over user mode, and kernel mode does not require replacing the underlying library. User mode computational power scheduling adopts a combination of time-slicing and space multiplexing, while kernel mode employs time-slicing reuse. Advanced features of user mode include offline mixing, video memory over-subscription to memory, and codec instances (independent allocation of encoding and decoding resources of AI acceleration cards), while kernel mode also supports offline mixing.How to utilize virtualization technology to enhance GPU utilization efficiency in AI scenarios? Below, we will share best practices in large-scale AI scenarios based on internal case studies. We first look at a typical scenario in an inference service. Due to the architecture of the model itself or the service’s latency requirements, some tasks can only run under a very small batch size or even a batch size of 1, leading to long-term low GPU utilization, with peak utilization as low as 10%. In this scenario, the first thought should be to mix multiple low-utilization tasks. We summarize this mixing strategy as shared mixing. Whether in development, training, or inference scenarios, we can adopt shared mixing among multiple low-utilization tasks.Combined with the previously mentioned process merging technology, we can achieve shared mixing of two or even multiple instances while ensuring service latency, increasing resource utilization by over 2 times. At the same time, most GPUs have independent encoding and decoding resources. In most scenarios, as shown in the lower left image, these resources remain idle for long periods. We can mix an encoding or decoding instance on top of shared computing resources to further enhance resource efficiency and activate idle resources.A typical load pattern for inference services shows significant peak and valley fluctuations throughout the day, with unpredictable short-term traffic surges. This results in high peaks but poor average utilization, often below 30% or even 20%. How to optimize efficiency in these clearly fluctuating and short-term surging services? We propose a preemptive mixing strategy. Preemptive mixing involves mixing a low-priority task that is latency-insensitive with a high-priority task that has high peak demands and latency sensitivity. Here, high-priority and low-priority are defined by the users themselves and explicitly stated when requesting resources. In our internal practice at Baidu, near-line and offline database brushing or training tasks are defined as low-priority, as these tasks have certain throughput requirements but no latency requirements. Utilizing the high-priority preemption mechanism in virtualization functions, high-priority tasks maintain control over resource occupancy. When traffic is in the valley, the overall load of the card is low, allowing low-priority tasks to run normally. However, when traffic peaks or short-term surges occur, the high-priority preemption mechanism can sense this in real-time and preempt computational power at the kernel level, causing low-priority tasks to be throttled or even completely pending, ensuring the service quality of high-priority tasks.This mixing mode may encounter insufficient video memory; however, computational power may still have significant redundancy. For such scenarios, we provide an implicit video memory over-subscription mechanism. Users can enable video memory over-subscription for low-priority tasks through environment variables, mixing more instances to ensure that computational power fills utilization valleys, maximizing overall utilization efficiency.The third type of business scenario may be familiar to everyone, which is the scenario of persistent video memory and intermittent computational power triggering. A typical representative is the development task and online training. Taking online training as an example, we know that many models need to be updated online based on user data received daily or even hourly, such as recommendation models, which require online training. Unlike offline training that fills throughput continuously, online training requires accumulating a batch of data before triggering a training session. Internally at Baidu, a typical pattern might be that a batch of data arrives every 15 minutes, but the actual training time is only 2 to 3 minutes, leaving the training process idle in video memory until the next batch of data arrives from upstream. During this time, utilization remains at 0, causing significant resource waste. Because this type of task occupies video memory almost entirely, we cannot use the previously mentioned shared mixing or preemptive mixing. Combined with the previously mentioned video memory swap mechanism, we propose a time-sharing mixing strategy. Time-sharing mixing is similar to time-sliced shared mixing, but in this case, video memory is swapped in and out along with the computation context. Since the underlying virtualization layer cannot perceive when the business requires computation, we maintain a global resource lock for each GPU card. We also encapsulate corresponding C++ and Python interfaces for user calls. Users only need to request this lock when computation is needed, and the video memory will automatically swap in from other spaces to the video memory space; after computation is completed, the lock is released, and the corresponding video memory will be swapped out to memory or disk space. With this simple interface, users can achieve time-sharing exclusive access to the GPU for multiple tasks. In online training scenarios, using time-sharing mixing can achieve up to 4/5 resource savings while boosting overall utilization.The best practices for the three scenarios mentioned above have been validated and scaled in Baidu’s internal business. The relevant functionalities have also been launched on Baidu’s Baijie AI heterogeneous computing platform, and everyone can apply and try them immediately. Finally, I will take about three more minutes to discuss some features that are still under internal verification, which will soon be completed and launched on the Baidu Baijie platform, further addressing common issues in large-scale AI scenarios such as uneven ratios, supply-demand imbalances, and resource fragmentation. Those working in infrastructure will often hear concepts like resource decoupling and pooling. How to implement the concept of pooling and transform it into actual productivity has been an ongoing exploration and advancement for us. As early as 2015, we achieved the industry’s first hardware pooling solution based on PCIe Fabric, which has been scaled internally at Baidu. This is the previously mentioned X-MAN 1.0 (now evolved to 4.0). By configuring the interconnection between CPUs and GPUs through the PCIe Fabric network, we achieve dynamic allocation of resources, addressing various scenarios’ ratio issues. However, due to hardware connection and protocol limitations, this solution can only solve pooling within a cabinet. Software layer pooling is considered a more flexible technical solution. As data center networks continue to upgrade, 100G or even 200G networks will likely become standard infrastructure, providing a high-speed communication highway for resource pooling. The decoupling and pooling of resources allow businesses greater flexibility and provide more room for efficiency optimization. For example, issues related to the ratio of CPU and GPU, long-term resource occupation and supply-demand imbalance in development scenarios, resource fragmentation blocking tasks in training scenarios, and issues like device exceptions and training restarts can all be resolved through pooling and its derived solutions. Finally, all the virtualization technologies and best practices shared above have been launched on Baidu’s Baijie AI heterogeneous computing platform. Search “Baidu Baijie” on the Baidu Intelligent Cloud website to accelerate AI tasks and inspire business imagination!Selected Q & A Q: Generally, resources are containerized through namespaces and cgroups. What technology does GPU use to achieve resource control?

A: Both namespaces and cgroups are mechanisms provided by the kernel, which fundamentally depend on the capabilities provided by hardware. Currently, this is not available on GPUs, which remain closed source. Only hardware providers can supply features that can be upstreamed to the kernel mainline. Current third-party solutions are non-standard implementations done in user mode or kernel mode, and there is no way to include them under the namespace and cgroup categories for now. However, we can consider that GPU virtualization aims to implement the mechanisms corresponding to these interfaces, and whether they can be standardized is another larger issue.

Q: Besides GPGPU virtualization technology, are we also developing NPU-related virtualization technology? Is it decoupled from NV technology stack? Thank you!

A: I understand that the NPU here refers to Network Processing Unit, which broadly encompasses all AI acceleration hardware. We are working on virtualization adaptations for other AI acceleration hardware, starting with Kunlun chips, where we have already adapted the virtualization capabilities mentioned above. As scenarios expand, we will continue to adapt other mainstream acceleration hardware.

Q: Are user mode and kernel mode two different products?

A: They are the same product, with different underlying implementation methods, but the user interface layer is unified.

Q: What granularity can user mode virtualization achieve?

A: Computational power can achieve 1% granularity, and video memory can achieve 1MB granularity.

Q: Will kernel mode virtualization incur significant control overhead?

A: Kernel mode virtualization is based on time slicing; the overhead here comes from time slicing. Precise isolation inevitably leads to computational power loss. If referring to application performance overhead, kernel mode indeed incurs more overhead than user mode.

Q: Based on time-slicing implementations, online inference seems to respond faster with free competition.

A: According to our test results, performance decreases in the following order: process merging, bare mixing (free competition), strict isolation.

Q: Can these two virtualization methods coexist in a k8s cluster?

A: From a mechanism and principle perspective, they can coexist. However, from a product dimension, we do not want to design it this complex, so they are still separated. If there is widespread demand for coexistence in the future, we will consider launching a similar coexistence solution.

Q: Can you elaborate on how to extend the k8s scheduler? Does it require the agent on the node to report GPU topology and total amount?

A: Yes, it requires a single-node agent to upload resource (including video memory resources and computational power resources) and topology information.

Q: Do you have any suggestions on the selection between time-slicing and space multiplexing?

A: For latency-sensitive online inference tasks, it is recommended to choose a space-based process merging scheme. For scenarios requiring strict isolation, it is recommended to choose a time-slicing scheme. In other scenarios, there is no difference between the two options.

Q: How far does kernel mode support CUDA versions? If NV updates, how long does Baidu Intelligent Cloud’s update cycle take?

A: Kernel mode does not have special requirements for CUDA versions because it is virtualization done in the kernel. It currently supports all CUDA versions. If NV updates CUDA, it is expected that no special support work will be needed.

Q: Does using kernel mode require a special OS image provided by Baidu Intelligent Cloud? A dedicated driver?

A: Kernel mode does not require a special OS image from Baidu Intelligent Cloud. Currently, we support both centos7 and ubuntu. However, our own deployment framework is required for use. There are no special requirements for container images; all can be transparently supported.

Q: Is it only available in public clouds? Can it be deployed privately?

A: It can be deployed and used in both public and private clouds.

Today’s Recommended Articles

Microsoft begins banning commercial open source: starting with the App Store, effective July 16?!

The migration is underway; is it time to say goodbye to GitHub?

Tencent Security responds to the withdrawal of data product lines; Musk claims to terminate the Twitter acquisition; Pinduoduo’s “cutting a knife” fraud case verdict announced | Q News

Why GitLab’s technology selection is so different: insisting on using outdated web frameworks for over a decade, resolutely avoiding microservices.