In the wave of rapid development of generative artificial intelligence, enterprise applications are accelerating from model research to business implementation. Whether it is large-scale data processing, training and inference of ultra-large parameter models, or deploying AI Agents that can automatically complete tasks, these scenarios require stable, efficient, and elastic resource scheduling and management capabilities.
Containers, with their advantages of environmental consistency, cross-platform deployment, and efficient scheduling, naturally meet the demands of AI scenarios for diverse computing power, rapid iteration, and large-scale distribution, becoming the de facto native foundation in the AI era. However, to meet the demands at production scale, product and technology forms must evolve accordingly.
Based on this background, this article will explore the core requirements of containers in AI scenarios across four key stages: large-scale data processing, model training, model inference, and AI Agent deployment, as well as how containers can achieve technological evolution and upgrades at each stage to support the efficient operation of AI workloads and their large-scale implementation in real business scenarios.
01｜Container Evolution in the AI Era
Reshaping Intelligent Computing Infrastructure
If we view AI systems as products or projects with a lifecycle, the process can be divided into four stages: large-scale data processing, model training, model inference, and AI Agent application deployment. Based on Alibaba Cloud’s technical practice experience in the open-source community and various industry case studies, these four stages share a series of underlying technical capabilities while also presenting different technical requirements due to their respective contexts.
Common Requirements of AI Systems for Containers
Computing power scheduling and resource utilization optimization: Achieving key task performance guarantees and global computing power utilization improvements in cross-regional, multi-tenant, and heterogeneous hardware environments.
Data access and throughput: Ensuring continuous and efficient operation of high-value computing power such as GPUs during training and inference by reducing data access latency.
Stability and recoverability: Building multi-dimensional observability capabilities, rapid fault detection, self-healing, and seamless recovery mechanisms for long-duration tasks, minimizing the impact of abnormal events on overall business operations.
Security and isolation: Providing runtime environment isolation and risk protection in scenarios involving code generation in Agent sandboxes, multi-toolchain calls, and high-concurrency access, preventing potential high-risk operations and resource misuse.
Specific Technical Requirements for Containers at Each Stage of AI Systems
Next are the specific technical requirements arising from the characteristics of each of the four stages.
Data Processing Stage: The core task is the scheduling and elastic optimization of large-scale data processing tasks, orderly orchestration of complex workflows, and efficient collaborative execution of heterogeneous computing resources (CPU/GPU).
Model Training Stage: The key requirement is to maximize GPU computing power, reasonably allocate computing resources for training tasks, achieve low-latency distributed communication, and provide high-speed loading channels for massive training data.
Model Inference Stage: The performance goal focuses on reducing startup time and optimizing long-tail response performance, improving inference service response speed through timely and reasonable elastic strategies while ensuring high availability of services.
AI Agent Stage: The infrastructure design must meet the security protection for tool calls and code execution, support large-scale elastic scaling, and provide task state persistence capabilities.
As a lightweight virtualization technology, containers encapsulate applications and their dependent environments, deploying and managing applications in a unified isolated runtime space. Compared to traditional virtual machines, containers offer faster startup speeds, higher resource utilization, and cross-environment consistency, allowing applications to maintain stable operation throughout the entire lifecycle of development, testing, deployment, and operation.
In the AI field, containers naturally meet the demands of AI scenarios for diverse computing power, rapid iteration, and large-scale distribution, becoming the de facto native foundation in the AI era. Gartner predicts that by 2028, 95% of new AI deployments worldwide will run in container environments. However, to meet the high performance and stability requirements of AI at production scale, container technology and product forms must evolve accordingly, with deep tuning and capability expansion for AI scenarios.
Based on the above analysis, the Alibaba Cloud container service team has comprehensively upgraded its products and leading open-source projects, providing a new paradigm for enterprises to build reliable and efficient AI-native infrastructure.
02｜Data Processing
From Workflow Orchestration to High-Performance Computing
In AI systems, data preparation is the starting point of the value chain, directly determining the quality and efficiency of model training and inference. This stage mainly faces two challenges: complex workflows and large-scale batch processing pressure.
-
Complex workflows: Involves multiple stages of tasks such as cleaning, labeling, sharding, augmentation, and feature extraction, with dynamic dependencies requiring high precision in version management and consistency assurance.
-
Large-scale batch processing pressure: Facing data volumes of TB to PB, the system must schedule massive computing resources under a mixed CPU and GPU computing architecture to efficiently complete these offline processing tasks within a limited time; even under long-term high-load operation, it must ensure system stability, high throughput, and resource utilization.
To address these challenges, the Alibaba Cloud technical team conducted in-depth analysis of existing open-source solutions, deeply optimized the community version of Argo Workflows, and innovatively proposed the Ray on ACK hybrid architecture.
Fully Managed Argo Workflows
Addresses the limitations of the community version in scheduling bottlenecks, resource utilization, task queuing, version conflicts, and control plane stability, supporting declarative task orchestration, queue management, and dependency control.
Ray on ACK Hybrid Architecture
Uses Ray as a high-performance distributed computing kernel to execute data processing tasks; the ACK scheduler uniformly manages mixed resources of CPU, GPU, and different Compute Classes, achieving fine-grained task scheduling, elastic scaling, and priority assurance through task queues, batch scheduling, and elastic scaling mechanisms.
Based on the above optimizations and architecture, Alibaba Cloud’s container technology has formed a complete closed loop in orchestration and execution capabilities, demonstrating significant advantages in ultra-large scale, heterogeneous computing power, high elasticity, and multi-tenant security:
-
A single cluster stably supports tens of thousands of nodes and hundreds of thousands of CPU/GPU mixed computing power, meeting the execution needs of tens of thousands of concurrent workflows and hundreds of thousands of tasks;
-
Supports efficient elastic scaling of CPU and GPU computing power through Alibaba Cloud Container Service (ACS) Serverless containers, reusing Alibaba Cloud Container Service Kubernetes version (ACK) cloud monthly nodes and hybrid cloud node pools to improve overall resource utilization;
-
Combining Gang scheduling, Capacity scheduling, Resource Policy elastic scheduling, and priority preemption ensures resource guarantees for critical tasks and fairness for multi-tenants;
-
Ray History Server, in conjunction with Alibaba Cloud ARMS, provides full-link logging and fault localization capabilities, significantly improving stability and operational efficiency.
In fact, the technological upgrades of Alibaba Cloud’s container technology for data processing have already brought significant results in practical production applications: data processing throughput has increased by about 10 times, GPU resource utilization has significantly improved, while operational costs have been reduced by about 30%, providing a solid data foundation for subsequent model training and inference.
03｜Model Training
Maximizing Computing Power Scheduling and Data Acceleration
Model training, as the core link of value creation, has evolved from the previous “computing power stacking” to a comprehensive test of underlying infrastructure scheduling and data link optimization, especially in scenarios with models exceeding billions of parameters, which typically face the following three challenges:
-
Distributed Communication Bottlenecks: The performance of distributed training highly depends on the communication bandwidth between GPUs; unreasonable job scheduling may lead to high-latency communication across switches or availability zones, significantly extending training cycles.
-
Data Loading Latency: Massive training data (TB to PB level) read directly from remote storage incurs high latency and repeated access overhead, causing GPUs to wait idly for data for extended periods.
-
Low GPU Resource Utilization: Debugging, small-scale experiments, or specific inference phases often do not require exclusive use of an entire GPU card, leading to inefficient use of expensive GPUs.
To address these challenges, Alibaba Cloud’s container technology system has made targeted optimizations at the scheduling layer, data layer, and resource management layer.
Scheduling Layer
ACK/ACS introduces a multi-level topology-aware scheduling mechanism, including node-level CPU Numa, PCIe, and Scale-Up network NVLink topology awareness, as well as inter-node Scale-Out RDMA network topology awareness, prioritizing the scheduling of Pods requiring high-bandwidth communication to the same switch or within the same node and arranging them in a reasonable order to optimize task set communication efficiency.
Data Layer
Utilizes the CNCF open-source project Fluid to build a distributed cache, caching remote datasets on-demand to computing nodes, supporting data pre-warming and multi-level caching, providing training tasks with reading performance close to local disks while meeting the demand for dynamically scaling data access bandwidth.
Resource Layer
Achieves GPU container sharing and isolation based on memory and computing power dimensions, splitting a single physical GPU into multiple logical instances for efficient multi-task utilization; simultaneously providing multi-tenant quotas and fair scheduling mechanisms to guarantee critical task resources.
Through optimizations in topology-aware scheduling, localized data acceleration, and fine-grained GPU sharing, the resource and task matching during the training phase has become more efficient:
-
Topology-aware scheduling has achieved sub-second communication latency minimization calculations, significantly improving the synchronization efficiency of distributed training, with typical Allreduce operator performance improved by 30%;
-
Fluid distributed caching has reduced remote data loading latency by over 90%, resulting in higher GPU MFU;
-
GPU containers split physical cards into multiple logical instances, increasing resource utilization by 3 to 5 times, while combining multi-tenant quotas and fair scheduling mechanisms to ensure critical tasks run with priority.
Production environment data shows that the overall time cost of large model training has significantly decreased, with the loading speed of a 600GB training dataset improved to over 10 times the original, and overall computing power costs reduced by more than 40%, effectively achieving a leap from “operable” to “optimal operation” performance.
04｜Model Inference
Fast Execution with Stability
The capabilities of large models have matured through long-term training, but from a technical implementation perspective, the maturity of model capabilities does not mean that applications can run seamlessly in enterprise production environments. Therefore, “how to efficiently implement inference services” has become a core issue.
First, to deeply understand the internal structure of the inference process, a complete inference request can be divided into two stages: Prefill and Decode:
-
Prefill stage calculates the KV values of prompts, characterized by being compute-intensive;
-
Decode stage requires storing all previously computed KV values, characterized by being memory-intensive.
When both stages run on the same GPU, due to their mutual exclusion in resource usage, one stage must wait for the other to release GPU compute/memory resources before it can continue, thus failing to utilize the GPU in parallel, leading to increased overall response time and decreased performance.
In addition, when deploying inference services at scale, the following common challenges are encountered:
-
Deployment architecture diversity: Various Prefill-Decode separation solutions provided by the open-source community (vLLM, SGLang, Mooncake, Dynamo, etc.) differ in component composition, deployment methods, and configuration requirements, leading to high complexity in user selection and operation.
-
Startup and loading latency: Loading ultra-large model images and model weight files can take tens of minutes, causing severe cold start phenomena in inference services, affecting business response speed.
-
High concurrency and long-tail response stability: During peak access or complex context scenarios, response latency fluctuates significantly, affecting model service SLA.
-
GPU stability risks: Long-term operation can easily lead to GPU hangs, driver crashes, memory anomalies, etc., causing service interruptions.
To solve these problems, Alibaba Cloud has launched the ACK AI Serving Stack suite, which integrates inference framework management, intelligent traffic routing, and distributed caching acceleration capabilities based on ACK as a unified foundation, addressing the challenges in managing the entire lifecycle of LLM inference services.
In the technical system of the ACK AI Serving Stack, three key components work together to provide core capabilities.
RoleBasedGroup (RBG)
Supports one-click deployment of various open-source PD separation frameworks, flexibly and dynamically configuring the Prefill/Decode ratio to meet different SLAs, and achieving unified runtime abstraction (InferenceRuntime) for multiple inference engines.
ACK Gateway with Inference Extension (GIE)
Request scheduling and load balancing capabilities deeply optimized for AI inference, significantly improving KV cache hit rates through precise prefix cache-aware scheduling, greatly enhancing the inference throughput of LLM services and reducing end-to-end latency, significantly optimizing inference costs.
Fluid
Builds a distributed cache, pre-warming remote model files to local nodes, achieving zero redundancy and high-speed startup.
In practical applications, this technical system has achieved multiple breakthroughs in deployment uniformity, response performance, and stability assurance:
-
One-click deployment of multiple inference frameworks and full lifecycle management of inference services, reducing operational complexity;
-
Dynamic adjustment of Prefill/Decode container ratios to ensure that key metrics such as response time and throughput meet SLA requirements;
-
Prefix-aware traffic routing has improved first packet latency in long-tail scenarios by about 73%, with overall response speed improved by about 40%;
-
Based on Fluid’s distributed caching and model pre-warming capabilities, the loading time of models over 600GB has been reduced from 40 minutes to less than 4 minutes (latency reduced by about 90%);
-
GPU fault detection and self-healing mechanisms have significantly reduced MTTR (Mean Time to Repair);
-
Online AI Profiling has improved the efficiency of locating GPU application performance bottlenecks by about 50%.
These breakthroughs enable model inference services to achieve efficient startup, stable operation, and continuous optimization in production environments, providing end users with a faster and more stable AI application delivery experience.
05｜AI Agent Deployment
Security, Elasticity, and State Persistence
As the training and inference capabilities of large models continue to improve, the application form of AI is evolving from “passive response” to “active execution.” More and more enterprises are beginning to build AI Agents capable of autonomous planning, tool invocation, and multi-step task completion for scenarios such as customer service, data analysis, and automated operations.
Unlike the batch processing-focused training process and some inference scenarios, AI Agents possess real-time interaction, multi-tool invocation, and multi-step task execution characteristics. While these characteristics inherit the performance and architectural baseline of inference systems, they also impose higher requirements on the underlying infrastructure—especially in terms of business security isolation capabilities, concurrent elastic scaling capabilities, and state persistence capabilities for long-duration tasks, all of which require systematic optimization:
-
A more secure isolated environment: Preventing attackers from inducing Agents to perform malicious actions through prompt engineering, such as accessing sensitive data, initiating network attacks, executing malicious code, or performing unauthorized operations. Or preventing models from hallucinating and executing erroneous instructions that delete root directories.
-
Greater scalability: AI applications driven by large models may have scenarios where one Agent loops to invoke tools, or even multiple Agents collaborate to invoke tools, amplifying the elasticity requirements.
-
Long-duration/multi-step/strong state characteristics: The state of the sandbox needs to be maintained, and costs need to be controlled more reasonably.
So, how to build a reliable Agent runtime environment? ACS has systematically optimized for AI Agent scenarios, achieving upgrades in strong security isolation, large-scale elasticity, and state persistence functions.
Strong Security Isolation
Using ACS to create code/browser/desktop sandboxes, with default computing environments providing strong security isolation, further enhanced by capabilities such as Network Policy and Fluid, achieving end-to-end security isolation from runtime environments to storage and networks.
Large-Scale Elasticity
ACS has newly released image caching acceleration capabilities. Based on cloud disk snapshot technology, it can support thousands of Pods with second-level image loading. Combined with sandbox resource pre-scheduling optimization based on user load characteristics, it achieves large-scale concurrent elasticity of 15,000 sandboxes per minute, allowing businesses to flexibly use the cloud as needed.
State Persistence Function
ACS will soon release the capability to maintain sandbox state, supporting one-click hibernation and rapid awakening of sandboxes. During hibernation, the CPU and memory resources of the sandbox are released, reducing resource costs during hibernation; at the same time, it can quickly awaken the sandbox based on user requests, maintaining the memory state of the sandbox and fully restoring it to the environment before hibernation, achieving the best balance between cost and experience.
In addition, the performance of ACS’s performance-type instances has increased by up to 24% per core, allowing existing customers to smoothly enjoy performance benefits without modifying any code, further enhancing the operational efficiency of various Agent loads.
These capabilities have been validated in practical applications: even in the face of instantaneous high concurrency, complex task chains, and long-running scenarios, the platform can still provide a stable, secure, and efficient execution environment, providing solid support for the large-scale implementation of the next generation of intelligent applications.
06｜Leading the AI-Native Era with
End-to-End Container Technology
In summary, Alibaba Cloud’s container technology system is significantly leading in the following areas.
Ultra-Large Scale Heterogeneous Scheduling
Supports efficient scheduling of GPU computing power and RDMA network resources, natively supports AI task scheduling, supports fine-grained sharing and isolation of GPUs, and includes built-in network topology-aware scheduling to enhance cluster resource utilization and operational efficiency; supports unified management and flexible elastic scaling of public cloud, hybrid cloud, on-premises IDC, and multi-cloud heterogeneous computing power;
End-to-End Performance and Stability Assurance
Covers the entire process of data processing, model training, and inference, accelerating access to training data and model data through distributed caching and data affinity scheduling. It also possesses rich monitoring and diagnostic capabilities, with second-level fault detection and self-healing capabilities, ensuring the stability and predictable response of AI services during long-term operations.
Security Isolation and Elastic Scaling for Agent Sandbox Scenarios
Provides secure sandboxes, state persistence, and minute-level large-scale elastic creation capabilities for scenarios involving AI Agents and multi-tool invocation, effectively preventing high-risk instruction execution and resource abuse risks.
Open Source Leadership and Ecosystem Co-Building
Provides core functional upgrades in projects such as Argo Workflows (ultra-large scale task orchestration), Fluid (distributed data acceleration), Koordinator (scheduler for AI applications and large-scale mixed workloads), and RoleBasedGroup (AI inference workload management), while also giving back to the open-source community in multiple technical achievements, promoting the standardization of AI workload management.
More details on open-source projects:
-
Argo Workflows: https://argoproj.github.io/workflows/
-
Fluid: https://fluid-cloudnative.github.io/
-
RoleBasedGroup (RBG): https://github.com/sgl-project/rbg
-
Koordinator: https://koordinator.sh/
This bottom-up end-to-end technological advantage can help enterprises not only support the stable operation of AI applications in the AI-native era but also continuously provide industry-leading foundational capabilities in performance optimization and cost control, laying a reliable foundation for building the standard form of future AI infrastructure.
Currently, Alibaba Cloud containers have been scaled and implemented in various AI scenarios across multiple industries in China, serving clients such as Kimi, Xiaopeng Motors, MiniMax, and Yulu Robotics.
The technical capabilities of Alibaba Cloud containers have also received high recognition from authoritative institutions.
Gartner® released the 2025 Magic Quadrant for Container Management report, in which Alibaba Cloud was selected as a “Leader” and is the only Chinese technology company to be included in the “Leader” quadrant for three consecutive years in the Asia-Pacific region. Additionally, in the 2025 Critical Capabilities for Container Management report published by Gartner®, Alibaba Cloud ranked third globally in AI workload scenarios.
At the same time, the IDC report “China Container Market Share, 2024: From Cloud Native to AI Native” indicates that Alibaba Cloud holds the first position in the overall market share of China’s container market in 2024 with a share of 26.6%, maintaining the top spot for three consecutive years. In the segmented public cloud container market, Alibaba Cloud ranks first with a share of 30.5%. The report highly recognizes Alibaba Cloud’s technical strength in the container field, evaluating it as having leading full-stack container capabilities.
Furthermore, at the KubeCon+CloudNativeCon North America conference held in Atlanta, USA, the Cloud Native Computing Foundation® (CNCF®) announced that Alibaba Cloud became one of the first vendors to pass the v1.0 certification. This certification means that Alibaba Cloud’s Kubernetes platform can provide customers with a consistent deployment experience and stable cross-environment performance when running mainstream AI frameworks and production-level AI applications.
In the future, AI technology will continue to drive digital upgrades across industries, and the requirements for infrastructure will continue to increase: larger model scales, more complex task chains, stricter security isolation, and shorter iteration cycles all mean that underlying technologies must continue to evolve. Alibaba Cloud containers will continue to optimize performance, stability, security, and cost control, providing a more solid support for the AI-native era, enabling enterprises to navigate the ever-changing technological waves steadily and create value together.
/ END /
Source: Alibaba Cloud
