Author Introduction
Zhang Yue
Researcher in Frontier Technology at China Mobile Communications Corporation, mainly engaged in research on industrial protocols and real-time virtualization technologies.
Liu Weizhe
Chief Researcher at China Mobile Communications Corporation, mainly engaged in research on real-time operating systems and industrial control systems.
Fang Zhengzheng
Researcher in Frontier Technology at China Mobile Communications Corporation, mainly engaged in research on real-time operating systems and industrial control systems.
Wang Xize
Technical Manager at China Mobile Communications Corporation, mainly engaged in research on the Internet of Things and industrial Internet.
Wang Xiaoying
Researcher in Frontier Technology at China Mobile Communications Corporation, mainly engaged in research on real-time operating systems and industrial control systems.
Ma Shuai
Deputy Director of the Internet of Things Technology and Application Research Institute at China Mobile Communications Corporation, mainly engaged in research on 5G and industrial Internet.
Reference Format:
Zhang Yue, Liu Weizhe, Fang Zhengzheng, et al. Research and Practice of 5G Virtualization PLC Technology[J]. Information Communication Technology and Policy, 2023, 49(11): 8-17.
Research and Practice of 5G Virtualization PLC Technology
Zhang Yue Liu Weizhe Fang Zhengzheng Wang Xize Wang Xiaoying Ma Shuai
(China Mobile Communications Corporation, Beijing 100053)
Abstract: With the standardization of PLC software and the development of 5G networks, PLCs are transforming from dedicated control devices to software functionality services, and industrial control is evolving from field to ubiquity, from hardware-software integration to software-defined directions. First, by analyzing the evolution trend of industrial control systems, a new type of 5G industrial control system based on 5G virtualization PLC as the technical foundation is proposed. Secondly, the technical characteristics of 5G virtualization PLC are elaborated from aspects such as architecture and key technologies, and the practice of 5G virtualization PLC technology conducted with the 5G industrial gateway is introduced. Finally, the development trend of virtualization PLC is anticipated, and the challenges faced by the development of 5G virtualization PLC technology are analyzed.
Keywords: Virtualization PLC; 5G Industrial Control; Real-Time Virtualization; End-Edge-Cloud Collaboration
0 Introduction
Industrial control is the core link of industrial production. The Programmable Logic Controller (PLC) is a key device for industrial automation control, widely used in various aspects of the national economy such as industry, transportation, and municipal affairs. PLCs are industrial control devices formed by introducing microelectronics, computers, automatic control, and communication technologies based on sequential controllers, aimed at replacing relays, executing logic, timing, counting, and other sequential control functions, establishing flexible programming control systems[1]. Traditional PLC industrial control software is tightly coupled with hardware, with foreign products occupying a major market share, leading to issues such as high costs and insufficient scalability.
1 Evolution Trends of Industrial Control Systems
With the integration of control science with computation, information, and communication disciplines, control theory has evolved from classical feedback control and modern control to data-driven intelligent control, and control systems have progressed from point control and networked control to distributed cloud control. Through the innovative fusion of Information and Communications Technology (ICT) and Operational Technology (OT), traditional industrial control systems are gradually evolving into new types of industrial control systems characterized by connectivity, scalability, and data-driven decision-making, specifically manifested in the following two major trends[1].
(1) Closed and isolated dedicated control architectures are moving towards open and decoupled universal control architectures.
With the emergence and development of 5G and edge computing technologies, the traditional five-layer industrial control architecture of ISA-95 is beginning to shift towards a three-layer architecture of “end-edge-cloud”. Among them, the evolution of control layer PLCs has become a hot topic of concern across industries. Technically, industrial control tasks are migrating from embedded dedicated devices to cloud-edge universal devices with real-time data processing capabilities. From a business perspective, soft real-time tasks such as logical control are gradually migrating to centralized control at the edge or cloud, while hard real-time tasks such as motion control remain executed on-site.
(2) Single control task processing is evolving into distributed multi-task collaborative processing.
Traditional PLCs process tasks sequentially, but the expansion of control scale has led to an increase in task volume, a wide variety of tasks, a widening gap in task priority differences, and heightened collaboration requirements among tasks, especially with the introduction of big data and artificial intelligence technologies. The efficient collaboration of traditional control tasks and data-driven IT tasks inevitably requires traditional control systems to shift from single-node processing to multi-node collaborative processing.
2 Technical Development Path of 5G Cloud-based PLC
With the development of ICT, especially network technologies represented by 5G, 5G cloud-based PLC has become an important means to solve traditional PLC problems. Based on the deployment location of PLC control tasks, there are currently three major technical development paths for 5G cloud-based PLC (see Figure 1).
Figure 1 Technical Development Paths of 5G Cloud-based PLC(1) On-site cloud-based PLC: Control tasks are deployed on industrial gateways, suitable for on-site medium to high-speed control scenarios, supporting control cycles of 1-5 ms and above, with high reliability and low deployment costs. (2) Edge cloud-based PLC: Control tasks are deployed on indoor baseband processing units (Building Baseband Unit, BBU), user plane functions (User Plane Function, UPF), or multi-access edge computing (Multi-Acess Edge Computing, MEC), suitable for workshop or factory-level medium to low-speed centralized control scenarios, supporting control cycles of 20 ms and above, with low deployment costs. (3) Wide-area cloud-based PLC: Control tasks are deployed on central cloud servers, suitable for factory-level low-speed collaborative control scenarios. With enhancements through fiber optics and deterministic network technologies, end-to-end latency can be reduced to 5 ms, but deployment costs are higher. Currently, the technical route of 5G cloud-based PLC faces issues such as non-unified architecture, lack of support for edge collaboration, and inability to orchestrate PLC applications, especially as 5G networks face challenges of latency, jitter, and reliability in industrial control. To address these issues, this paper proposes a 5G virtualization PLC technical architecture that achieves three “unifications”: a unified operating environment, unified deployment scheduling, and a unified development and operation portal through “end-edge-cloud” collaboration.3 Technical Architecture of 5G Virtualization PLC3.1 Typical Features of Virtualization PLCVirtualization technology is a resource management technology that creates an abstraction layer on computer hardware using software technology, dividing the hardware resources of a single computer into multiple virtual computers, thereby enhancing resource utilization efficiency and security. Virtualization technology is widely used in cloud computing, and with the continuous development and maturation of virtualization technology, it is also being applied in more fields and different hardware architectures[2]. Virtualization PLC (vPLC) separates PLC control tasks from hardware devices by creating a virtualized operating environment, allowing PLC tasks to be deployed on various different network element devices. vPLC has three typical features. (1) Decoupling of PLC hardware and software: Traditional PLCs use embedded hardware and real-time operating systems with tight coupling. vPLC introduces a PLC operating environment that provides loading, execution, and scheduling of PLC tasks, achieving decoupling of PLC tasks from real-time operating systems. (2) Virtual operation of PLC: Through virtualization technology, heterogeneous operating systems can be deployed and run on general-purpose hardware. The PLC operating environment is deployed on a virtual operating system, achieving decoupling of PLC tasks from underlying hardware. (3) Orchestration and scheduling of PLC: vPLC is essentially a PLC software service that can be orchestrated and deployed within the “end-edge-cloud” system, and can also be dynamically scheduled during operation, providing flexibility and scalability that are significantly different from traditional PLCs. Compared to traditional PLCs, vPLC improves system flexibility and scalability, reduces equipment and operation and maintenance costs, and greatly promotes the updating and redesign of production lines[3].3.2 Virtualization PLC Provides a Unified Technical Foundation for 5G Industrial Control SystemsThe 5G industrial control system is centered around vPLC, aiming to provide an open system architecture for Control as a Service (CaaS), where PLC control functions can be deployed ubiquitously and flexibly reused, enabling devices to be plug-and-play under heterogeneous networks. The architecture of the 5G industrial control system is designed based on the concept of “end-edge-cloud” collaboration, including an operational layer, service layer, and development layer (see Figure 2).
Figure 2 Architecture of 5G Industrial Control System(1) Operational Layer: Based on real-time virtualization technology, it provides a unified real-time operating environment for vPLC on general-purpose hardware. Virtualization supports the mixed deployment of real-time operating systems and non-real-time operating systems, supporting the dynamic expansion of vPLC. (2) Service Layer: Responsible for deploying vPLC to different physical nodes while managing its lifecycle, achieving unified deployment and scheduling of PLC industrial control services. (3) Development Layer: Provides a PLC application development environment, compilation, and debugging tools, offering a unified development and operation portal for the 5G industrial control system. The 5G industrial control system has several technical advantages. First, it provides a unified architecture for three types of cloud-based PLC technologies. On one hand, it supports the migration of control centers from the field to the edge and central cloud, freeing PLC control from the constraints of the on-site environment and enabling centralized deployment. On the other hand, edge-side vPLC ensures low-latency control of on-site devices, supporting various high real-time applications including motion control. Secondly, deploying vPLC on various 5G network elements provides an integrated capability of “connectivity + computing power + PLC capability” for industrial control, creating a new flat industrial control system that breaks the traditional monopoly ecology of industrial control. Thirdly, based on a general-purpose software and hardware architecture, industrial control costs can be reduced, pushing “software-defined industry” towards maturity.4 Key Technologies of 5G Virtualization PLC4.1 Real-Time VirtualizationPLC tasks operate as processes on the operating system, running in cycles according to the industrial control cycle, and must ensure that the PLC process can receive external inputs and obtain CPU processing time in each industrial control cycle. To provide reliable PLC control services, the real-time performance of the operating system is a key factor. Real-time virtualization technology refers to the mixed deployment of real-time computing tasks and non-real-time computing tasks on general-purpose hardware through software and hardware virtualization, supporting the consolidation of computing tasks from multiple embedded devices to run on the same general device. Real-time virtualization technology not only ensures real-time performance but also leverages the good hardware adaptation and rich application capabilities of general-purpose operating systems, with advantages such as reduced equipment costs, size, and power consumption, achieving compatibility in heterogeneous device ecosystems. Currently, various virtualization technologies exist in the industry, mainly divided into hardware partitioning, full virtualization, para-virtualization, and operating system virtualization, among which three are particularly suitable for real-time virtualization transformation. (1) Hardware Partitioning: Dividing the underlying hardware resources into mutually independent partitions, each with its own independent operating system. Hardware partitioning has good real-time performance, close to bare-metal performance, but cannot achieve resource sharing, has insufficient scalability, and low resource utilization. Particularly, peripheral I/O needs to be partitioned in advance, which increases technical costs. (2) Real-Time Linux Containers: Transforming Linux from a time-sharing system into a real-time system through Preemption Patch, Xenomai, etc., and then providing resource isolation through lightweight virtualization in containers to build a real-time, virtualized operating environment for PLC. This solution can fully utilize the mature software and hardware ecosystem of Linux, reducing the cost of PLC software migration. Although the container running overhead is low, the real-time performance of the modified Linux kernel is still lower than that of real-time operating systems (Real-Time Operating System, RTOS), experiencing jitter issues in scenarios such as servo motion control. (3) Microkernel Virtualization: Using a microkernel as Type-1 virtualization software (Hypervisor) to achieve mixed deployment of RTOS and general-purpose operating systems (General Purpose Operating System, GPOS). Microkernels are simplified in functionality compared to monolithic kernels, have low overhead, and good security, providing both hardware virtualization and high real-time performance. Currently, this technology is actively developing in industrial, automotive, and robotic fields, but faces challenges such as immature technical ecosystems and significant hardware adaptation difficulties. The 5G industrial control system can choose different real-time virtualization implementation methods based on different network elements and scenarios. For edge and cloud-side vPLC, real-time Linux container technology is prioritized due to its low modification overhead and convenient deployment. On the other hand, container orchestration and redundancy backup technologies are mature and carry low risk. Given the high real-time requirements of edge-side vPLC and the need for edge collaboration, real-time Linux container or microkernel virtualization solutions are prioritized for edge-side vPLC.4.2 5G Deterministic NetworksThe 5G industrial control system has two major requirements for network connectivity. First is extremely low latency in the air interface. In scenarios such as industrial motion control, inter-controller communication, and high-speed logic control, the control cycle is short (1-5 ms), and reliability requirements are high (>99.9999%), placing high demands on wireless air interface performance for vertical and horizontal data transmission across layers and systems. Second is deterministic data transmission in heterogeneous network environments. The uncertainty of network-induced latency is a key factor affecting the stability of control systems. Existing network linear jump system modeling can compensate for unstable random delays, but still struggles to meet real-time control performance demands[4]. Deterministic networking is a type of network that can provide users with deterministic quality of service and has the ability to flexibly switch between deterministic and non-deterministic services and autonomously control the level of quality of service provided. Typical deterministic network technologies are shown in Table 1. Overall, deterministic network technology is the main pathway for achieving network connectivity in 5G industrial control systems. 5G Deterministic Networking (5GDN) utilizes high-precision clock synchronization, traffic shaping, resource reservation, and other technologies to realize deterministic bandwidth, deterministic latency, and 99.9999% connection reliability based on 5G network slicing, creating a predictable, plannable, and verifiable wireless network with deterministic capabilities that provides a “differentiated + deterministic” service experience[5]. The combination of 5G deterministic networks with on-site networks and edge computing technologies can achieve end-to-end deterministic control[6].Table 1 Typical Deterministic Network Technologies
4.3 End-Edge-Cloud Collaborative OrchestrationThrough real-time virtualization, traditional PLC hardware devices transform into PLC software services decoupled from hardware. In the 5G cloud-edge-end architecture, flexible deployment of PLC software services is required, hence the need for a unified orchestration and scheduling platform for vPLC. vPLC orchestration and scheduling support deploying vPLC as containers or virtual machines to 5G industrial gateways, 5G industrial base stations, 5G industrial UPFs, and MEC network elements. The orchestration methods include interconnecting computing chips with IO chips (Controller to IO, C2IO) and interconnecting computing chips (Controller to Controller, C2C). C2IO refers to communication from PLC to IO, including both main PLC to IO and slave PLC to IO scenarios. C2C refers to communication from PLC to PLC, mainly the main PLC controlling the slave PLC. In typical scenarios, production control systems, industrial software, and enterprise information management systems are deployed in the cloud. The edge side mainly deploys the main PLC, responsible for interfacing with cloud systems, generating control instructions for subordinate PLCs deployed on the end side through C2C type. The subordinate PLC on the end side is mainly responsible for C2IO communication, receiving control instructions from the edge main PLC and controlling on-site IO devices. Unlike orchestration systems such as Kubernetes and Kubevirt, vPLC has high real-time and reliability requirements, making it challenging for ordinary containers or virtual machines to meet the stringent industrial control cycle requirements. Orchestration and scheduling of vPLC need to sacrifice some scalability to achieve higher real-time performance and reliability. Among them, vPLC redundancy hot backup is a core component of orchestration and scheduling. The 5G vPLC orchestration and scheduling model is shown in Figure 3.
Figure 3 5G vPLC Orchestration and Scheduling Model5 Practice of 5G Virtualization PLC5.1 Experimental Scenarios and SolutionsTo promote the deep integration of 5G and industry, China Mobile, in collaboration with industry partners, has conducted technical practices of 5G vPLC, which have been validated in over ten industrial enterprises. Taking a typical customer as an example, China Mobile deployed a 5G network for this customer in the production workshop, providing network services for Automated Guided Vehicle (AGV) material handling. The business requires planning different moving paths for AGVs based on different types of materials, guiding AGVs to move between warehouses and different production sections. In response to this scenario’s needs, the 5G industrial control system has built an “end-edge collaboration architecture of main vPLC-slave vPLC” (see Figure 4). On the edge side, the main vPLC is deployed on the UPF, achieving centralized deployment of PLC. The main vPLC is responsible for receiving task instructions issued by the Manufacturing Execution System (MES), packaging the tasks into control instructions, and sending them to the slave vPLC carried by the AGV while also receiving feedback information during the execution process from the slave vPLC. On the field side, the slave vPLC is deployed on the 5G industrial gateway, achieving a dual replacement of PLC and 5G data transmission devices (Data Terminal Unit, DTU). On one hand, the slave vPLC receives control instructions sent by the main vPLC, controlling the AGV’s driving system through sensors to achieve precise movement and positioning. On the other hand, when the AGV reaches its destination, the slave vPLC feeds back the task execution results to the main vPLC, reporting the completion status of the task.
Figure 4 Application Scenario Illustration5.2 System DevelopmentReal-time container technology is used to deploy vPLC on the 5G UPF and industrial gateway, transforming them into 5G industrial UPF and 5G industrial gateways, with the main configurations shown in Table 2.Table 2 UPF and Gateway Software and Hardware Configurations
The 5G industrial UPF, 5G industrial gateway, and orchestration platform form the system architecture shown in Figure 5, with the main R&D work as follows.
Figure 5 System Technical Architecture(1) Linux Kernel Modification: First, by integrating the Preempt-RT patch, the original kernel of the UPF and gateway is modified into a real-time kernel, with the modified kernel versions being Linux 4.18.16-rt and Linux 4.4.167-rt. Under high load, the maximum processing delay of processes can be reduced from 8 ms to 500 μs. Secondly, CPU resource isolation is configured to keep real-time tasks resident on specific CPU cores, reducing task switching overhead and further lowering the maximum processing delay jitter to below 300 μs. Thirdly, interrupt routing is designed to route peripheral interrupt responses to other CPU cores, reducing interference with the operation of real-time tasks. Through the above Linux kernel modifications, the maximum processing delay jitter can be reduced to below 100 μs (see Figures 6 and 7).
Figure 6 Processing Delay of Processes Before Linux Kernel Modification
Figure 7 Processing Delay of Processes After Linux Kernel Modification(2) PLC-runtime Containerization: To support dynamic expansion of PLC, a container virtual operating environment needs to be built for PLC-runtime. First, a PLC container image is built based on Alpine, mainly including the integration of PLC-runtime and 32/64-bit dynamic link libraries. Currently, the system has adapted two domestic PLC-runtimes. Secondly, a container volume corresponding to the vPLC instance is created for data persistence generated during the PLC container’s runtime. Thirdly, during the PLC container’s runtime, access to external peripheral resources is required. For RS232/485, CAN, and other peripheral interfaces, access is achieved through the mapping of device files. For IP network interfaces, access is achieved through port mapping, mainly mapping ports such as PLC program download interfaces and external service ports like Modbus TCP slave service ports.(3) vPLC Orchestration: To support unified orchestration of UPF and gateways and reduce gateway resource overhead, a lightweight orchestration system is developed using a B/S architecture. Users perform front-end operations through a browser, while the back-end is divided into management and orchestration modules to realize PLC container orchestration and deployment. The management module is deployed on the server, providing functions such as web access, container image downloading, configuration, and monitoring of PLC containers. The orchestration module is deployed on UPF and gateways, mainly providing functions such as container image pulling, container environment configuration, and operational management. The management and orchestration modules communicate with each other through HTTP. For instance, in container operation monitoring, the orchestration module periodically communicates with the local Docker service through Docker Client to obtain container operation status information and then reports it to the management module, which takes action when it detects anomalies. The vPLC orchestration system interface is shown in Figure 8.
Figure 8 Virtualization PLC Orchestration System Interface5.3 Experimental EffectsAfter the on-site deployment of the 5G industrial UPF and 5G industrial gateway, the main vPLC and slave vPLC, as well as the slave vPLC and IO, are configured to communicate via Modbus TCP, with the slave vPLC’s industrial control cycle set to 5 ms, and the main vPLC’s industrial control cycle set to 20 ms, with each cycle of the main vPLC including one communication with the slave vPLC. After long-term operational tests (14 days), the AGV scheduling operated well without any downtime or route deviation failures. The average execution time of the main vPLC was measured at 457 μs, with a maximum execution time of 599 μs and a maximum jitter of 532 μs (see Figure 9). In industrial control, the general requirement is to control latency jitter within 10%-15% of the industrial control cycle. Based on this calculation, the 5G industrial control system using the main-slave vPLC end-edge collaboration architecture can support industrial control cycles of 5 ms and above, meeting the performance needs of medium to high-speed industrial control. Given that the current 5G air interface latency is still above 5 ms, the end-edge collaboration scheme combines the advantages of centralized deployment and support for medium to high-speed control compared to the single deployment scheme of vPLC on the edge side.Figure 9 Measurement of Virtualization PLC PerformanceTraditional PLC scheme costs include main PLC, slave PLC, 5G DTU, and other device costs, while the 5G industrial control system scheme includes the cost of 5G industrial UPF software licensing and the 5G industrial gateway, significantly reducing the procurement price of equipment by over 50% compared to traditional PLC schemes. In addition, the 5G industrial control system supports unified orchestration, allowing centralized remote operations for PLC application downloading, updates, and maintenance, with flexible expansion and reducing operation and maintenance time by over 80%. In summary, the 5G industrial control system can provide PLC industrial control services through software upgrades on 5G network elements without the need for hardware modifications, not interfering with the existing services of UPF or gateways, supporting medium to high-speed industrial control, and offering advantages such as a rich application scenario, cost reduction, ease of maintenance, and scalability.6 ConclusionWith the arrival of the Fourth Industrial Revolution, PLCs, as the core of industrial control, are increasingly unable to meet the needs of industrial interconnectivity development. Promoting the innovative integration of ICT and OT, accelerating the fusion of technology with various production factors, and constructing a new type of 5G industrial control system present broad prospects. On one hand, 5G vPLC provides a unified technical architecture and path for 5G cloud-based PLC, and on the other hand, building a 5G vPLC technical ecosystem will encourage PLC manufacturers to transition from hardware products to software services, providing new development space for domestic PLC technologies and potentially breaking the existing market pattern of “seven countries and eight systems, foreign monopolies”. Currently, the 5G industrial control system is still in its early development stage and requires collaboration among industry players to tackle key aspects such as microkernel virtualization, deterministic networks, and redundancy hot backups, continuously improving and refining the 5G industrial control system. Additionally, strengthening cooperation with universities to cultivate composite technical talents in industrial automation and information technology is essential to promote high-quality development of industrial control.