Click on the blue text丨Follow us
Source: Journal of Electronics and Information
Authors: Zhou Xiaotian, Sun Shang, Zhang Haixia, Deng Yiqin, Lu Binbin
Abstract
Keywords
AI quality inspection is an important part of smart manufacturing. The equipment generates a large number of computation-intensive and delay-sensitive tasks during product quality inspection. Due to insufficient computing power of the equipment, the delay in executing inspection tasks is significant, greatly affecting production efficiency. Multi-access edge computing (MEC) provides nearby computing power for devices by offloading tasks to edge servers, thus improving task execution efficiency. However, dynamic factors such as channel variation and random task arrival in the system greatly affect offloading efficiency and pose challenges to task scheduling. This paper focuses on the AI quality inspection task scheduling system empowered by MEC, studying the long-term delay minimization problem of joint task scheduling and resource allocation. Given the large state space of the problem and the continuous variables in the action space, this paper proposes a real-time task scheduling algorithm based on deep deterministic policy gradient (DDPG). The proposed algorithm can provide optimal decisions based on real-time system state information. Simulation results show that the proposed algorithm outperforms benchmark algorithms in terms of performance and task execution delay.
1. Introduction
AI quality inspection [1,2] is a crucial part of smart industrial production, effectively ensuring product quality while reducing costs and increasing efficiency. Intelligent tasks generated during the AI quality inspection process, such as product logo recognition and defective parts detection, require substantial computing resources and exhibit computation-intensive characteristics. Additionally, the industrial production line has high delay requirements for quality inspection; if the task completion delay is too long, it can lead to production interruptions and economic losses. Therefore, AI quality inspection tasks are also delay-sensitive. However, quality inspection terminal devices typically have limited computing power, making it difficult to process these tasks with both delay sensitivity and computation intensity within a short time. To address this issue, some studies have proposed using cloud computing-enabled industrial Internet of Things (IoT) technology [3,4] to offload tasks to remote cloud centers [5], leveraging the strong computing resources of cloud servers for rapid task processing. However, the physical distance between cloud servers and devices introduces significant communication delays during task offloading. Moreover, concentrating a large number of tasks for processing at a single cloud server adds pressure to the backbone network and can easily lead to network congestion. Therefore, in cloud architecture-based industrial IoT, high communication delays can lead to increased end-to-end service delays, making it difficult to meet the low-latency requirements of AI quality inspection.
In this context, multi-access edge computing [6,7] (MEC) has been proposed by the industry as an evolution of cloud computing. Edge computing provides nearby computing services for devices by distributing small servers at the network edge. Compared to cloud architecture, edge servers are closer to the terminals, reducing communication delays during task offloading. Additionally, the decentralized distributed deployment avoids congestion in the backbone network caused by mass terminal offloading. Therefore, compared to cloud computing architecture, edge computing architecture can better realize computing power sinking, ensuring the low-latency requirements of AI quality inspection tasks and improving production efficiency. However, in practical applications, the industrial IoT environment based on wireless transmission is dynamic [8,9], and task offloading delays are severely affected by time-varying channel fading. Furthermore, the distributed deployment of computing resources in edge computing architecture can lead to mismatches between computing resources and terminal computing demands. Therefore, how to design an effective joint optimization strategy for task scheduling and resource allocation to match the dynamic changes in the network environment and achieve long-term minimization of AI quality inspection task processing delays is a key research point.
Regarding this issue, literature [10] modeled the joint computation offloading and bandwidth allocation problem aimed at minimizing the average application response time for a system with a three-layer cloud-edge-terminal network architecture. By transforming it into a piecewise convex optimization problem, they obtained the optimal solution. Literature [11] designed a strategy for delay-sensitive tasks that combines task segmentation and resource allocation, proposing a multi-dimensional search and adjust (MDSA) offline algorithm to minimize system delays. Literature [12] focused on the scenario of cooperative computing among multiple edge servers in MEC, modeling the task queue as an M/M/s model under storage capacity and offloading success rate constraints. They designed an optimization problem to minimize the average task waiting delay and obtained resource allocation decision schemes through one-dimensional optimization search methods. Literature [13] constructed a joint optimization problem for computation offloading and resource allocation under delay constraints in ultra-dense heterogeneous edge computing networks, designing a hybrid particle swarm optimization algorithm for problem solving. Literature [14] designed a collaborative task offloading and resource management strategy aimed at minimizing energy consumption for multi-user MEC networks, employing an adaptive genetic algorithm to obtain optimization schemes.
The above research has addressed the task scheduling and resource allocation issues in MEC-enabled industrial IoT systems, but they have employed strong static environment assumptions, such as unchanged channel states during task offloading and fixed inter-arrival times of tasks, without considering the long-term performance of the system. However, in AI quality inspection systems, the movement of terminal devices and humans can lead to channel fading changes [8,9], and the actual task generation arrival times are also random. Therefore, existing works cannot fully meet the needs of long-term continuous operation and dynamic environmental changes in AI quality inspection scenarios. Furthermore, most of the aforementioned studies adopt traditional convex optimization or heuristic optimization algorithms, which have high computational complexity and can only obtain quasi-static instantaneous optimization solutions, making them unsuitable for solving sequential decision problems under real-time perception state changes in AI quality inspection scenarios.
Considering the dynamic changes in channel conditions and random task arrivals, this paper studies the joint optimization problem of task scheduling and computing resource allocation for AI quality inspection tasks with both computation-intensive and delay-sensitive characteristics, aiming to minimize the long-term task processing delay of the system. Since the modeled problem is a sequential decision problem, this paper transforms it into a Markov Decision Process (MDP) for solving. The transition probability of the MDP problem is unknown, and the state space is large. The action space contains both discrete task offloading decision variables and continuous resource allocation decision variables, making it difficult to solve using traditional dynamic programming methods. Therefore, this paper leverages reinforcement learning techniques to propose a real-time task scheduling algorithm based on deep deterministic policy gradient (DDPG). The algorithm achieves separation of mixed action space by improving the output layer structure of the Actor network, enabling real-time perception of the system environment and real-time scheduling of tasks. Compared to existing research, the proposed real-time task scheduling scheme based on DDPG can better adapt to the dynamics of AI quality inspection systems, achieve reasonable optimization of computing resource allocation, and effectively reduce system task processing delays. Simulation experiments demonstrate that the proposed real-time task scheduling algorithm based on DDPG in the MEC-enabled AI quality inspection system has fast convergence advantages and significantly outperforms other benchmark schemes.
2. System Modeling and Problem Establishment
2.1 System Architecture
As shown in Figure 1, the MEC-enabled AI quality inspection system considered in this paper consists of one MEC server and N AI quality inspection devices. The edge cloud layer includes the server and a centralized control center, which is responsible for collecting system environmental state information, generating task scheduling and computing resource allocation decisions. The tasks generated by AI quality inspection devices can be executed in two modes: they can be processed locally by the device or offloaded to the MEC server for processing, with the execution results returned. In this system, it is assumed that the coverage radius of the MEC server is L (m), and all AI quality inspection devices are within its service range. Each device has limited computing resources, and the device set is represented as N={1,2,…,N}. It is assumed that the entire system operates in time slots [15], with a working time of T time slots, and each time slot length is T_s. At the beginning of each time slot, the centralized control center collects system state information, makes task scheduling and computing resource allocation decisions, and sends control instructions to AI quality inspection devices and the MEC server. The system will execute the command for the remaining time of that time slot, completing task offloading or local computation.
Figure 1: Task Scheduling Structure of MEC-enabled AI Quality Inspection System
To align with the actual AI quality inspection working mode, it is assumed that the tasks generated by AI quality inspection devices are serial task streams, with tasks arriving randomly following a Poisson distribution. Considering the workload differences among different devices, the average task arrival rate of the AI quality inspection device set is defined as λ=(λ_1,…,λ_n,…,λ_N), where λ_n is the average task arrival rate of the n-th device. The task set generated by the n-th device is represented as M_n={1,2,…,M_n}, which is the total number of tasks generated during operation. It is assumed that there are F types of tasks in the system, corresponding to different detection products. The task set is defined as G={G_1,…,G_j,…,G_F}, where type G_j represents the j-th type of task, which can be further described by the tuple G_j=(d_j,k_j), where d_j is the size of the input data for the task, and k_j is the number of CPU cycles required to process one unit of data, measured in cycle/bit. Considering that different types of tasks have different detection targets, their data sizes and CPU cycle requirements per unit of data also differ. Additionally, since the output data size after task processing is much smaller than the input data size, it is assumed that during processing at the MEC server, the result return time can be neglected [16].
During the task scheduling process, all unprocessed tasks are stored in a virtual task queue waiting for execution, with earlier tasks having higher execution priority. When a task is executed, it is first stored in the local scheduling unit waiting for scheduling. Subsequently, according to the scheduling decision, the task will be transferred to the local execution unit for processing or offloaded to the MEC server via the data transmission unit. After the MEC server’s execution unit receives the raw data, it processes it and ultimately sends the computation results back to the AI quality inspection device. The system devices will handle subsequent processing such as diverting defective products based on the task execution results. For convenience of description, this paper defines the data volume in the scheduling unit at time slot t as 
, the data volume in the MEC execution unit as 
, and the data volume being transmitted in the data transmission unit as 
.
2.2 Communication Model
Assuming that data transmission between the MEC server and AI quality inspection devices uses orthogonal frequency division multiplexing, each device uses a different frequency band. According to literature [17], the maximum transmission rate from device n to the MEC server is defined as
(1)
where p_n is the transmission power of device n, B is the channel bandwidth, N_0 is the power spectral density of Gaussian white noise. g_n(t) is the wireless channel gain of device n at time slot t, which includes large-scale and small-scale fading. The large-scale fading is represented as 
, where d is the distance from the device to the edge server, and β_1 and β_2 are the path loss constant and path loss exponent, respectively. Small-scale fading follows a Rayleigh distribution [9,17].
In time slot t, let x(t)=(x_1(t),…,x_n(t),…,x_N(t)) represent the offloading decisions of AI quality inspection devices. For device n, x_n(t)=0 indicates that the task is processed locally, while x_n(t)=1 indicates that the task is offloaded to the MEC server for processing. Thus, the data volume offloaded by device n in time slot t is
(2)
2.3 Computing Model
Let the processing frequency of device n’s processor be 
. In time slot t, the amount of data that needs to be computed locally is represented as
(3)
where k_n is the number of CPU cycles required for device n to process 1 bit of task data.
Additionally, let φ(t)=(φ_1(t),…,φ_n(t),…,φ_N(t)) denote the proportion of computing resources allocated by the MEC server at time slot t, where φ_n(t) represents the proportion of computing resources allocated to device n by the MEC server. Therefore, the amount of data processed by device n through the MEC server in time slot t is
(4)
where f^c is the processing frequency of the MEC server, and k_n is the number of CPU cycles required for device n to process 1 bit of task data.
Based on the above definitions and formulas, the transition process of the system state in adjacent time slots can be derived as
(5)
(6)
2.4 Optimization Problem Establishment
This paper aims to minimize the long-term task processing delay of the system through the joint optimization of task offloading decisions and computing resource allocation. Since this paper schedules tasks on a time slot basis and considering that each device has limited storage space, once the queue length of waiting tasks exceeds the queue capacity, subsequent arriving tasks will overflow. Define the task overflow indicator variable as α_n,m(t), where α_n,m(t)=1 indicates that the m-th task of device n has overflowed, otherwise α_n,m(t)=0. Clearly, the value of α_n,m(t) is influenced by the length of the task queue at time slot t and the decision variables x_n(t) and φ_n(t). Due to the uncertainty of decisions and the randomness of task arrivals, this paper assumes that the value of α_n,m(t) can be obtained by monitoring the device queue state. Specifically, if it is detected that the length of device n’s task queue at time slot t exceeds capacity, α_n,m(t) takes the value of 1, otherwise it is 0. Define d_n,m(t) as the delay of the m-th task of device n at time slot t. If the task is waiting in the virtual queue or is being executed, then d_n,m(t)=T_s, otherwise d_n,m(t)=0. Moreover, to ensure system reliability and avoid task overflow, this paper adds a time penalty parameter ξ for overflow tasks. Thus, the optimization problem can be expressed as
(7)
In problem (7), C1 ensures that each device can only choose one computing mode in the current time slot. C2 specifies the constraints on the proportion of computing resources that each device can obtain from the server, and C3 indicates that the total computing resources allocated by the MEC server to all devices cannot exceed the total computing resources of the MEC server.
3. Algorithm Design
The above optimization problem is a mixed-integer nonlinear programming sequential decision problem that includes random environmental variables. This problem has a large state space, and the action space contains continuous and discrete variables, making it difficult to solve using traditional optimization methods. Therefore, this paper models the problem as an MDP and proposes a real-time task scheduling algorithm based on DDPG using deep reinforcement learning (DRL) tools.
3.1 MDP Problem Establishment
Define the five key factors of the MDP as M=(S,A,P,r,γ) [18], where S is the state space, A is the action space, P represents the state transition probability, r is the reward function, and γ∈[0,1] is the discount factor.
(1) State Space: At the beginning of each time slot, the centralized control center collects system state information as an agent. The state space can be described as
(8)
where 
is the data volume in the MEC execution unit, 
represents the data volume in the device scheduling unit, 
represents the length of the task queue of AI quality inspection devices, and 
is the channel gain between each device and the server. The dimensionality of the system’s state space is 4N.
(2) Action Space: After collecting the state information, the agent makes decisions and sends control action signals to the MEC server and AI quality inspection devices. The corresponding action space is described as
(9)
where x(t)=(x_1(t),…,x_n(t),…,x_N(t)) represents the device offloading decisions, and φ(t)=(φ_1(t),…,φ_n(t),…,φ_N(t)) represents the MEC server computing resource allocation decisions. The dimensionality of the system’s action space is 2N.
(3) State Transition Probability: The state transition process studied in this paper is determined by equations (5) and (6), with its probability defined as P(s(t+1)|s(t),a(t)), indicating the probability of the system transitioning from state s(t) to state s(t + 1) when action a(t) is chosen. It should be noted that due to the dynamic randomness of the system environment, state transition probabilities are generally difficult to obtain directly. Therefore, this paper solves the problem using reinforcement learning methods.
(4) Reward Function: The reward function r(t) represents the instantaneous reward obtained by choosing action a(t) in state s(t). Considering that the optimization goal of this paper is to minimize the long-term task execution delay of the system, the negative value of the system optimization target for each time slot is used as the reward function, expressed as
(10)
3.2 DDPG-Based Real-Time Task Scheduling Algorithm Design
The action space of the above MDP problem includes discrete offloading decisions and continuous resource allocation decisions, making it impossible to solve using traditional value-based DRL methods. Therefore, this paper proposes a DDPG-based real-time task scheduling algorithm that can solve continuous action space problems. As shown in Figure 2, the algorithm employs a dual network model consisting of an Actor network and a Critic network, and achieves separation of mixed action space by improving the output layer structure of the Actor network. Specifically, the output layer of the Actor network handles continuous resource allocation actions and discrete offloading actions in two different ways. The neurons that output resource allocation decisions use a Softmax activation function to normalize the resource allocation decisions and ensure they satisfy the constraints in equation (7) C3; the neurons that output offloading actions use a Sigmoid activation function to limit the output values within [0,1], representing the probability of task offloading, with 0.5 as the threshold for discrete action selection between 0 and 1.
Figure 2: Algorithm Structure Diagram
Define the policy π as the mapping from state to action, i.e., a(t)=π(s(t)). Define the Q-value function as Q(s(t),a(t)), representing the expected cumulative return of taking action a(t) in state s(t). According to the Bellman equation [19], Q(s(t),a(t)) is expressed as
(11)
In the algorithm shown in Figure 2, the Actor network outputs a deterministic action a(t) according to the policy π(s(t)|θ^μ(t)), i.e., a(t)=π(s(t)|θ^μ(t)). The Critic network evaluates this action by outputting an estimated Q-value Q(s(t),a(t)|θ^Q(t)), where θ^μ(t) and θ^Q(t) represent the network parameters of the Actor and Critic networks at time slot t, respectively. The training objective of the Critic network is to make its output estimated Q-value Q(s(t),a(t)|θ^Q(t)) approximate the actual value of the Q-value function Q(s(t),a(t)) [17], i.e.,
(12)
The algorithm also uses target networks to assist in training, making the learning process more stable and converging faster. Define the output of the target Actor network as 
and the output of the target Critic network as 
, where 
are the network parameters of the target Actor and Critic at time slot t. To make the output value of the Critic network approximate the actual Q-value, define the loss function as the mean square error between the two, and use stochastic gradient descent to update the network parameters θ^Q(t), with the loss function expressed as
(13)
where 
is the actual Q-value obtained from calculations.
The training objective of the Actor network is to find the policy π(s(t)|θ^μ(t)) that maximizes the Critic’s output Q-value 
. The proposed algorithm achieves this goal by adjusting the gradient parameters θ^μ(t) of the Actor network [21], with the gradient of the Actor network expressed as
(14)
Furthermore, this paper uses an experience replay buffer to store historical data tuples (s(t),a(t),r(t),s(t+1)), and during training, small batches of data are randomly sampled for model training to eliminate data correlation. To fully explore environmental information and obtain better policies, this paper adds Ornstein-Uhlenbeck (OU) noise to the deterministic actions and sets it to gradually decrease as training time increases. To enhance learning stability, the soft replacement strategy is also used for updating the target network parameters, with the target network parameter update process during each training cycle expressed as
(15)
where θ′(t) is the target network parameter at time slot t, θ(t) is the estimated network parameter at time slot t, and ε is the policy parameter.
The execution flow of the algorithm is shown in Algorithm 1. First, initialize the network parameters and other system parameters. During model training, collect data tuples (s(t),a(t),r(t),s(t+1)) through interaction with the AI quality inspection system environment and store them in the experience replay buffer. Specifically, obtain the environmental state s(t), input it into the Actor network, add noise to the output of the Actor network to obtain action a(t), execute the action, and the system outputs reward r(t) and transitions to the next state s(t+1). In each time slot training, a small batch of data samples is randomly drawn from the experience replay buffer to update the estimated network parameters of the Actor and Critic according to equations (13) and (14), and the target network parameters are updated according to equation (15). Additionally, the data in the experience replay buffer is also updated during the training process. Finally, train the network for K rounds until convergence to obtain the final model. After training, the Actor network will be deployed in the centralized control center. During system operation, the centralized control center performs real-time environmental state detection at each time slot, inferring task scheduling and resource allocation decisions from the Actor network model and sending control instructions to devices and the MEC server to complete task scheduling.
Algorithm 1: Real-Time Task Scheduling Algorithm Based on DDPG
4. Performance Simulation and Result Analysis
4.1 Simulation Settings
To verify the performance of the proposed algorithm, this paper models the MEC-enabled AI quality inspection system simulation environment using Python and conducts algorithm experimental verification using TensorFlow. The simulation parameters are shown in Table 1.
Table 1: Simulation Parameter Settings
In the proposed algorithm, both the Actor network and Critic network adopt a 4-layer fully connected structure, including 1 input layer, 3 hidden layers, and 1 output layer. The algorithm parameter settings are shown in Table 2.
Table 2: Algorithm Parameter Settings
4.2 Convergence Analysis
This paper sets three learning rates for model training comparison, conducting 3,500 training rounds, using the system’s cumulative reward value as the evaluation metric. The results are shown in Figure 3. As seen in Figure 3, when the learning rate is set to 8×10–5, the algorithm converges around 2,000 rounds, achieving the optimal convergence value. When the learning rates are set to 1.6×10–4 and 3×10–5, the algorithm converges around 1,500 and 2,000 training rounds, respectively. A larger learning rate accelerates convergence but leads to greater fluctuations, making it easy to jump out of the global optimal solution, while a smaller learning rate slows convergence and risks falling into local optima. Based on these results, this paper ultimately selects a learning rate of 8×10–5 for subsequent experiments.
Figure 3: Cumulative Reward under Different Learning Rates
4.3 Performance Comparison and Analysis
To verify the performance of the proposed algorithm, this paper compares it with three benchmark schemes, analyzing the long-term task execution delay and average task delay of the system under different numbers of devices, MEC server computing resources, and task computing resource requirements. To ensure data reliability, the average of 200 experimental runs is taken as the final experimental result. The three selected benchmark schemes are as follows:
(1) Random Offloading-Average Allocation (RO-AA) Scheme: Tasks are randomly offloaded to the MEC server, and the MEC server’s computing resources are evenly allocated to each device;
(2) All Offloading-Average Allocation (AO-AA) Scheme: All device tasks are offloaded to the MEC server, and the MEC server’s computing resources are evenly allocated to each device;
(3) Non-Real Time-Deep Deterministic Policy Gradient (NRT-DDPG) Scheme: A non-real-time environment-aware task scheduling and resource allocation scheme based on DDPG. All devices receive a fixed scheduling scheme only when tasks are generated, without real-time task scheduling and resource allocation during task execution [17].
Figure 4 shows the long-term task execution delay of the four schemes under different numbers of devices, different MEC server computing resources, and different task computing requirements. It can be seen that in all cases, the AO-AA scheme takes the longest time, followed by the RO-AA scheme, then the NRT-DDPG scheme, while the proposed algorithm has the shortest delay. The AO-AA scheme offloads all tasks to the server, leading to the underutilization of local computing resources of devices, resulting in the longest delay. Although the NRT-DDPG scheme improves performance through joint task scheduling and resource allocation, it only optimizes at task generation and does not adjust in real-time for changes in environmental states such as channels, thus performing worse than the proposed real-time scheduling scheme. Specifically, as shown in Figure 4(a), the system’s long-term task execution delay increases with the number of users. When the number of devices is 10, the proposed scheme reduces the delay by 29%, 50%, and 11% compared to the other three schemes, respectively. As shown in Figure 4(b), the long-term task execution delay of the system decreases with an increase in MEC server computing resources. When the server computing resources are 6 GHz, the proposed scheme reduces the delay by 30%, 55%, and 12% compared to the other three schemes, respectively. Figure 4(c) reveals that the long-term task execution delay of the system increases with the increase in task computing resource requirements. When the required computing resources for tasks are 2000 cycle/byte, the proposed scheme reduces the delay by 29%, 50%, and 11% compared to the other three schemes, respectively.
Figure 4: Comparison of Long-Term Task Execution Delay of Schemes
Figure 5 presents the average task execution delay of the four schemes under different numbers of devices, different MEC server computing resources, and different task computing requirements. As shown in Figure 5, the proposed algorithm has the smallest average task execution delay in all cases. When the number of devices is 10, the proposed scheme reduces the average task delay by 20%, 45%, and 5% compared to the other three schemes, respectively. When the server computing resources are 6 GHz, the proposed scheme reduces the average task delay by 30%, 53%, and 6% compared to the other schemes, respectively. When the required computing resources for tasks are 2,000 cycle/byte, the proposed scheme reduces the average task delay by 41%, 63%, and 26% compared to the other schemes, respectively.
Figure 5: Comparison of Average Task Execution Delay of Schemes
5. Conclusion
This paper studies the task scheduling problem of the MEC-enabled AI quality inspection system, aiming to minimize the long-term task execution delay of the system, and designs a task scheduling scheme that jointly optimizes offloading and computing resource allocation. Since the proposed problem is a sequential decision problem, this paper models it as an MDP. Considering the dynamic changes in the system’s environmental state, the large state space, and the coexistence of continuous and discrete optimization variables in the action space, this paper proposes a real-time task scheduling algorithm based on DDPG. This algorithm achieves separation of mixed action space by improving the output layer structure of the Actor network. The proposed algorithm can achieve real-time scheduling of tasks and real-time allocation of computing resources through real-time perception of the AI quality inspection environment. Simulation results show that the proposed algorithm has good convergence and achieves minimal system task processing delay compared to other benchmark algorithms.
Disclaimer:The articles and images reproduced in the public account are provided for reference and discussion for non-commercial educational and research purposes, and do not imply support for their views or confirm the authenticity of their contents. Copyright belongs to the original author. If the reproduced manuscript involves copyright issues, please contact us immediately for deletion.
Scan the QR code
Register for the corresponding course