✅ Author Profile: A Matlab simulation developer passionate about research, skilled in data processing, modeling simulation, program design, obtaining complete code, reproducing papers, and scientific simulation.
🍎 Previous reviews, follow the personal homepage:Matlab Research Studio
🍊 Personal motto: Investigate things to gain knowledge, complete Matlab code and simulation consultation content via private message.
🔥 Content Introduction
With the rapid development of 5G/6G communication technology, the Internet of Things, and artificial intelligence, Mobile Edge Computing (MEC) has emerged as a technology that deploys computing, storage, and network resources at the edge of the network (close to terminal devices). It effectively addresses issues such as high data transmission latency and heavy core network load in traditional cloud computing, providing strong technical support for applications with high real-time requirements (such as intelligent transportation, industrial IoT, emergency communication, etc.). Drones, with their flexible deployment, wide coverage, and independence from ground infrastructure, have become ideal carriers for mobile edge computing—leading to the emergence of multi-UAV mobile edge computing systems.
In multi-UAV mobile edge computing systems, drones serve as aerial edge nodes, providing low-latency computing services, data relaying, and communication coverage for ground users. However, this system faces numerous challenges: on one hand, the endurance, computing resources, and communication bandwidth of drones are limited, necessitating reasonable resource allocation to meet the task demands of multiple users; on the other hand, the distribution of users and task requests is dynamic, requiring drones to adjust their positions and paths in real-time while avoiding obstacles and optimizing service quality (such as task processing latency, energy consumption, user satisfaction, etc.). Furthermore, the collaborative cooperation among multiple drones (such as task offloading division, resource sharing, and path conflict avoidance) further increases the complexity of the system.
Traditional optimization methods (such as integer programming and heuristic algorithms) often struggle to adapt to environmental changes when dealing with such dynamic, high-dimensional, and strongly coupled problems, and they have high computational complexity. In contrast, reinforcement learning, through continuous interaction between agents and the environment, learns optimal decision-making strategies, demonstrating strong adaptability and self-learning capabilities, making it particularly suitable for solving dynamic path planning and resource management issues in multi-UAV mobile edge computing. Therefore, research on multi-UAV mobile edge computing and path planning based on reinforcement learning has become a hot topic.
Core Principles of Reinforcement Learning and Adaptability to Multi-UAV Systems
Basic Principles of Reinforcement Learning
Reinforcement learning is a machine learning method that enables agents to learn autonomously through a “trial and error” mechanism. Its core framework includes an agent, environment, state, action, reward, and policy. The agent perceives the current state in the environment, executes actions based on the policy, and the environment provides feedback in the form of rewards based on the actions taken, transitioning to a new state. The agent’s goal is to iteratively learn to find the optimal policy that maximizes cumulative rewards.
In a multi-UAV scenario, each drone can be viewed as an agent, with the environment encompassing the distribution of ground users, task requests, obstacle locations, and the states of other drones; the state can be described by the drone’s position, remaining battery, computing resource utilization, and user task queue; actions include movement direction, speed adjustment, local task processing or offloading to other drones/edge nodes, and resource allocation; the reward function must comprehensively consider task processing latency (lower latency yields higher rewards), energy consumption (lower consumption yields higher rewards), obstacle avoidance (negative rewards for collisions), and user task completion rates (higher completion rates yield higher rewards).
Multi-Agent Reinforcement Learning (MARL) and Collaborative Mechanisms
The multi-UAV system is a typical multi-agent system, where single-agent reinforcement learning is insufficient, necessitating the use of multi-agent reinforcement learning (MARL). In MARL, agents must consider their own interests while also taking into account global optimality to achieve collaborative decision-making.
MARL can be divided into centralized and distributed architectures based on whether agents share information. In a centralized architecture, a central controller collects the states and actions of all agents and learns a unified policy, suitable for small-scale systems but with poor scalability; in a distributed architecture, each agent learns independently, using local observations and limited communication for collaboration, which is more suitable for large-scale multi-UAV systems but must address the “credit assignment” problem (how to fairly distribute global rewards to each agent’s actions).
Common MARL algorithms include multi-agent Q-learning and deep multi-agent reinforcement learning (such as MADDPG, QMIX), etc. For example, MADDPG (Multi-Agent Deep Deterministic Policy Gradient) achieves distributed execution and centralized learning by designing a Critic network (assessing global action value) and an Actor network (generating individual actions) for each agent, effectively addressing the policy coupling problem in multi-agent collaboration.
Key Applications of Reinforcement Learning in Multi-UAV Mobile Edge Computing and Path Planning
Task Offloading and Computing Resource Allocation
One of the core tasks of multi-UAV mobile edge computing is to handle computing tasks from ground users (such as video analysis, data processing). Due to the limited computing resources of drones, load balancing must be achieved through task offloading (transferring some tasks to other drones or ground edge nodes). Reinforcement learning can assist drones in learning when to process locally, when to offload, and to which node to offload.
For instance, the state of the agent includes the current amount of pending tasks, local computing resource utilization, the load status of surrounding drones, and communication latency; actions involve selecting the proportion of task offloading and the target node; the reward function is designed as “- (total task processing latency + offloading energy consumption)”, meaning that lower latency and energy consumption yield higher rewards. Through reinforcement learning, drones can dynamically adjust their offloading strategies based on real-time states, avoiding resource overload or idleness.
In terms of resource allocation, reinforcement learning can optimize the allocation ratios of CPU, memory, and other resources for drones, ensuring that high-priority tasks (such as emergency rescue data processing) are prioritized. For example, deep reinforcement learning (DQN) can model the resource allocation problem as a decision-making problem in a discrete action space, using neural networks to fit the mapping from states to optimal resource allocation strategies.
Path Planning Integrating MEC Requirements
Path planning for multiple UAVs not only needs to avoid obstacles (static obstacles like buildings, dynamic obstacles like other drones) but also needs to deeply integrate with MEC service requirements—the position of drones directly affects the communication quality with users (the closer the distance, the lower the communication latency) and the efficiency of task offloading. Therefore, the goal of path planning is to optimize the distribution of drone positions within the service area while minimizing total travel distance and service latency, under the constraints of mobility (such as maximum flight speed and endurance).
In reinforcement learning-based path planning, the state of the agent includes the current position, remaining battery, known obstacle locations, and hotspots of user task distribution; actions involve the next movement direction and speed; the reward function must comprehensively consider: proximity to high-priority task areas (positive reward), distance from obstacles (positive reward within a safe distance, negative reward if too close), energy consumption (lower consumption yields positive rewards), and task completion progress (positive reward for completed tasks).
For multiple UAVs, path planning must also avoid collisions and resource competition. For example, using multi-agent reinforcement learning (such as QMIX), drones can consider the paths of other drones when making decisions, achieving collaborative path optimization through rewards for “avoiding close distances” and “complementary coverage of service areas,” ensuring service coverage without dead zones and minimal interference between drones.
Dynamic Environment Adaptability Optimization
In real-world scenarios, user task requests (such as sudden increases in tasks in certain areas), obstacles (such as temporary no-fly zones), and network conditions are dynamically changing. The online learning capability of reinforcement learning allows it to quickly adapt to environmental changes—drones can continuously interact with the dynamic environment, updating strategies without needing to retrain models.
For example, when a large number of urgent tasks suddenly appear in a certain area, the state of the agent will reflect the changes in task hotspots in real-time. Through the exploration mechanism of reinforcement learning, drones can autonomously adjust their paths to converge on that area while collaborating on task offloading to avoid overloading a single drone. This dynamic adaptability is unmatched by traditional static path planning methods (such as pre-planned fixed routes).
Typical Cases and Performance Analysis
Emergency Communication and Task Processing Scenarios
In disaster rescue scenarios such as earthquakes, where ground communication infrastructure is damaged, multiple UAVs, as mobile edge nodes, need to quickly establish temporary communication networks and handle image recognition tasks at the rescue site (such as locating trapped individuals). The challenges in this scenario include dynamic task distribution (rescue points change over time), limited drone endurance, and the need for collaborative coverage of large areas.
In a multi-UAV system using multi-agent reinforcement learning (MADDPG), the state of each drone includes its own position, remaining battery, the amount of tasks within its coverage area, and the positions of other drones; actions involve movement direction and task offloading decisions; the reward function is “(number of successfully processed tasks × weight) – (flight energy consumption + communication latency) – (collision penalty).”
Experimental results show that compared to traditional “random path + local processing” strategies, the reinforcement learning-based system reduces task processing latency by over 40%, increases task completion rates by 30%, and improves drone endurance utilization efficiency by 25% (due to more optimal paths and reduced ineffective flights).
Urban Logistics and Edge Service Scenarios
In urban logistics, multiple UAVs need to deliver goods while providing edge computing services (such as real-time navigation data processing) to ground users. This scenario features dense static obstacles (buildings) and dynamic obstacles (vehicles, pedestrians), with user tasks being random.
In a path planning and task offloading collaborative strategy using deep reinforcement learning (PPO algorithm), drones perceive the three-dimensional urban environment and user distribution through neural networks, dynamically adjusting flight heights (to avoid tall buildings) and paths, and deciding whether to process locally or offload to ground MEC base stations based on the type of user tasks (such as lightweight navigation computing or heavy video rendering).
Compared to static path planning based on the grey wolf optimization algorithm, the reinforcement learning method reduces average service latency by 28% in dynamic task scenarios, as it can respond in real-time to task hotspot migrations; simultaneously, the collision rate between drones drops to below 0.5% (about 3% for traditional methods), reflecting the advantages of multi-agent collaboration.
Challenges and Future Prospects
Current Challenges
- Explosion of State and Action Spaces: In multi-UAV systems, the state includes information from all drones and users, resulting in extremely high dimensions; the action space grows exponentially with the number of drones, leading to slow convergence rates and high computational costs for reinforcement learning.
- Multi-Agent Credit Assignment Problem: Global rewards are difficult to accurately assign to each drone’s actions, which may lead to some agents “free-riding” (receiving high rewards without contributing), affecting collaboration efficiency.
- Insufficient Robustness to Dynamic Environments: When environmental changes exceed the training range (such as sudden large-scale tasks), reinforcement learning strategies may fail, and generalization capabilities need improvement.
- Communication and Privacy Constraints: Multi-UAV collaboration relies on information sharing, but in practice, communication bandwidth is limited, and user data (such as task content) privacy protection must be considered in reinforcement learning.
Future Research Directions
- Lightweight Reinforcement Learning Models: Combining model compression and knowledge distillation techniques to reduce neural network complexity, improve real-time performance, and adapt to the limited computing resources of drones.
- Federated Reinforcement Learning Integration: Through a federated learning framework, multiple UAVs can train models locally, sharing only parameter updates without transmitting raw data, resolving the conflict between privacy protection and collaborative learning.
- Hybrid Intelligent Optimization Strategies: Combining reinforcement learning with traditional optimization algorithms (such as the previously mentioned grey wolf optimization algorithm)—reinforcement learning handles dynamic decisions, while optimization algorithms manage static path planning, enhancing overall performance.
- Explainable Reinforcement Learning: Enhancing the interpretability of reinforcement learning strategies to facilitate understanding and debugging by engineers, increasing reliability in critical scenarios (such as medical rescue).
Conclusion
Multi-UAV mobile edge computing and path planning based on reinforcement learning effectively address the challenges of dynamic environments, limited resources, and multi-objective optimization through autonomous learning and collaborative decision-making of agents, providing efficient solutions for emergency communication, urban services, and other scenarios. Although there are currently issues such as state space explosion and credit assignment, with the integration of lightweight models and federated learning technologies, this field will develop towards being smarter, more reliable, and more practical, promoting the large-scale application of multi-UAV systems in mobile edge computing.
⛳️ Operational Results
🔗 References
[1] Xu Shaoyi, Yang Lei. Trajectory Design of Multi-UAV Assisted Mobile Edge Computing Based on Multi-Agent Deep Reinforcement Learning [J]. Journal of Beijing Jiaotong University, 2024, 48(5):1-9.
[2] Communication Engineering (including broadband networks, mobile communications, etc.). Research on Resource Optimization of UAV Mobile Edge Computing Systems Based on Deep Reinforcement Learning [D]. 2024.
[3] Lin Feng. Research on Resource Optimization of UAV Mobile Edge Computing Systems Based on Deep Reinforcement Learning [D]. Nanchang University, 2024.
📣 Partial Code
🎈 Some theoretical references are from network literature; if there is any infringement, please contact the author for deletion.
👇 Follow me to receive a wealth of Matlab e-books and mathematical modeling materials
🏆 The team specializes in guiding customized MATLAB simulations in various research fields, helping to realize research dreams:
🌈 Various intelligent optimization algorithm improvements and applications
Production scheduling, economic scheduling, assembly line scheduling, charging optimization, workshop scheduling, departure optimization, reservoir scheduling, three-dimensional packing, logistics site selection, cargo position optimization, bus scheduling optimization, charging pile layout optimization, workshop layout optimization, container ship loading optimization, pump combination optimization, medical resource allocation optimization, facility layout optimization, visual domain base station and UAV site selection optimization, knapsack problem, wind farm layout, time slot allocation optimization, optimal distributed generation unit allocation, multi-stage pipeline maintenance, factory-center-demand point three-level site selection problem, emergency supply distribution center site selection, base station site selection, road lamp post arrangement, hub node deployment, transmission line typhoon monitoring devices, container scheduling, unit optimization, investment optimization portfolio, cloud server combination optimization, antenna linear array distribution optimization, CVRP problem, VRPPD problem, multi-center VRP problem, multi-layer network VRP problem, multi-center multi-vehicle VRP problem, dynamic VRP problem, two-layer vehicle routing problem (2E-VRP), electric vehicle routing problem (EVRP), hybrid vehicle routing problem, mixed flow shop problem, order splitting scheduling problem, bus scheduling optimization problem, flight shuttle vehicle scheduling problem, site selection path planning problem, port scheduling, port shore bridge scheduling, parking space allocation, airport flight scheduling, leak source localization
🌈 Time series, regression, classification, clustering, and dimensionality reduction in machine learning and deep learning
2.1 BP time series, regression prediction, and classification
2.2 ENS voice neural network time series, regression prediction, and classification
2.3 SVM/CNN-SVM/LSSVM/RVM support vector machine series time series, regression prediction, and classification
2.4 CNN|TCN|GCN convolutional neural network series time series, regression prediction, and classification
2.5 ELM/KELM/RELM/DELM extreme learning machine series time series, regression prediction, and classification
2.6 GRU/Bi-GRU/CNN-GRU/CNN-BiGRU gated neural network time series, regression prediction, and classification
2.7 Elman recurrent neural network time series, regression prediction, and classification
2.8 LSTM/BiLSTM/CNN-LSTM/CNN-BiLSTM long short-term memory neural network series time series, regression prediction, and classification
2.9 RBF radial basis function neural network time series, regression prediction, and classification
2.10 DBN deep belief network time series, regression prediction, and classification
2.11 FNN fuzzy neural network time series, regression prediction
2.12 RF random forest time series, regression prediction, and classification
2.13 BLS broad learning time series, regression prediction, and classification
2.14 PNN pulse neural network classification
2.15 Fuzzy wavelet neural network prediction and classification
2.16 Time series, regression prediction, and classification
2.17 Time series, regression prediction, and classification
2.18 XGBOOST ensemble learning time series, regression prediction, and classification
2.19 Transform various combinations of time series, regression prediction, and classification
Directions cover wind power prediction, photovoltaic prediction, battery life prediction, radiation source identification, traffic flow prediction, load forecasting, stock price prediction, PM2.5 concentration prediction, battery health status prediction, electricity consumption prediction, water body optical parameter inversion, NLOS signal identification, precise prediction of subway stops, transformer fault diagnosis
🌈 In image processing
Image recognition, image segmentation, image detection, image hiding, image registration, image stitching, image fusion, image enhancement, image compressed sensing
🌈 In path planning
Traveling salesman problem (TSP), vehicle routing problem (VRP, MVRP, CVRP, VRPTW, etc.), UAV three-dimensional path planning, UAV collaboration, UAV formation, robot path planning, grid map path planning, multimodal transport problems, electric vehicle routing problem (EVRP), two-layer vehicle routing problem (2E-VRP), hybrid vehicle routing problem, ship trajectory planning, full path planning, warehouse patrol
🌈 In UAV applications
UAV path planning, UAV control, UAV formation, UAV collaboration, UAV task allocation, UAV secure communication trajectory online optimization, vehicle collaborative UAV path planning
🌈 In communication
Sensor deployment optimization, communication protocol optimization, routing optimization, target localization optimization, Dv-Hop localization optimization, Leach protocol optimization, WSN coverage optimization, multicast optimization, RSSI localization optimization, underwater communication, communication upload and download allocation
🌈 In signal processing
Signal recognition, signal encryption, signal denoising, signal enhancement, radar signal processing, signal watermark embedding and extraction, electromyography signals, electroencephalography signals, signal timing optimization, electrocardiogram signals, DOA estimation, encoding and decoding, variational mode decomposition, pipeline leakage, filters, digital signal processing + transmission + analysis + denoising, digital signal modulation, bit error rate, signal estimation, DTMF, signal detection
🌈 In power systems
Microgrid optimization, reactive power optimization, distribution network reconstruction, energy storage configuration, orderly charging, MPPT optimization, household electricity
🌈 In cellular automata
Traffic flow, crowd evacuation, virus spread, crystal growth, metal corrosion
🌈 In radar
Kalman filter tracking, trajectory association, trajectory fusion, SOC estimation, array optimization, NLOS identification
🌈 In workshop scheduling
Zero-wait flow shop scheduling problem (NWFSP), Permutation flow shop scheduling problem (PFSP), Hybrid flow shop scheduling problem (HFSP), zero idle flow shop scheduling problem (NIFSP), distributed permutation flow shop scheduling problem (DPFSP), blocking flow shop scheduling problem (BFSP)
👇