Click the top left corner to follow us
Gift to Readers
In the wonderful world of coding research, we can gain many unique insights. From the perspective of algorithm optimization, it is like carefully polishing a piece of art; each time we streamline the code or improve the algorithm, it is akin to removing impurities, making it more efficient. This inspires us to continuously examine our ways of working in life and at work, seeking optimization opportunities to enhance efficiency. For instance, when dealing with complex data, skillfully utilizing data structures and algorithms can transform originally chaotic information into an orderly format, reminding us to be adept at finding patterns and summarizing methods when facing complex problems.
01/Overview
Introduction:
In the field of machine learning, reinforcement learning is an important method that allows agents to learn how to make optimal decisions to obtain the maximum reward through continuous interaction with the environment and trying different behaviors. The SARSA (State – Action – Reward – State – Action) algorithm, as a classic algorithm in reinforcement learning, plays a key role in many fields such as robot control, gaming, and automated decision-making. It enables agents to gradually explore effective action strategies in complex environments, continuously adapt to changes, and achieve goals.
Algorithm Principle
The core of the SARSA algorithm lies in “online learning,” meaning that the agent continuously updates its strategy with each interaction with the environment. During the action process, the agent remembers its current state, the action taken, the reward received, and the new state entered, while also considering the next action to take in the new state. It then adjusts its value judgments for different actions in each state based on this information. The agent maintains an “action value table” to record the value of taking each action in different states. Initially, the values in this table are set randomly. As the agent continues to interact with the environment, it updates the values in this table based on the rewards received and the conditions of subsequent states. If a certain action yields a good reward, the agent will consider the value of taking that action in the corresponding state to be high, making it more likely to choose that action again in the future; conversely, if an action leads to poor results, the agent will lower the value of that action in the corresponding state.
Algorithm Process
1. Initialization: The agent starts by randomly initializing the values in the action value table and setting some learning parameters, such as learning rate and the importance of future rewards. Then, the agent is placed in the initial state of the environment.
2. Action Selection: The agent selects an action to execute from the available actions in the current state based on the current action value table and a certain strategy (e.g., randomly selecting actions with a certain probability to explore new possibilities; or choosing the action it currently believes has the highest value).
3. Execute Action and Obtain Feedback: After executing the selected action, the environment changes, entering a new state and providing the agent with a corresponding reward. The agent then selects the next action based on the new state and the action value table.
4. Update Action Value Table: The agent updates the action value table based on the current state, the action taken, the reward received, the new state, and the action chosen in the new state, making it more accurately reflect the value of each action.
5. Iterative Loop: The agent takes the new state as the current state and repeats the process of selecting actions, executing actions, obtaining feedback, and updating the action value table until a stopping condition is met, such as reaching a predetermined number of learning iterations or successfully completing a task.
Advantages and Limitations
Advantages:1. Strong Real-time Adaptability: Due to online learning, the SARSA algorithm can adjust its strategy in real-time based on new situations in a constantly changing environment. For example, when a robot explores an unknown environment and encounters obstacles or changes in layout, it can quickly learn new path planning strategies. 2. No Need for Environment Model: Unlike some algorithms that require precise knowledge of the environment model, the SARSA algorithm does not need to know detailed information about the environment in advance, such as the probabilities of state transitions. It can learn directly through interaction with the environment, making it applicable in many complex and hard-to-model environments, such as monitoring tasks in natural ecological settings.3. Balance Between Exploration and Exploitation: Through a reasonable action selection strategy, the SARSA algorithm can find a balance between exploring new actions and utilizing existing experiences. It will try some new actions to discover strategies that may yield higher rewards while also leveraging already learned good actions to ensure a certain degree of stability.
Limitations:1. Poor Scalability of State Space: When the state space of the environment is very large, the action value table can become extremely large, requiring a significant amount of storage space and computational resources. For example, in complex gaming scenarios, there may be millions of states, which can greatly reduce the efficiency of the algorithm. 2. Slow Convergence Speed: In complex environments, the SARSA algorithm may require a large number of attempts and learning to converge to a good strategy. This means it may take a long time to learn, which may not be suitable for scenarios that require quick decision-making.
Improvements and Application Expansion
To overcome these limitations, researchers have proposed many improvement methods. For example, neural networks can be used to replace the action value table, leveraging the strong learning capabilities of neural networks to handle large-scale state spaces, which is the idea behind deep reinforcement learning. Some techniques for accelerating learning can also be employed, such as adjusting learning parameters and introducing experience replay. The SARSA algorithm and its improved versions have wide applications in many fields. In gaming, it can enable agents to learn to play various complex games, such as Go and poker. In industrial automation, it can be used for robot path planning and task scheduling. In finance, it can help investors formulate optimal investment strategies.
02/Running Results
03/Partial Code
clear all;clc;
% n => Size of the maze
n=8;
maze=-50*ones(n,n);
% Randomly Generating Path/Links
for i=1:(n-3)*length(maze)
maze(randi([1,n]),randi([1,n]))=1;
end
% Starting Node
maze(1,1)=1;
% Goal
maze(n,n)=10;
%Plot of the MAZE
figure
matrixPlot(maze)
04/References
[1] Yu Kaize. Joint Optimization of Charging Decisions and Order Assignment for Electric Ride-Hailing Vehicles Based on Reinforcement Learning[D]. University of Electronic Science and Technology of China, 2024. DOI:10.27005/d.cnki.gdzku.2024.005849.
[2] Li Yun. Research on the Generation Method of Test Sequences Based on Sarsa Reinforcement Learning Algorithm[D]. China University of Mining and Technology, 2024.
Some content in this article is sourced from the internet, and references will be noted or cited as references. If there are any inaccuracies, please feel free to contact us for removal.
Scan to contact us!