A Submodule in Rainbow DQN: Multi-Step Learning

1. What is a “Merged Experience Tuple”?

A single-step experience tuple is the data generated from each interaction between the agent and the environment(s, a, r, s', terminal, done), while a “merged experience tuple” combines N consecutive single-step experiences into a “multi-step experience tuple”. The purpose is to use the “N-step cumulative reward” instead of the single-step reward, making the value estimation more accurate (reducing bias).

The format of the merged multi-step experience tuple is:(s_0, a_0, R, s_n, terminal_n) where:

s_0, a_0: The state and action at step 0 (initial state and action);
R: N-step cumulative reward (total reward after discount summation);
s_n, terminal_n: The next state and termination flag at step N (final state and termination flag).

2. The Specific Logic of Merging

1. Retrieve the Initial State and Action

Extract s_0 (initial state) and a_0 (initial action) from the cached data of step 0.

2. Calculate the N-step Cumulative Reward R:

From step 0 to step N-1, accumulate the rewards using the discount factor gamma, with the formula:

where d_i is the done flag at step i (if d_i=1 at any step, it indicates that the step has terminated, and subsequent rewards will not be accumulated).

3. Retrieve the Final State and Termination Flag:

If there is no termination within N steps (all d_i=0), take the s’ (i.e., s_n) and terminal (terminal_n) from step N-1.

If at any intermediate step d_i=1 (premature termination), take that step’s s’ and terminal as the final state.

3. Timing of Merging

The trigger for the merging operation is:

When the number of cached single-step experiences reaches N steps.

The specific process is as follows:

Cache Single-Step Experiences:

Each single-step experience generated from interactions (s, a, r, s', terminal, done) will first be stored in n_steps_deque (a double-ended queue with a fixed length of n_steps).

Check if N Steps are Full:

When the number of experiences cached in n_steps_deque equals n_steps (i.e., N consecutive steps of data have been collected), it triggers the merging.

Execute Merging and Store:

Call get_n_steps_transition to generate the multi-step experience tuple, store it in the experience pool buffer, and record the priority in sum_tree.

n_steps is the value of N, which is a manually defined hyperparameter that needs to be adjusted based on the task characteristics to balance bias and variance for optimal performance.

Practical tuning suggestions in reinforcement learning tasks indicate that the common range for n_steps is 3 to 10 (for example, in the Rainbow paper, n_steps=3 was used in Atari games). The specific choice should be validated through experiments:

For tasks with low environmental randomness and sparse rewards, n_steps can be increased (e.g., 5 to 10);
For tasks with high environmental randomness and dense rewards, it is recommended to use a smaller n_steps (e.g., 3 to 5) to avoid excessive variance affecting training stability.