Alternative handling of max steps in environment

[oysteinsolheim]

Is it possible to have an alternative way of handling max_steps in continuing environments? As of now the terminal field is set to 'true' when the environment reaches the max_steps even though it's state is random and not a terminal one. In practice it probably doesn't matter much but as it is now the update of the value function is incorrect when reaching the max_steps, I think. Agree?

[findmyway]

Could you elaborate on why "... the update of the value function is incorrect when reaching the max_steps...?

[oysteinsolheim]

I became aware of it when i was reading https://deepreinforcementlearningbook.org/

In chapter 18 they write:
The “done” signal is commonly applied in episodic
reinforcement learning when trying to finish the episodes, and it is a function of the state of the environment, set to be true whenever the terminal state is reached by the agent. Note that here the terminal state is defined to be a state indicating the agent has finished the episode, either in success or failure cases, rather than an arbitrary state when the time limit or maximal episode length is reached. It is not trivial to require that the “done” signal is set to be true only when the state is a terminal state. For example, if the task is to manipulate the robot arm to reach a specific point in the space, the “done” signal is supposed to be true only when the robot has indeed reached the point, but not for the cases when a default maximal length of episode is reached. To understand the differences, we need to know that in reinforcement learning some environments are infinite and some are finite. But in the sampling process, the algorithms usually handle the trajectories in a finite length. Two common ways of achieving that are either setting a maximal episode length or using the “done” signals as feedback from the environment to finish the episode with a break in loops. When applying the “done” signal as a breakpoint in sampling, it should not be set to be true when the episode is finished due to the accomplishment of the maximal length, but only when the terminal state is reached. If the robot finishes its trajectory somewhere else rather than the target point with the done signal set to be true, it may negatively affect the learning process. Specifically, take the example of PPO algorithm, cumulative rewards starting from states St are used to estimate the value of states V (St), while a terminal state has the value zero. If the “done” signal is true for a non-terminal state, the state value is set to be zero although it should not be, it will confuse the value network when estimating its previous states and therefore hinder the learning process.

[findmyway]

I see. Thanks for the link.

I don't think there's an easy way to do it. In current implemntation, most algorithms cannot distinguish the difference above. To support it, all the update!(::AbstractPolicy, ::AbstractTrajectory) need to be changed to accept an extra flag argument. And the inner discounted reward calculation should be updated correspondingly to take the last state into account.

[findmyway]

Personally I think one should avoid to define environments like this. But as you may have noticed that the CartPoleEnv is defined just like the one you quoted 🤦🏻.

[oysteinsolheim]

Thanks for the answer!

[oysteinsolheim]

After thinking about this for a while, wouldn't this issue simply be solved by resetting the environment every max_steps via a DoEveryNStep(max_step) hook? And skip the max_steps variable of the environment altogether?

[findmyway]

I'm afraid not. Let's take one step Q-learning for example:

Usually we will remove the $\gamma * \max Q(s_{t+1}, a)$ part if game terminates at step $t$. However, if the game is not terminated intrinsically but only due to reaching the max_steps, then we shouldn't remove that part.

To distinguish these two cases, we need to add a trait (IsInfiniteHorizon). So that when we reach a terminal state, we know whether we need to remove the estimation of the terminal state or not.

[oysteinsolheim]

Thank for the answer. I don't quite get it. Even though i remove the max_steps, I still do an intrinsic termination if it reaches a terminal state (which is not due to reaching max steps). So in my case the environment either reaches a terminal state before max_steps, in which case we remove the \gamma-max-part, or it is reset at max_steps externally and then the \gamma-max-part is kept.

[findmyway]

Yeah, I understand the trick here. However, we still need to distinguish these two kinds of terminations. This is kind of difficult to explain. But I'll try to make myself clear here. The main confusion lies in the experience replay buffer. (This reminds me of the discussion with @jbrea a long time ago. How time flies! ;)

First we need to understand how the experience replay buffer (ERB) is updated currently.

Suppose the game terminates at step $T$. We have the ERB like this:

$$s^i_0, a^i_0, r^i_0, t^i_0, ..., s^i_T, a^i_T, r^i_T, t^i_T, s^i_{T+1}, a^i_{T+1}$$

Here we only consider one-step Q-learning for now, but the idea is the same with N-step Q-learnig or PPO. Note that $s^i_{T+1}$ usually has no specific meaning and $a^i_{T+1}$ is a dummy action. If the last step of the episode $i$ is chosen as training data, then $\gamma * Q$ part is set to $0$. After the updating step, we will pop out $s^i_{T+1},a^i_{T+1}$ from the ERB and push $s^{i+1}_0, a^{i+1}_0$ into it. And now we have

$$s^i_0, a^i_0, r^i_0, t^i_0, ..., s^i_T, a^i_T, r^i_T, t^i_T, s^{i+1}_0, a^{i+1}_0$$

So even if the $T$-th step is chosen as a training data, we have $t^i_T == true$. So that we can use this flag to ignore the next state $s\prime$ after $s^i_T$, no matter $s\prime == s^i_{T+1}$ or $s\prime == s^{i+1}_0$. But if the game is of Infinite Horizon, then we need to distinguish that difference and the ERB can not be of the compact SARTSA format anymore. We need to manually split it into multiple episodes as [SAR, ... SAR, SA], ... (Here we don't need the terminal field anymore) And here we still need to distinguish which kind of environment we are dealing with. (Afterall, the ERB of finite-horizon games can also have this kind of representation)

[oysteinsolheim]

Ah, I see. So when I reset the the environment without having the terminal set to true, it ruins the chronology of the replay buffer. Makes sense! Thanks a lot for your explanation!

[findmyway]

In our current design, there's really little we can improve here. So I'll close this issue for now. But I may revisit it in the future.