Hindsight Experience Replay

Implementation of standard off-policy method in the bit-flipping environment augmented with HER.

The intention is trying to reproduce the original paper results. Details of the environment and the technique can be found there.

Different models were trained for each bit count. The success rate is the percent of episodes where the algorithms makes it from the start state to the goal.

The policy is unable to succeed for larger values of N because it doesn't have any sample with reward different from -1. This problem is fixed with HER improvement.

Algorithm

N := Number of bits
B := Buffer experience replay

Initialize Critic and Actor Neural Networks

For epoch = 1..EPOCHS:
    For episodes = 1..EPISODES:
        g := Sample random goal status
        st_0 := Sample initial status
        s_0 := (st_0, g) # Initial state

        For t = 0..N-1:
            a_t = Sample action using epsilon greedy policy according to actor policy
            s_{t+1}, r_t = Observe state and reward after action a_t on state s_t

        For t = 0..N-1:
            Add to B experience (s_t, a_t, r_t, s_{t+1})

            Sample status positions in current episode st_r:
                Add to B this experience with different goal and updated reward

    For t=1..TRAIN:
        MB := Sample a minibatch from B
        Perform an optimization step on critic policy using MB

    If K epoch has passed since last actor update:
        Update actor weights with critic weights