[Bug] action_noise never got used in HER

[ShangqunYu]

🐛 Bug

Hi when I was trying to use HER with DDPG, I noticed that the action noise I feed into HER never got used by DDPG.
Please correct me if I am wrong, if you take a look at the code on 197 at HER.py

            rollout = self.collect_rollouts(
                self.env,
                n_episodes=self.n_episodes_rollout,
                n_steps=self.train_freq,
                action_noise=self.action_noise,
                callback=callback,
                learning_starts=self.learning_starts,
                log_interval=log_interval,
            )

I have printed out self.action_noise, it is NONE. the action_noise got fed into self.model during the init part (line 103 as kwargs), but I think it never got assigned to it self as an attribute, maybe we shall do self.model.action_noise there? Let me know if I am mistaken.

[ShangqunYu]

I think I just reproduce the problem by copying the code from one of your google colabs stable_baselines_HER.ipynb.
`env = gym.make("parking-v0")
n_actions = env.action_space.shape[0]
noise_std = 0.2
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=noise_std * np.ones(n_actions))

model = HER('MlpPolicy', env, DDPG, n_sampled_goal=4,
goal_selection_strategy='future', online_sampling=True,
verbose=1, buffer_size=int(1e6),
learning_rate=1e-3, action_noise=action_noise,
gamma=0.95, batch_size=256,
policy_kwargs=dict(net_arch=[256, 256, 256]), max_episode_length=100)

model.learn(int(2e5))

model.save('her_ddpg_highway')`
and in her.py, before line 197, check the value of self.action_noise, I think you will get a None.
Please let me know if you need more information.

[araffin]

Hello,
thanks for reporting that bug.
The issue comes from the fact that self.action_noise is defined in the base class, so it does not use the model action noise.

[araffin]

I pushed the fix here ;)
01c5aa1
(should be merged soon)

[ShangqunYu]

Got it, thank you araffin :)