Default hyperparamters for `run_atari.py` (using P-DDDQN) fail with Pong and Breakout (log files attached)

[DanielTakeshi]

The default hyperparameters of baselines/baselines/deepq/experiments/run_atari.py, which presumably is the script we should be using for DQN-based models, fail to gain any noticeable reward for both Breakout and Pong. I've attached log files later and the steps to reproduce in this issue; the main reason why I'm filing it is that it probably makes sense to have default hyperparameters be working for the scripts that are provided. Or, alternatively, perhaps list the ones that work somewhere? Upon reading run_atari.py it seems like the number of steps is a bit low and the replay buffer should be 10x larger, but I don't think that's going to fix the issue since Pong should be able to learn quickly with this kind of setup.

I know this is probably not the top priority now but in theory this is easy to fix (just run it with the correct hyperparameters), and it would be great for users since running even 10 million steps (the default value right now) can take over 10 hours on a decent personal workstation. If you're in the process of refactoring this code, is there any chance you can take this feedback into account? Thank you!

Steps to reproduce:

• Use a machine with Ubuntu 16.04.
• I doubt this matters, but I'm also using an NVIDIA Titan X GPU with Pascal.
• Install baselines as of commit 36ee5d1
• I used a Python 3.5 virtual environment with the following packages, with Tensorflow 1.8.0.
• Enter the experiments directory: cd baselines/baselines/deepq/experiments/
• Finally, run python run_atari.py with either PongNoFrameskip-v4 or BreakoutNoFrameskip-v4 as the --env argument. I kept all other parameters their default value, so this was prioritized dueling double DQN.

By default the logger in baselines will create log.txt, progress.csv, and monitor.csv files that contain information about training runs. Here are the Breakout and Pong log files:

breakout_log.txt
pong_log.txt

Since GitHub doesn't upload csv files, here are the monitor.csv files for Breakout and then Pong:

https://www.dropbox.com/s/ibl8lvub2igr9kw/breakout_monitor.csv?dl=0
https://www.dropbox.com/s/yuf3din6yjb2swl/pong_monitor.csv?dl=0

Finally, here are the progress.csv files for Breakout and the for Pong:

https://www.dropbox.com/s/79emijmnsdcjm37/breakout_progress.csv?dl=0
https://www.dropbox.com/s/b817wnlyyyriti9/pong_progress.csv?dl=0

[vpj]

I too got similar results for Breakout, with default parameters.

[DanielTakeshi]

Thanks @vpj .

Not sure if anyone on the team has been able to check this. Hopefully this will be updated in their code refactor, which I think they are doing behind the scenes.

In the meantime I'm using an older version of the code to get DQN-based algorithms to get the published literature results.

[uotter]

Hi @DanielTakeshi, I met the same problems, and can you tell me which version of the code you are using now?

[DanielTakeshi]

@uotter It's a bit unfortunate, I am actually using this old commit

4993286

Because right after that is the one which changed a bunch of image processing stuff.

[uotter]

Thanks @DanielTakeshi , does this version work well and reach the scores in the paper published?

[DanielTakeshi]

Yes, that version works well. I've reproduced publishable scores from all the 20 games I tried.

[andytwigg]

@DanielTakeshi Seeing that commit 4993286 makes me nervous about VecFrameStack used in PPO2, which I've also had trouble with. step_wait contains the following:

self.stackedobs = np.roll(self.stackedobs, shift=-1, axis=-1)
...
self.stackedobs[..., -obs.shape[-1]:] = obs

so I'm wondering if this should be updated to match the changes in that commit?

[meet-cjli]

@DanielTakeshi Ok, I closed the issue. How did you solve this problem？The old version code does not seem to have this problem. Do you know what is the problem？

[DanielTakeshi]

@Sudsakorn123 unfortunately I do not know. I went through the current version of the code (the one I can't get training to work) line by line, and also checked preprocessing of images, but didn't seem to find anything unusual.

[DanielTakeshi]

Oh, I just saw an older issue

#176

Where the users were having some similar issues. Unfortunately it seems like nothing got resolved there.

[skpenn]

Met the same issue and finally solved by this pull request Fix dtype for wrapper observation spaces.
wondering why this pull request is not merged into main branch

[DanielTakeshi]

@skpenn Good news, looks like the pull request you linked is 'effectively merged'!

[Michalos88]

I believe, that the issue has not been solved yet. I tried training the deepq model on breakout and pong with the default hyper parameters and even after 40M time steps the average episode return wouldn't be greater than 0.4. I tired tuning the hyper parameters, but it didn't really help.
I am using similar setup to @DanielTakeshi:

• Ubuntu 16.04 LTS
• NVIDIA Titan X GPU with Pascal
• Python 3.5.2
• baselines as of commit 3900f2a4473ce6b26a8129372ca8d5e02c766c9cl
• In main baselines main folder running: python -m baselines.run --alg=deepq --env=BreakoutNoFrameskip-v4 --num_timesteps=1e7

requirements.txt
monitor.txt

[DanielTakeshi]

@Michalos88 really? That's unfortunate.

For hyperparameters I strongly suggest sticking with defaults here (or with what the DeepMind paper did) since it's too expensive for us to keep tweaking with those. The repository here will eventually, I think, get results standardized for Atari and DQN based models.

I'll run a few trials on my end as well (maybe next week) to see if default DQN parameters can make progress.