[rllib] IMPALA: FloatingPointError: invalid value encountered in double_scalars

[devinbarry]

What is the problem?

IMPALA metrics makes assumption about items not being empty in the WindowStat class. If it is in fact empty, we get an exception rather that nan

/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
Traceback (most recent call last):
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/axion/git/jengu/jengu/train/impala_train.py", line 43, in <module>
    result = agent.train()
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 505, in train
    raise e
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 491, in train
    result = Trainable.train(self)
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/tune/trainable.py", line 261, in train
    result = self._train()
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 161, in _train
    res = self.collect_metrics()
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 899, in collect_metrics
    selected_workers=selected_workers)
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/optimizers/policy_optimizer.py", line 113, in collect_metrics
    res.update(info=self.stats())
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/optimizers/async_samples_optimizer.py", line 166, in stats
    stats["learner_queue"] = self.learner.learner_queue_size.stats()
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/ray/rllib/utils/window_stat.py", line 25, in stats
    self.name + "_mean": float(np.mean(self.items[:self.count])),
  File "<__array_function__ internals>", line 6, in mean
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 3257, in mean
    out=out, **kwargs)
  File "/home/axion/anaconda3/envs/trading/lib/python3.7/site-packages/numpy/core/_methods.py", line 161, in _mean
    ret = ret.dtype.type(ret / rcount)
FloatingPointError: invalid value encountered in double_scalars

Ray version and other system information (Python version, TensorFlow version, OS):

Python 3.7
All versions of Ray up to the latest nightlies
Tensorflow 2.1.0

Reproduction (REQUIRED)

I don't have a reproduction for this because the environment that triggers it is custom. Its pretty obvious what is going on though. It just needs the same treatment as the other metrics bugs of this class had here 797e6cf

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[09wakharet]

The commit #9213 should have fixed this! Let us know if you still have any issues.