[rllib]Misleading use of sgd_batch_size & train_batch_size in multi_gpu_optimizer

[praveen-palanisamy]

Describe the problem

The use of sgd_batch_size and train_batch_size in multi_gpu_optimizer.py is misleading. As per the discussion, the intended use is indeed to do a number of epochs with minibatchs (M) sampled from the training batch within each outer-loop iteration (Currently used only by the Actor-Critic style PPO implementation)

The minibatch size (through the sgd_minibatch_size config parameter) is supposed to be << train_batch_size but, in the current implementation of localMultiGPUOptimizer, the train_batch_size is more or less ignored by the optimizer interface (?) and the sgd_minibatch_size is used as the batch_size to perform num_sgd_iter number of SGD updates.

Minimal example to illustrate the misuse:

python ray/rllib/train.py --env=PongDeterministic-v4 --run=PPO --config '{"num_workers":2, "sample_batch_size":2, "sgd_minibatch_size":16, "train_batch_size":4, "num_gpus":2}

Note that the "sgd_minibatch_size":16 is >> "train_batch_size":4 but RLlib trains it with no complains and the training batch size used in an iteration is not 64. This makes it difficult to compare performance.

[ericl]

I don't think train_batch_size is ignored right?

ray/python/ray/rllib/optimizers/multi_gpu_optimizer.py

Line 102 in 995ac24

self.train_batch_size)

I guess in your example, what's going on is that the minibatch size is rounded down to 4. So it becomes non-minibatch SGD. Is that the concern?

[praveen-palanisamy]

Hmmm. The capping of sgd_batch_size to 4 in the above example should be fine. Not a concern.

So, this dig started because the number of samples used to train in one outer loop iter by PPO was not matching the intended values. I think I pre-maturely settled on the suspicion that it is due to the above described mismatch. It actually turns out to be due to the batch mode ( especially with complete_episodes).

As I understand, in the complete_episodes batch mode, the minibatch-sgd/train-batch size configs will not have any effect & the number of steps used in one training iter totally depends on the env & its dynamics (hopefully they have timeouts). right?

[praveen-palanisamy]

Is this not a concern in general? or is there a better way?
Because, if there are a number of policy evaluators with several env instances per evaluator/worker and if each of the rollout it going to have infinite horizon (complete episodes), then the actual batch size used in one SGD step may be much higher & probably may result in GPU OOM.

[ericl]

I agree the default behavior of complete_episodes probably isn't great. We could potentially default to truncate_episodes instead, to make the batch size fixed.

[ericl]

@praveen-palanisamy let me know what you think of #2967

[praveen-palanisamy]

The change to the default looks fine & will help in some cases to avoid unexpected behaviors by default. Thanks!

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[praveen-palanisamy]

Closing the issue since enough clarify was brought in by #2967 way back in time.