add ack.

dmahlow · Mar 15, 2023 · 38fade9 · 38fade9
2 parents 1807a44 + 3a50f61
commit 38fade9
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/README.md b/README.md
@@ -166,6 +166,7 @@ torchrun --nproc_per_node=4 --master_port=<your_random_port> train.py \
 ```
 
 Note the given training script is meant to be simple and easy to use, and is not particularly optimized.
+To run on more gpus, you may prefer to turn down `gradient_accumulation_steps` to keep a global batch size of 128. Global batch size has not been tested for optimality.
 
 ### Authors
 All grad students below contributed equally and the order is determined by random draw.