auto_find_batch_size=True and eval_steps=ratio unexpected behavior

[edmcman]

System Info

• transformers version: 4.30.1
• Platform: Linux-5.7.19-050719-generic-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.15.1
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I don't have a full example that I can share, but I think this is a simple enough problem that one may not be needed.

I am using TrainingArguments(auto_find_batch_size=True, eval_steps=0.1, per_device_train_size=1024). With batch size of 1024, I have 657 steps. The eval ratio appears to be evaluated on this, with evaluation happening every 66 steps.

However, the automatic batch size adjusts to 16, and a corresponding 83787 steps. But the evaluation is still performed every 66 steps.

Expected behavior

I expected the eval steps to be recomputed when the batch size updated. In the example above, I expected evaluation to occur every ~8000 steps.

[sgugger]

cc @muellerzr

[muellerzr]

Any chance you could provide a minimal reproducer I can test with?

Otherwise please try installing via pip install git+https://github.com/huggingface/transformers@muellerzr-ratio to see if that fixes it? 🙏

[edmcman]

Let me try your patch first.

[edmcman]

With the patch, still evaling every 66 steps. Let me try to make a reproducer. It probably won't be minimal though...

[edmcman]

notebook.zip

[edmcman]

Looks like max_steps is not being updated

[edmcman]

Very strange. Here is some debug output:

Currently training with a batch size of: 8
The following columns in the training set don't have a corresponding argument in `RobertaForSequenceClassification.forward` and have been ignored: Addr, Binary, Name, text. If Addr, Binary, Name, text are not expected by `RobertaForSequenceClassification.forward`,  you can safely ignore this message.
***** Running training *****
  Num examples = 223,431
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83,787
  Number of trainable parameters = 83,452,418

Total optimization steps is printing max_steps... 😕

[edmcman]

I see the problem I think:

        if args.eval_steps and args.eval_steps < 1:
            args.eval_steps = math.ceil(max_steps * args.eval_steps)

Since this actually modifies args.eval_steps, the ratio will be lost the first time we run this code. E.g., this will set args.eval_steps to 66 and lose 0.1.

[muellerzr]

Okay, I think it should be fixed now. Can you try again via the same branch?

[edmcman]

Still eval'ing at 66 :-(

[edmcman]

I did upload the notebook as a .zip above, but I'm trying to put it on colab to make things easier.

[edmcman]

I can't run it on colab because I'm out of free GPU usage, but I did upload it, and I think it should work if you have GPU access there:

https://colab.research.google.com/drive/1A-MzFHIbWtrtO4tjf2GROAdfAueEHidw?usp=sharing

[muellerzr]

re; Total optimization steps is printing max_steps... 😕, yes we don't perform gradient accumulation with this, so if you happen to get small enough that max steps < steps w/ reduction multiplier, that does make sense.

Looking into this still. Thanks for the reproducer

[muellerzr]

Thanks again, I'll need to run this in the AM to verify but I believe I've fixed this now by storing the steps away in a data struct before we loop over again: https://github.com/huggingface/transformers/compare/muellerzr-ratio?expand=1

Once verified I'll place a PR in

[edmcman]

I'm sorry to report that I still think it is broken!