[Benchmark] HF Trainer on A100

[stas00]

🖥 Benchmarking transformers w/ HF Trainer on a single A100 40GB

We are going to use a special benchmarking tool that will do all the work for us. #14934

This is the index post and specific benchmarks are in their own posts below:

1. fp16 vs bf16 vs tf32 vs fp32
2. gradient accumulation steps
3. batch size
4. gradient checkpointing
5. optimizers
6. combining winning strategies ~3x speed improvement!
7. RTX-3090 vs A100

Note that each benchmark was run only once, so multiple runs and averaging is probably going to give slightly different results. The purpose here though is to see relative differences roughly and not try to give an exact number.

See also the same benchmarks for RTX-3090

[stas00]

precision: fp16 vs bf16 vs tf32 vs fp32

Main interest: benchmarking the new --bf16 and --tf32 on Ampere/RTX-3090, comparatively to fp16 and fp32 modes.

• bf16 is autocast(dtype=torch.bfloat16)
• tf32 is torch.backends.cuda.matmul.allow_tf32 = True

Benchmark

The benchmark uses 3 different t5 models, and at the end of the section also gpt2. For t5 the main script is:

CUDA_VISIBLE_DEVICES=0 python \
 examples/pytorch/translation/run_translation.py --model_name_or_path t5-small \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 64 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 40000 --dataloader_num_workers 2 

and now adding one of:

--tf32 0 # fp32
--tf32 0 --fp16
--tf32 0 --bf16
--tf32 1
--tf32 1 --fp16
--tf32 1 --bf16

But we are going to use a special benchmarking tool that will do all the work for us. #14934

Important notes:

1. --tf32 0 --fp16 0 combo is just fp32 (which is the default mode - we don't have this option per se)
2. I changed --per_device_train_batch_size in the base command from 32 (t5-small) to 16 (t5-base) to 8 (t5-large) to be able to fit into the GPU memory while keeping it as occupied as possible.
3. I changed --max_train_samples in the base command from 20k (t5-small) to 10k (t5-base) to 5k (t5-large) to give each run about 1-3min of run time so that the benchmark doesn't take too too long, but is long enough to put strain on the card.

*** Setup:

Datetime    : 2022-01-03 22:43:38

Software:
transformers: 4.16.0.dev0
torch       : 1.10.1
cuda        : 11.3
python      : 3.8.12

Hardware:
1 GPUs      : NVIDIA A100-SXM4-40GB, 39.59GB

Benchmark 1: t5-small

Variation	Train samples per second	Diff %	Train loss
--tf32 0	272.59	0	2.49
--tf32 1	581.61	113	2.49
--fp16 --tf32 0	643.07	136	2.49
--fp16 --tf32 1	635.24	133	2.49
--bf16 --tf32 0	616.23	126	2.50
--bf16 --tf32 1	612.59	125	2.50

Conclusions:

• fp16 is 136% faster than fp32
• bf16 is ~4% slower than fp16
• tf32 is 113% faster than fp32, and only ~10% slower than fp16
• tf32 makes ~1% impact on bf16 and fp16 modes

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-small \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 64 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 40000 --dataloader_num_workers 2 ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'|--fp16|--bf16' '--tf32 0|--tf32 1' --report-metric-keys train_loss \
--repeat-times 1 --base-variation '--tf32 0'

Benchmark 2: t5-base

Variation	Train samples per second	Diff %	Train loss
--tf32 0	80.10	0	2.21
--tf32 1	214.10	167	2.21
--fp16 --tf32 0	219.20	174	2.21
--fp16 --tf32 1	218.46	173	2.21
--bf16 --tf32 0	214.17	167	2.22
--bf16 --tf32 1	225.44	181	2.22

Conclusions:

• fp16 is 173% faster than fp32
• bf16 is about the same as fp16
• tf32 is 167% faster than fp32, and is about the same as fp16

CUDA_VISIBLE_DEVICES=0 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 32 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'|--fp16|--bf16' '--tf32 0|--tf32 1' --report-metric-keys train_loss \
--repeat-times 1 --base-variation '--tf32 0'

Benchmark 3: t5-large

Variation	Train samples per second	Diff %	Train loss
--tf32 0	31.59	0	2.03
--tf32 1	36.13	14	2.03
--fp16 --tf32 0	34.86	10	0.00
--fp16 --tf32 1	36.77	16	0.00
--bf16 --tf32 0	31.35	-1	2.04
--bf16 --tf32 1	31.30	-1	2.04

Conclusions:

• fp16 overflows here (loss=0). (this is a very well known issue with many bf16-pretrained models that are being attempted to be finetuned in fp16).
• tf32 is only 14% faster than fp32
• bf16 for some reason performs really badly - same as fp32 (this same benchmark on RTX-3090 doesn't have this problem)

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-large \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 8 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 5000 --dataloader_num_workers 2 ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'|--fp16|--bf16' '--tf32 0|--tf32 1' --report-metric-keys train_loss \
--repeat-times 1 --base-variation '--tf32 0'

If I use a higher bs=16 instead of 8, bf16 does deliver a better than fp32 performance, but still not on par with fp16:

Variation	Train samples per second	Diff %	Train loss
--tf32 0	39.59	0	2.10
--tf32 1	67.24	70	2.10
--fp16 --tf32 0	70.88	79	0.00
--fp16 --tf32 1	70.38	78	0.00
--bf16 --tf32 0	61.37	55	2.12
--bf16 --tf32 1	59.95	51	2.12

It'd be great to know why CUDA doesn't activate some optimization since not everybody is going to run benchmarks, but if you do run benchmarks and find yourself in this situation Eddie Yan proposed that adding --gradient_accumulation_steps to create a much larger batch for the scheduler to step with which should help a lot.

So let's try --per_device_train_batch_size 16 --gradient_accumulation_steps 4 for a total of effective bs=64:

Variation	Train samples per second	Diff %	Train loss
--tf32 0	45.60	0	2.35
--tf32 1	86.78	90	2.36
--fp16 --tf32 0	77.47	70	0.00
--fp16 --tf32 1	79.63	75	0.00
--bf16 --tf32 0	75.85	66	2.41
--bf16 --tf32 1	73.19	61	2.41

Both bf16 and tf32 show a much better performance here.

Benchmark 4: gpt2

Variation	Train samples per second	Diff %	Train loss
--tf32 0	28.77	0	3.34
--tf32 1	63.51	121	3.34
--fp16 --tf32 0	69.60	142	3.34
--fp16 --tf32 1	69.98	143	3.34
--bf16 --tf32 0	70.37	145	3.34
--bf16 --tf32 1	69.88	143	3.34

Conclusions:

• fp16 is ~140% faster than fp32
• bf16 is on par with fp16
• tf32 is 121% faster than fp32, and only ~8% slower than fp16
• tf32 makes almost no impact on bf16 and fp16 modes

*** The benchmark command line was:

CUDA_VISIBLE_DEVICES=0 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/language-modeling/run_clm.py --model_name_or_path gpt2 \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--logging_strategy no --save_strategy no --do_train --max_train_samples 5000 \
--per_device_train_batch_size 16 --num_train_epochs 1 --warmup_steps 8 \
--block_size 512 --report_to none ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'|--fp16|--bf16' '--tf32 0|--tf32 1' --report-metric-keys train_loss \
--repeat-times 1 --base-variation '--tf32 0'

Benchmark 5: gpt2-medium

Variation	Train samples per second	Diff %	Train loss
--tf32 0	10.60	0	2.98
--tf32 1	24.81	134	2.98
--fp16 --tf32 0	27.67	161	2.99
--fp16 --tf32 1	27.62	160	2.99
--bf16 --tf32 0	27.57	160	2.99
--bf16 --tf32 1	27.55	160	2.99

Conclusions:

• fp16 is ~160% faster than fp32
• bf16 is on par with fp16
• tf32 is 134% faster than fp32, and only ~10% slower than fp16
• tf32 makes no impact on bf16 and fp16 modes

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/language-modeling/run_clm.py --model_name_or_path gpt2-medium \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--logging_strategy no --save_strategy no --do_train --max_train_samples 2500 \
--per_device_train_batch_size 8 --num_train_epochs 1 --warmup_steps 8 \
--block_size 512 --report_to none ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'|--fp16|--bf16' '--tf32 0|--tf32 1' --report-metric-keys train_loss \
--repeat-times 1 --base-variation '--tf32 0'

[stas00]

gradient accumulation steps

Let's choose t5-base model to test with as it's pretty large yet doesn't overflow like t5-large.

Let's measure --gradient_accumulation_steps 1,2,4,8,16,32 with different precision configurations.

Variation	Train samples per second	Diff %	Train loss
--gradient_accumulation_steps 1 --tf32 0	93.68	0	2.21
--gradient_accumulation_steps 1 --tf32 1	210.53	125	2.21
--gradient_accumulation_steps 1 --tf32 0 --fp16	217.75	132	2.21
--gradient_accumulation_steps 1 --tf32 0 --bf16	224.09	139	2.22
--gradient_accumulation_steps 2 --tf32 0	97.48	4	2.28
--gradient_accumulation_steps 2 --tf32 1	236.39	152	2.28
--gradient_accumulation_steps 2 --tf32 0 --fp16	244.81	161	2.28
--gradient_accumulation_steps 2 --tf32 0 --bf16	246.08	163	2.29
--gradient_accumulation_steps 4 --tf32 0	99.68	6	2.39
--gradient_accumulation_steps 4 --tf32 1	248.24	165	2.40
--gradient_accumulation_steps 4 --tf32 0 --fp16	259.20	177	2.41
--gradient_accumulation_steps 4 --tf32 0 --bf16	263.39	181	2.42
--gradient_accumulation_steps 8 --tf32 0	100.67	7	2.58
--gradient_accumulation_steps 8 --tf32 1	252.45	169	2.58
--gradient_accumulation_steps 8 --tf32 0 --fp16	261.59	179	2.58
--gradient_accumulation_steps 8 --tf32 0 --bf16	267.37	185	2.62
--gradient_accumulation_steps 16 --tf32 0	100.97	8	2.83
--gradient_accumulation_steps 16 --tf32 1	253.68	171	2.84
--gradient_accumulation_steps 16 --tf32 0 --fp16	256.13	173	2.84
--gradient_accumulation_steps 16 --tf32 0 --bf16	274.14	193	2.89

Let's filter out just one subset so that it's easier to compare the gradient accumulation differences alone, so re-running with just bf16 enabled (--tf32 0 --bf16):

Variation	Train samples per second	Diff %	Train loss
--gradient_accumulation_steps 1	228.41	0	2.22
--gradient_accumulation_steps 2	248.77	9	2.29
--gradient_accumulation_steps 4	263.54	15	2.42
--gradient_accumulation_steps 8	270.12	18	2.62
--gradient_accumulation_steps 16	271.99	19	2.89

Conclusions:

• that's a significant speed up for even 4 steps
• at 16 the impact is almost negligible over 8
• the loss gets much bigger with the higher accumulation steps - my benchmark is very short and with less steps to take when the batches are larger, the model simply doesn't have a chance to step down far enough. The same can be observed with just normal batch size changes.
  non-zero lr warm up too plays a role here since it's a very short run.

1.

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 32 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--gradient_accumulation_steps 1|--gradient_accumulation_steps 2|--gradient_accumulation_steps 4|--gradient_accumulation_steps 8|--gradient_accumulation_steps 16' \
'--tf32 0|--tf32 1|--tf32 0 --fp16|--tf32 0 --bf16' --report-metric-keys \
train_loss --repeat-times 1 

2.

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 32 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 --tf32 0 --bf16 ' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--gradient_accumulation_steps 1|--gradient_accumulation_steps 2|--gradient_accumulation_steps 4|--gradient_accumulation_steps 8|--gradient_accumulation_steps 16' \
--report-metric-keys train_loss --repeat-times 1

[stas00]

batch size

Variation	Train samples per second	Diff %	Train loss
--per_device_train_batch_size 1	7.77	0	1.90
--per_device_train_batch_size 2	15.51	100	2.01
--per_device_train_batch_size 4	29.66	282	2.09
--per_device_train_batch_size 8	61.16	687	2.16
--per_device_train_batch_size 16	115.84	1392	2.25
--per_device_train_batch_size 32	224.96	2797	2.38

Conclusions:

• No surprise here, the speed here is directly proportional to the gpu capacity utilization. In this particular configuration BS=16 is the highest BS we can fit. So when we use BS=1 we greatly underutilize the GPU. The speed up is linear and almost directly proportional to the batch-size.
• as with gradient accumulation steps lm loss gets worse with the increase in the batch size because my benchmark is very short and with less steps to take when the batches are larger, the model simply doesn't have a chance to step down far enough.

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 5000 --dataloader_num_workers 2 --bf16' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--per_device_train_batch_size 1|--per_device_train_batch_size 2|--per_device_train_batch_size 4|--per_device_train_batch_size 8|--per_device_train_batch_size 16|--per_device_train_batch_size 32' \
--report-metric-keys train_loss --repeat-times 1 

[stas00]

gradient checkpointing

Variation	Train samples per second	Diff %	Train loss
--gradient_checkpointing 0	225.67	24	2.30
--gradient_checkpointing 1	182.68	0	2.30

Conclusions:

• as expected since gradient checkpointing recalculates forward activations it should be slower - we get a 24% slowdown here.

Let's look at memory:

Variation	Train samples per second	Diff %	Train loss	Train mem gpu alloc delta	Train mem gpu peaked delta
--gradient_checkpointing 0	122.81	35	2.38	2739MB	1155MB
--gradient_checkpointing 1	90.92	0	2.38	2697MB	3229MB

We can clearly see that peak GPU memory is ~2/3 less.

note: I had to half BS in the 2nd benchmark as I was getting OOM.

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 32 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 10000 --dataloader_num_workers 2 --bf16' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--gradient_checkpointing 0|--gradient_checkpointing 1' --report-metric-keys \
train_loss --repeat-times 1 

CUDA_VISIBLE_DEVICES=3 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --per_device_train_batch_size 32 --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 5000 --dataloader_num_workers 2 --bf16 --skip_memory_metrics 0' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--gradient_checkpointing 0|--gradient_checkpointing 1' --report-metric-keys \
'train_loss train_mem_gpu_alloc_delta train_mem_gpu_peaked_delta' \
--repeat-times 1

[stas00]

optimizers

Let's do fp32 first:

Variation	Train samples per second	Diff %	Train loss
--optim adamw_hf	214.55	2	2.21
--optim adamw_torch	209.72	0	2.21
--optim adafactor	158.56	-24	2.21
--optim adamw_apex_fused	227.96	9	2.21

Observations:

• apex's FusedAdam is the fastest.

fp16:

Variation	Train samples per second	Diff %	Train loss
--optim adamw_hf	221.08	5	2.21
--optim adamw_torch	209.85	0	2.21
--optim adafactor	160.69	-23	2.21
--optim adamw_apex_fused	231.71	10	2.21

bf16:

Variation	Train samples per second	Diff %	Train loss
--optim adamw_hf	221.28	4	2.22
--optim adamw_torch	212.83	0	2.22
--optim adafactor	164.21	-23	2.22
--optim adamw_apex_fused	237.31	12	2.22

Observations:

• The relative speed up is similar

# fp32
CUDA_VISIBLE_DEVICES=3 python scripts/benchmark/trainer-benchmark.py --base-cmd \
' \
examples/pytorch/translation/run_translation.py --model_name_or_path t5-base --output_dir output_dir \
--do_train --label_smoothing 0.1 --logging_strategy no --save_strategy no --per_device_train_batch_size 32 \
--max_source_length 512 --max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: "  --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 \
' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--optim adamw_hf|--optim adamw_torch|--optim adafactor|--optim adamw_apex_fused' \
--report-metric-keys train_loss --base-variation '--optim adamw_torch'

# fp16 - just add --fp16 to base-cmd

# bf16 - just add --bf16 to base-cmd

[stas00]

combining winning strategies

Now let's combine the winning strategies from each individual benchmark above and compare with the baseline:

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	92.15	0	2.21
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --bf16	267.21	190	2.62

Getting an almost 3x improvement in speed!

CUDA_VISIBLE_DEVICES=3 python \
../transformers-stas/scripts/benchmark/trainer-benchmark.py --base-cmd \
' \
examples/pytorch/translation/run_translation.py --model_name_or_path t5-base --output_dir output_dir \
--do_train --label_smoothing 0.1 --logging_strategy no --save_strategy no --per_device_train_batch_size 32 \
--max_source_length 512 --max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: "  --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 \
' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0|--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --bf16' \
--report-metric-keys train_loss --base-variation '--optim adamw_torch'

[stas00]

RTX-3090 vs A100

In all the benchmarks above I was making the batch size bigger and run more samples comparative to the same RTX-3090 benchmarks as A100 40GB card can handle more than RTX-3090 24GB, but let's compare now the 2 using the same config. So we will have RTX-3090 fully loaded, but A100 will be only partially loaded.

Also each card is running on a different machines so there is a bit of hardware difference as well.

A100

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	85.99	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --bf16	153.72	79	2.42

RTX-3090

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	88.94	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --bf16	173.15	95	2.42

Observations:

• This is very unexpected, RTX-3090 appears to be faster when we use only 1/2 of A100's capacity to match the 2.
• as the machines aren't the same it'd be good to find a machine that has both cards and test them on equal hardware.

Same software was used for both setups:

transformers: 4.16.0.dev0
torch       : 1.10.1
cuda        : 11.3
python      : 3.8.12

CUDA_VISIBLE_DEVICES=0 python \
/hf/transformers-trainer-benchmark/scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' \
examples/pytorch/translation/run_translation.py --model_name_or_path t5-base --output_dir output_dir \
--do_train --label_smoothing 0.1 --logging_strategy no --save_strategy no --per_device_train_batch_size 16 \
--max_source_length 512 --max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: "  --warmup_steps 50 \
--max_train_samples 20000 --dataloader_num_workers 2 \
' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0|--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --bf16' \
--report-metric-keys train_loss

I thought that perhaps this had to do with bf16, so I re-did the same with --fp16 instead of --bf16, but the outcome is similar that RTX-3090 appears to be faster on the same benchmark:

A100

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	85.89	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --fp16	144.20	68	2.39

RTX-3090

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	88.95	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 --fp16	168.28	89	2.39

Still not good for A100. Let's try w/o tf32:

A100

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	86.35	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 0 --fp16	156.16	81	2.39

RTX-3090

Variation	Train samples per second	Diff %	Train loss
--optim adamw_torch --gradient_accumulation_steps 1 --tf32 0	88.87	0	2.16
--optim adamw_apex_fused --gradient_accumulation_steps 8 --tf32 0 --fp16	167.36	88	2.39

This is better for A100. So tf32 was making things worse here for some reason.

Eddie Yan explained the reason for RTX-3090 being faster:

When there is underutilization on GPUs of similar architectures, it may come down to clock rates, and 3090 does have faster peak clock rates than A100.