Experimental environment questions for each model

[1st-hoan]

Hello. Thank you for sharing your amazing research.
I have a question about train environment.

1. I am currently training with 50 epochs using gradient accumulation step 4 in an environment equipped with two RTX 4090 GPUs. However, I am unable to reproduce the performance as shown in the attached log.txt file for DeiT small < DeiT base. Which part of the environment is wrong?
2. For this reason, I would like to ask about the epoch, learning rate, batch size per GPU, and number of GPUs used for each model.

The command is as follows:

python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py \
  --model deit_small_patch16_224 \
  --teacher_model deit_base \
  --epochs 50 \
  --batch-size 128 \
  --data-path /Dataset_classification/imagenet/ \
  --distillation-type soft \
  --distillation-alpha 0.5 \
  --distillation-tau 1 \
  --input-size 224 \
  --maskedkd --len_num_keep 98 \
  --accum-steps 4 \
  --output_dir /MaskedKD/work_dir/2_node_accumul4/

log.txt is as follows:
log.txt

Thank you!

[effl-lab]

Sorry for the very late reply. The first author has graduated, and I think he might not have been received the notification.
Tagging the lead author (@seungwoo-s) to ping him for the reponse.

[seungwoo-s]

@effl-lab
Thanks for the ping!

---

@1st-hoan
Sorry for the late reply,

Just to clarify, does "DeiT small < DeiT base" mean a DeiT-Small student distilled from a DeiT-Base teacher? if so, here’s what we used:

• gpus: 8
• epochs: 300
• batch size per gpu: 128 (total effective batch size: 1024)
• others: default DeiT settings

your effective batch size (2 GPUs × 4 accum × 128 = 1024) matches ours, but we trained for 300 epochs, not 50. Note that, since we didn’t use accumulation techniques, there might be performance gap from it.