Token label train test on CIFAR10

[leondgarse]

• This is a very basic test, and currently not working as well as expected.

• Token label is implementation of Github zihangJiang/TokenLabeling, paper PDF 2104.10858 All Tokens Matter: Token Labeling for Training Better Vision Transformers.

• First of all, train a model on CIFAR10 using EfficientNetV2B0. Detail explanation in EfficientNetV2B0 transfer learning on cifar10 testing freezing backbone #55.

  CUDA_VISIBLE_DEVICES='1' TF_XLA_FLAGS="--tf_xla_auto_jit=2" ./train_script.py -m efficientnet.EfficientNetV2B0 \
  --data_name cifar10 --lr_decay_steps 20 --lr_cooldown_steps 2 --batch_size 256 --pretrained imagenet \
  --freeze_backbone --summary --seed 0 -s freeze_backbone_test && \
  CUDA_VISIBLE_DEVICES='1' TF_XLA_FLAGS="--tf_xla_auto_jit=2" ./train_script.py -m efficientnet.EfficientNetV2B0 \
  --data_name cifar10 --lr_decay_steps 20 --lr_cooldown_steps 2 --batch_size 256 --epochs 64 \
  --pretrained checkpoints/freeze_backbone_test_latest.h5 --initial_epoch 22 --seed 0 -s freeze_backbone_test

  Best validation accuracy is 0.9792.

• Create token label file using trained model in the first step.

  from keras_cv_attention_models.imagenet import token_label
  token_label.build_token_label_file('cifar10', 'checkpoints/freeze_backbone_test_latest.h5')
  # >>>> Saved to: cifar10_efficientnet_v2-b0_5_5_5.pkl

• Training with --token_label_file. Other parameters are same with first step:

  CUDA_VISIBLE_DEVICES='1' TF_XLA_FLAGS="--tf_xla_auto_jit=2" ./train_script.py -m efficientnet.EfficientNetV2B0 \
  --data_name cifar10 --lr_decay_steps 20 --lr_cooldown_steps 2 --batch_size 256 --pretrained imagenet \
  --freeze_backbone --summary --seed 0 --token_label_file cifar10_efficientnet_v2-b0_5_5_5.pkl \
  -s freeze_backbone_test_TL && \
  CUDA_VISIBLE_DEVICES='1' TF_XLA_FLAGS="--tf_xla_auto_jit=2" ./train_script.py -m efficientnet.EfficientNetV2B0 \
  --data_name cifar10 --lr_decay_steps 20 --lr_cooldown_steps 2 --batch_size 256 --epochs 64 \
  --pretrained checkpoints/freeze_backbone_test_TL_latest.h5 --initial_epoch 22 \
  --seed 0 --token_label_file cifar10_efficientnet_v2-b0_5_5_5.pkl -s freeze_backbone_test_TL

• Results

  token_label_file	Train acc, loss	Best val_acc, val_loss	Epoch 64 val_acc
  None	0.72766, 0.1572	Epoch 55, 0.9757, 0.01697	0.9749
  cifar10_efficientnet_v2-b0_5_5_5.pkl	0.78688, 0.1396	Epoch 56, 0.9792, 0.01478	0.9776