Rethinking Pre-Training and Self-Training

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, Quoc V. Le [arXiv]

We release the checkpoints of teacher model and student model in rethinking pre-training and self-training.

Checkpoint

Object detection on COCO (results with SoftNMS):

model	#FLOPs	#Params	AP (val)	AP (test_dev)	download
SpineNet-143	524B	67M	50.9	51.0	ckpt | config
SpineNet-143 w/self-training	524B	67M	52.6	52.8	ckpt | config
SpineNet-190	1885B	164M	52.6	52.8	ckpt | config
SpineNet-190 w/self-training	1885B	164M	54.2	54.3	ckpt | config

Semantic segmentation on PASCAL VOC 2012:

model	#FLOPs	#Params	mIOU (val)	mIOU (test)	download
EfficientNet-B7-NASFPN	60B	71M	85.2	-	ckpt | config
EfficientNet-B7-NASFPN w/ self-training	60B	71M	86.7	-	ckpt | config
EfficientNet-L2-NASFPN	229B	485M	88.7	-	ckpt | config
EfficientNet-L2-NASFPN w/ self-training	229B	485M	90.0	90.5	ckpt | config

Prepare Data

The training expects the data in TFExample format stored in TFRecord. Tools and scripts are provided to download and convert datasets.

Dataset	Tool
ImageNet	instructions
COCO	instructions
PASCAL	instructions

Citation

@article{zoph20selftraining,
  title={Rethinking pre-training and self-training},
  author={Barret Zoph and Golnaz Ghiasi and Tsung-Yi Lin and Yin Cui and Hanxiao Liu and Ekin D. Cubuk and Quoc V. Le},
  journal={CoRR},
  volume={abs/2006.06882},
  year={2020}
}