Add simclr imagenet resnet50 benchmark

[guarin]

Changes

• Add benchmarks directory
• Add code for SimCLR ImageNet benchmark
• Add linear evaluation code
• Add knn evaluation code
• Add online linear evaluation code
• Add LARS optimizer

Closes #569

How was it tested?

• Most parts were manually tested
• Will add unit tests later

Runs

Initial run: https://tensorboard.dev/experiment/VD0GPdaoSLa5uonTp6k6dQ/#scalars
There seems to be an issue with the linear eval validation logic, top1 accuracy collapses after a couple epochs:

Doesn't happen for train top1 but we get max 51%. I would expect 64% val top1 (https://github.com/facebookresearch/vissl/blob/main/MODEL_ZOO.md#simclr).

Issues:

• We accidentally increased SimCLR default output dimension from 128 to 2048 but this shouldn't have a big impact on the accuracy: This was already fixed in Set default SimCLR output dim to 128 #1173

• Will check accuracy with pretrained supervised model to check if there is something off with our validation logic, would expect 76% val top1 (https://github.com/facebookresearch/vissl/blob/main/MODEL_ZOO.md#supervised)
  --> Got 73.1% knn val top1 and 76.3% linear eval top1 tensorboard.

• Verified that ntx_ent_loss is correct for non-distributed training

• Will check if there if the results differ between distributed and non-distributed training to verify that loss/backprop/sync works as expected

• Other options are

  • Incorrect training parameters
  • Incorrect augmentations
  • Incorrect collate -> Verified that multi view collate works as expected
  • Incorrect projection head (our implementation slightly differs from solo-learn/vissl in that it has a batch norm layer at the end of the head, but this is the way it is implemented in the authors code).

Edit: Issues fixed in #1175

Final Results

Ran the benchmark again. We get now 63.2% linear eval top1 accuracy, compared to the paper which reports 62.8% for the same settings. Full results:

Imagenet
	max val_online_cls_top1: 	0.6309599876403809
	max val_online_cls_top5: 	0.8507199883460999
	knn val_top1: 			0.44929999113082886
	knn val_top5: 			0.7418799996376038
	max linear val_top1: 		0.6319800019264221
	max linear val_top5: 		0.8527600169181824

Imagenet100
	max val_online_cls_top1: 	0.6898000240325928
	max val_online_cls_top5: 	0.8980000019073486
	knn val_top1: 			0.5752000212669373
	knn val_top5: 			0.8485999703407288
	max linear val_top1: 		0.7233999967575073
	max linear val_top5: 		0.9165999889373779
	max finetune val_top1: 		0.7954000234603882
	max finetune val_top5: 		0.949400007724762

Imagenette
	max val_online_cls_top1: 	0.8166072368621826
	max val_online_cls_top5: 	0.9801324605941772
	knn val_top1: 			0.7763627171516418
	knn val_top5: 			0.9793683290481567
	max linear val_top1: 		0.8494651317596436
	max linear val_top5: 		0.9857361316680908
	max finetune val_top1: 		0.6314314603805542
	max finetune val_top5: 		0.943708598613739

Training was 100 epochs with batch size 256 (=2x128). This corresponds to the first entry in this table: