Reproducibility Problem(ResNet-50 on ImageNet)

[ildoonet]

Below is the top1 errors with N and M grid search, epoch=180.

M\N	1	2	3
5	0.2318	0.2303	0.2327
7	0.2281	0.2294	0.2323
9	0.2292	0.2289	0.2301
11	0.2264	0.2284	0.2320
13	0.2282	0.2294	0.2294
15	0.2258	0.2265	0.2297

But, I changed epoch from 180(paper's) to 270(autoaugment's), the result with N=2, M=9 are similar to the reported value(top1 error = 22.4).

[ildoonet]

According to the author's email reply, I changed some parameters.

With epoch 180 (paper(RandAugment)'s epoch)

M\N	1	2	3
5	0.2284	0.2298	0.2321
7	0.2288	0.2286	0.2298
9	0.2286	0.2287	0.2308
11	0.2262	0.2265	0.2316
13	0.2283	0.2264	0.2299
15	0.2286	0.2264	0.2304

The performance with N=2, M=9 (reported optimal value) was not matched to the reported performance(top1 error = 22.4). Also there is no model that performs as well as the paper claims, even though we evaluated all models in the same space.

With epoch 270 (AutoAugment's epoch)

M\N	1	2	3
5	0.2271	0.2284	0.2312
7	0.2271	0.2287	0.2263
9	0.2262	0.2287	0.2286
11	0.2255	0.2253	0.2276
13	0.2241	0.2250	0.2275
15	0.2224	0.2246	0.2271

If we increase the training epoch to 270, one of the results is outperforming 'AutoAugment' and 'RandAugment'.

[BarretZoph]

Have you tried using the randuagment code in https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet? This was the code used for training the ResNet model. Also we are almost done opensourcing the ResNet model.

[ildoonet]

https://github.com/tensorflow/tpu/blob/8462d083dd89489a79e3200bcc8d4063bf362186/models/official/efficientnet/autoaugment.py#L663

@BarretZoph Is this the code you mentioned? Some parts are mismatched with this code, I will change them and conduct experiments soon.

[ildoonet]

@BarretZoph Is this correct?

https://github.com/tensorflow/tpu/blob/8462d083dd89489a79e3200bcc8d4063bf362186/models/official/efficientnet/autoaugment.py#L181

Since SolarizeAdd have a addition parameter with "0"(zero), this doesn't affect the image.

[BarretZoph]

Yes that is the code I mentioned. Also the code for ResNet-50 will be opensourced soon!

No I believe that is that the magnitude hyperparameter control. It is the threshold value that does not change.

[ildoonet]

@BarretZoph Thanks for opensourcing. I really look forward to it!

Also for SolarizeAdd, sorry for my misunderstanding. I will update this repo and start new experiments. Thanks.

[BarretZoph]

Great! Let me know how everything goes. The open sourcing will be done in a week or two!

[ildoonet]

I need to examine more since the performance doesn't match after I change augmentation search space as RandAugment's code. Best top1 error is 22.65 by training 180 epochs.

cc @BarretZoph

	1	2	3
5	0.2318	0.2334	0.2371
7	0.2272	0.2330	0.2365
9	0.2287	0.2295	0.2338
11	0.2279	0.2287	0.2352
13	0.2285	0.2265	0.2337
15	0.2262	0.2280	0.2315

[BarretZoph]

Hmm well the code in https://github.com/tensorflow/tpu/tree/master/models/official/resnet will be opensourced shortly. Hopefully that will resolve all of your issues.

[ildoonet]

@BarretZoph Thanks, I will look into your opensourced codes.

By checking codes provided by you, I'm not sure what is different. Below items are few things that might ruin the performance.

• In your paepr, "The image size was 224 by 244" seems to be a typo. Image size should be 224 by 224, right?
• Is this the base configuration you used? : https://github.com/tensorflow/tpu/blob/master/models/official/resnet/configs/resnet_config.py
  • Parameters(learning rate, epochs, ...) are different than the paper. (I guess you modified this before you trained the model)
  • dropblock is used, which is not in this repo.
  • precision is fp16, where mine uses fp32.
  • do you use Lars optimizer when you increase batch size(eg 4k)?

I guess if you opensource your codes as well as the configuration, it would be very helpful. Many Thanks!

[BarretZoph]

Thanks I just fixed the image size in the paper! Yes it should be 224x224.

Yes that is the base config but a few things were changed (180 epochs and batch size of 4096 with 32 replicas).

Dropblock is actually not used in this. Since 'dropblock_groups': '' there are no groups specified, so it will not be applied. https://github.com/tensorflow/tpu/blob/master/models/official/resnet/resnet_main.py#L88

I believe that changing that changing the precision will not make much of a difference.

No LARS is not used with the 4K batch size.