[BC-breaking] Change default eps value of FrozenBN by datumbox · Pull Request #2933 · pytorch/vision

datumbox · 2020-10-29T11:10:50Z

The good news is that changing the FrozenBatchNorm2d default eps value from 0.0 to 1e-5 does not affect the accuracy of the detection models. You will see some tiny variations from the reported metrics but nothing concerning (could be the result of floating point errors). So I believe we can safely land the change without problems.

Click here for the accuracy stats of fasterrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.369
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.480
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.307
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.545
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644

Click here for the accuracy stats of retinanet_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.363
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.557
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.191
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.489
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.500
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.539
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.339
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.695

Click here for the accuracy stats of maskrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.379
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.591
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.410
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.215
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.414
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.494
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.313
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.495
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.519
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.321
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.559
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666
IoU metric: segm
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.346
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.560
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.368
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.156
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.507
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.455
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.475
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.269
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.515
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.636

Click here for the accuracy stats of keypointrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.546
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.830
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.597
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.380
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.628
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.702
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.187
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.555
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.636
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.700
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.775
IoU metric: keypoints
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.651
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.861
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.716
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.604
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.730
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.717
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.906
 Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.774
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.670
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.784

The bad news is that the following detection unit-tests are brittle and changing the eps value causes them to break:

test_keypointrcnn_resnet50_fpn_cpu
test_keypointrcnn_resnet50_fpn_cuda
test_retinanet_resnet50_fpn_cpu

I tried updating the expecting values to new ones but unfortunately there is too much variation across different devices and platforms. As observed at #2812, the flakiness is caused by floating point errors and the use of unstable sort. The problem is the result of initializing the model weights with random values and using random input in the unit-tests.

To fix the problem I introduce a VERY hacky workaround where I switch the default value to 0.0 for specific unit-tests. I really don't like this hack but I don't see how else we can keep the unit-tests on and while changing the eps . If anyone has another idea, I'm happy to give it a try.

codecov · 2020-10-29T18:09:12Z

Codecov Report

Merging #2933 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2933   +/-   ##
=======================================
  Coverage   73.41%   73.41%           
=======================================
  Files          99       99           
  Lines        8801     8801           
  Branches     1389     1389           
=======================================
  Hits         6461     6461           
  Misses       1915     1915           
  Partials      425      425

Impacted Files	Coverage Δ
torchvision/ops/misc.py	`86.04% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fe1e11...e4f3ce1. Read the comment docs.

fmassa

Thanks for the PR!

It's great that the changes on accuracy don't seem to be affected in a significant way.
I suspect the results before the change are also the same values, or does it decrease mAP by 0.1?

I have one remark about the hack, I think it can be made such that we don't need to change the implementation in the op itself

test/test_models.py

datumbox · 2020-10-30T10:57:31Z

Here are the baseline vs new comparisons:

fasterrcnn_resnet50_fpn:

MASTER: boxAP=0.370
BRANCH: boxAP=0.369

retinanet_resnet50_fpn:

MASTER: boxAP=0.364
BRANCH: boxAP=0.363

maskrcnn_resnet50_fpn:

MASTER: boxAP=0.379 / maskAP=0.346
BRANCH: boxAP=0.379 / maskAP=0.346

keypointrcnn_resnet50_fpn:

MASTER: boxAP=0.546 / kpAP=0.650
BRANCH: boxAP=0.546 / kpAP=0.651

So overall I think we are OK.

fmassa

Awesome, thanks a lot!

* Change default eps value of FrozenBN. * Update the unit-tests.` * Update the expected values. * Revert the expected value and use original eps=0 value for flaky tests. * Post init change of eps. * Styles.

datumbox added 3 commits October 29, 2020 11:09

Change default eps value of FrozenBN.

28476c1

Update the unit-tests.`

13e85cc

Update the expected values.

2f92072

datumbox force-pushed the bugfix/frozenbn_eps branch from e21d669 to ab66c93 Compare October 29, 2020 16:27

Revert the expected value and use original eps=0 value for flaky tests.

7c8c5ae

datumbox force-pushed the bugfix/frozenbn_eps branch from ab66c93 to 7c8c5ae Compare October 29, 2020 16:34

datumbox requested a review from fmassa October 29, 2020 17:09

datumbox changed the title ~~[WIP] Change default eps value of FrozenBN~~ Change default eps value of FrozenBN Oct 29, 2020

fmassa requested changes Oct 30, 2020

View reviewed changes

test/test_models.py Outdated Show resolved Hide resolved

datumbox added 2 commits October 30, 2020 09:18

Post init change of eps.

b07b4e3

Styles.

e4f3ce1

fmassa approved these changes Oct 30, 2020

View reviewed changes

fmassa changed the title ~~Change default eps value of FrozenBN~~ [BC-breaking] Change default eps value of FrozenBN Oct 30, 2020

fmassa merged commit 45e027c into pytorch:master Oct 30, 2020

datumbox deleted the bugfix/frozenbn_eps branch October 30, 2020 12:02

This was referenced Oct 30, 2020

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

Merged

[DONOTMERGE] Update the accuracy metrics of detection models #2942

Closed

frgfm mentioned this pull request Dec 10, 2020

Change default value of eps in FrozenBatchNorm to match BatchNorm #2599

Closed

fmassa mentioned this pull request Sep 21, 2021

Investigate decrease in mAP in retinanet resnet50 model #4437

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BC-breaking] Change default eps value of FrozenBN#2933

[BC-breaking] Change default eps value of FrozenBN#2933
fmassa merged 6 commits intopytorch:masterfrom
datumbox:bugfix/frozenbn_eps

datumbox commented Oct 29, 2020 •

edited

Loading

Uh oh!

codecov bot commented Oct 29, 2020 •

edited

Loading

Uh oh!

fmassa left a comment

Uh oh!

Uh oh!

datumbox commented Oct 30, 2020 •

edited

Loading

Uh oh!

fmassa left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

datumbox commented Oct 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fmassa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

datumbox commented Oct 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fmassa left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

datumbox commented Oct 29, 2020 •

edited

Loading

codecov bot commented Oct 29, 2020 •

edited

Loading

datumbox commented Oct 30, 2020 •

edited

Loading