Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

datumbox · 2020-10-30T17:00:21Z

Potential fix for BC of #2933

By overwriting the FrozenBN.eps value to 0.0 when pretrained=True, we get the same accuracy stats as before.

Click here for the accuracy stats of fasterrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.370
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.211
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.307
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.545
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.647

Click here for the accuracy stats of retinanet_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.364
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.557
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.191
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.500
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.539
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.339
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.581
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.696

Click here for the accuracy stats of maskrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.379
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.592
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.411
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.215
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.414
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.313
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.495
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.520
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.321
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.559
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666
IoU metric: segm
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.346
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.560
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.368
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.156
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.506
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.455
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.475
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.269
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.515
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.636

Click here for the accuracy stats of keypointrcnn_resnet50_fpn

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.546
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.830
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.595
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.380
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.627
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.702
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.187
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.555
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.635
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.699
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.774
IoU metric: keypoints
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.650
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.861
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.714
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.603
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.730
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.717
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.906
 Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.774
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.669
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.785

codecov · 2020-10-30T19:43:32Z

Codecov Report

Merging #2940 into master will increase coverage by 0.02%.
The diff coverage is 92.30%.

@@            Coverage Diff             @@
##           master    #2940      +/-   ##
==========================================
+ Coverage   73.41%   73.43%   +0.02%     
==========================================
  Files          99       99              
  Lines        8801     8813      +12     
  Branches     1389     1391       +2     
==========================================
+ Hits         6461     6472      +11     
- Misses       1915     1916       +1     
  Partials      425      425

Impacted Files	Coverage Δ
torchvision/models/detection/retinanet.py	`75.87% <50.00%> (-0.21%)`	⬇️
torchvision/models/detection/_utils.py	`85.43% <100.00%> (+0.39%)`	⬆️
torchvision/models/detection/faster_rcnn.py	`81.81% <100.00%> (+0.42%)`	⬆️
torchvision/models/detection/keypoint_rcnn.py	`88.88% <100.00%> (+0.31%)`	⬆️
torchvision/models/detection/mask_rcnn.py	`90.62% <100.00%> (+0.30%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36daee3...07bc7a0. Read the comment docs.

fmassa

Thanks a lot for fixing the BC Vasilis!

I only have a couple of minor comments which are non-blocking but would be good to have (at least the comment). Let me know what you think

torchvision/models/_utils.py

torchvision/models/detection/faster_rcnn.py

fmassa

Thanks a lot!

…pytorch#2940) * Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. * Moving the method to detection utils and adding comments.

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models.

233a388

datumbox changed the title ~~Overwriting FrozenBN eps=0.0 if pretrained=True for detection models.~~ [DONOTMERGE] Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. Oct 30, 2020

facebook-github-bot added the cla signed label Oct 30, 2020

datumbox requested a review from fmassa October 30, 2020 17:36

datumbox mentioned this pull request Oct 30, 2020

[DONOTMERGE] Update the accuracy metrics of detection models #2942

Closed

fmassa approved these changes Nov 3, 2020

View reviewed changes

torchvision/models/_utils.py Outdated Show resolved Hide resolved

torchvision/models/detection/faster_rcnn.py Show resolved Hide resolved

datumbox mentioned this pull request Nov 3, 2020

[RFC] How to handle BC breaking changes on Model weights or hyper-parameters #2955

Open

Moving the method to detection utils and adding comments.

07bc7a0

datumbox changed the title ~~[DONOTMERGE] Overwriting FrozenBN eps=0.0 if pretrained=True for detection models.~~ Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. Nov 3, 2020

fmassa approved these changes Nov 3, 2020

View reviewed changes

fmassa merged commit 5b61a5c into pytorch:master Nov 3, 2020

datumbox deleted the feature/overwrite_eps branch November 3, 2020 18:12

fmassa mentioned this pull request Dec 10, 2020

Change default value of eps in FrozenBatchNorm to match BatchNorm #2599

Closed

datumbox mentioned this pull request Sep 21, 2021

Investigate decrease in mAP in retinanet resnet50 model #4437

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

datumbox commented Oct 30, 2020 •

edited

Loading

codecov bot commented Oct 30, 2020 •

edited

Loading

fmassa left a comment

fmassa left a comment

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

Overwriting FrozenBN eps=0.0 if pretrained=True for detection models. #2940

Conversation

datumbox commented Oct 30, 2020 • edited Loading

codecov bot commented Oct 30, 2020 • edited Loading

Codecov Report

fmassa left a comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

datumbox commented Oct 30, 2020 •

edited

Loading

codecov bot commented Oct 30, 2020 •

edited

Loading