Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN #3205

datumbox · 2020-12-22T23:45:29Z

Related to #3198

This PR investigates whether there is any benefit in adding a score threshold for RegionProposalNetwork similar to what we do for RoIHeads. The idea is to reduce the number of candidates before we do the expensive NMS step and we assess the speed and the accuracy changes from the change.

Effects of Threshold on Inference (pre-trained models)

First of all let's check the effect of various threshold values on inference in the pre-trained models. This will give us an idea on how sensitive the accuracy is to different values (this will be used later once we train the models from scratch):

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
       --dataset coco --model fasterrcnn_resnet50_fpn --pretrained --test-only

Before every experiment we adapt manually the threshold directly on the source code. Here are the results for different values:

threshold=0.0

Submitted batch job 34066683

0: Test: Total time: 0:00:38 (0.0622 s / it)
0: Averaged stats: model_time: 0.0446 (0.0465)  evaluator_time: 0.0069 (0.0100)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.370
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.211
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.307
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.545
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.647

threshold=0.05

Submitted batch job 34066468

0: Test: Total time: 0:00:38 (0.0616 s / it)
0: Averaged stats: model_time: 0.0431 (0.0453)  evaluator_time: 0.0064 (0.0103)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.370
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.586
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.210
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.404
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.308
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.485
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.509
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.545
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.647

threshold=0.1

Submitted batch job 34067146

0: Test: Total time: 0:00:35 (0.0560 s / it)
0: Averaged stats: model_time: 0.0402 (0.0425)  evaluator_time: 0.0055 (0.0085)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.369
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.585
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.210
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.403
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.308
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.484
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.508
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.314
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.545
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.645

threshold=0.5

Submitted batch job 34067396

0: Test: Total time: 0:00:37 (0.0601 s / it)
0: Averaged stats: model_time: 0.0421 (0.0440)  evaluator_time: 0.0067 (0.0100)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.369
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.584
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.398
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.210
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.402
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.482
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.306
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.481
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.504
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.310
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.539
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644

threshold=0.75

Submitted batch job 34067628

0: Test: Total time: 0:00:37 (0.0600 s / it)
0: Averaged stats: model_time: 0.0418 (0.0439)  evaluator_time: 0.0064 (0.0099)
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.368
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.583
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.396
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.209
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.400
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.481
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.305
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.477
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.499
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.308
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.533
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.638

threshold=0.9

Submitted batch job 34067969

0: Test: Total time: 0:00:36 (0.0581 s / it)
0: Averaged stats: model_time: 0.0415 (0.0433)  evaluator_time: 0.0058 (0.0091)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.366
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.580
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.394
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.206
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.397
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.480
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.303
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.472
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.492
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.300
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.525
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.633

threshold=0.95

Submitted batch job 34068760

0: Test: Total time: 0:00:32 (0.0525 s / it)
0: Averaged stats: model_time: 0.0389 (0.0406)  evaluator_time: 0.0047 (0.0073)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.363
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.575
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.392
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.203
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.394
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.479
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.301
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.465
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.484
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.290
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.516
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.621

threshold=1.0

Submitted batch job 34067918

0: Test: Total time: 0:00:30 (0.0484 s / it)
0: Averaged stats: model_time: 0.0386 (0.0410)  evaluator_time: 0.0017 (0.0020)
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

Based on the above stats we observe that:

Adding a threshold does not provide massive speed up on inference because this is a less expensive part of the predictions.
One can use quite large threshold values without hurting accuracy too much.

Effects of Threshold on Training & Accuracy

Based on the above we will use a moderate threshold of 0.5 and retrain all the existing *RCNN models. All baselines are taken from the stats estimated at #2954 (the PR's branch stats are our current master).

fasterrcnn_resnet50_fpn

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
       --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26 --lr-steps 16 22 --aspect-ratio-group-factor 3

Branch: boxAP=38.1, trainingTime=11:31:39

Submitted batch job 34069542

0: Test: Total time: 0:00:37 (0.0592 s / it)
0: Averaged stats: model_time: 0.0402 (0.0469)  evaluator_time: 0.0053 (0.0089)
0: Accumulating evaluation results...
0: DONE (t=4.06s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.381
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.587
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.411
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.213
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.407
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.510
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.311
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.482
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.502
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.305
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.534
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.651
0: Training time 11:31:39

Master: boxAP=38.0, trainingTime=12:10:40

Submitted batch job 32444455

0: Test: Total time: 0:00:38 (0.0613 s / it)
0: Averaged stats: model_time: 0.0446 (0.0488)  evaluator_time: 0.0068 (0.0113)
0: Accumulating evaluation results...
0: DONE (t=5.83s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.380
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.587
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.411
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.214
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.413
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.500
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.316
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.496
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.521
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.327
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.557
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.664
0: Training time 12:10:40

maskrcnn_resnet50_fpn

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
       --dataset coco --model maskrcnn_resnet50_fpn --epochs 26 --lr-steps 16 22 --aspect-ratio-group-factor 3

Branch: boxAP=39.0, maskAP=35.1, trainingTime=14:14:00

Submitted batch job 34144523

0: Test: Total time: 0:01:16 (0.1218 s / it)
0: Averaged stats: model_time: 0.0556 (0.0737)  evaluator_time: 0.0194 (0.0355)
0: Accumulating evaluation results...
0: DONE (t=4.75s).
0: Accumulating evaluation results...
0: DONE (t=4.60s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.390
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.594
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.422
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.217
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.417
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.523
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.319
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.493
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.514
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.314
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.547
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.659
0: IoU metric: segm
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.351
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.563
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.373
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.160
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.373
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.523
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.295
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.448
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.464
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.263
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.498
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.623
0: Training time 14:14:00

Master: boxAP=38.7, maskAP=34.7, trainingTime=14:43:42

Submitted batch job 32444457

0: Test: Total time: 0:01:25 (0.1363 s / it)
0: Averaged stats: model_time: 0.0703 (0.0881)  evaluator_time: 0.0333 (0.0456)
0: Accumulating evaluation results...
0: DONE (t=6.12s).
0: Accumulating evaluation results...
0: DONE (t=5.94s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.387
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.592
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.420
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.216
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.421
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.512
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.321
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.506
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.531
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.333
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.569
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.683
0: IoU metric: segm
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.347
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.559
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.371
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.152
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.373
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.519
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.297
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.458
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.478
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.278
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.514
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.642
0: Training time 14:43:42

keypointrcnn_resnet50_fpn

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
       --dataset coco_kp --model keypointrcnn_resnet50_fpn --epochs 46 --lr-steps 36 43 --aspect-ratio-group-factor 3

Branch: boxAP=55.6, kpAP=63.9, trainingTime=13:25:04

Submitted batch job 34144524

0: Test: Total time: 0:00:39 (0.0630 s / it)
0: Averaged stats: model_time: 0.0456 (0.0517)  evaluator_time: 0.0039 (0.0074)
0: Accumulating evaluation results...
0: DONE (t=0.72s).
0: Accumulating evaluation results...
0: DONE (t=0.21s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.556
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.828
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.607
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.380
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.631
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.723
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.192
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.559
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.633
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.476
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.699
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.789
0: IoU metric: keypoints
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.639
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.857
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.694
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.593
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.716
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.710
0:  Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.898
0:  Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.759
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.664
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.777
0: Training time 13:25:04

Master: boxAP=55.5, kpAP=65.1, trainingTime=14:30:21

Submitted batch job 32444458

0: Test: Total time: 0:00:47 (0.0761 s / it)
0: Averaged stats: model_time: 0.0509 (0.0600)  evaluator_time: 0.0036 (0.0090)
0: Accumulating evaluation results...
0: DONE (t=0.99s).
0: Accumulating evaluation results...
0: DONE (t=0.31s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.555
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.833
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.603
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.381
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.634
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.719
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.190
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.564
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.647
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.499
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.709
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.796
0: IoU metric: keypoints
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.651
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.865
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.706
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.605
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.730
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.719
0:  Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.911
0:  Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.768
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.670
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.787
0: Training time 14:30:21

Based on the above we see that the introduction of a moderate threshold enables us to achieve a speed improvement of roughly 3.5-7.5% in training time without hurting the accuracy. Note that the above thresholds are not tuned and one could potentially achieve greater gains by using different thresholds.

Edit:

The keypointrcnn_resnet50_fpn seems to require a slightly different threshold to boost accuracy. Using a value of 0.1 yields the following results:

Branch: boxAP=56.1, kpAP=64.7, trainingTime=13:51:37

Submitted batch job 34346469

0: Test: Total time: 0:00:38 (0.0621 s / it)
0: Averaged stats: model_time: 0.0449 (0.0528)  evaluator_time: 0.0038 (0.0077)
0: Accumulating evaluation results...
0: DONE (t=0.72s).
0: Accumulating evaluation results...
0: DONE (t=0.22s).
0: IoU metric: bbox
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.561
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.838
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.613
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.381
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.636
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.725
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.191
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.562
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.638
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.483
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.705
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.793
0: IoU metric: keypoints
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.647
0:  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.864
0:  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.706
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.598
0:  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.726
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.713
0:  Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.902
0:  Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.767
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.663
0:  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.784
0: Training time 13:51:37

The above indicates that the threshold can be beneficial both for improving the accuracy metrics and speed but some tuning might be required to achieve best results.

datumbox · 2021-01-07T09:56:15Z

@t-vi I discussed with @fmassa about adding the specific threshold in the FasterRCNN implementation as it seems to benefit both the speed and the accuracy metrics and your name came up. Any thoughts on this?

t-vi · 2021-01-07T10:17:27Z

I think @masahi had opinions on it, I didn't really look at it in detail.

masahi · 2021-01-07T10:39:41Z

well not really an opinion, but nothing changes by default and the only extra costs are one additional parameter (of which there are already many) and one call to sigmoid and where, both negligible. So I'd say why not.

codecov · 2021-01-13T19:31:17Z

Codecov Report

Merging #3205 (26fc770) into master (d0063f3) will increase coverage by 0.03%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master    #3205      +/-   ##
==========================================
+ Coverage   73.45%   73.49%   +0.03%     
==========================================
  Files         102      101       -1     
  Lines        9329     9241      -88     
  Branches     1494     1480      -14     
==========================================
- Hits         6853     6792      -61     
+ Misses       2012     1989      -23     
+ Partials      464      460       -4

Impacted Files	Coverage Δ
torchvision/models/detection/faster_rcnn.py	`74.41% <ø> (ø)`
torchvision/models/detection/keypoint_rcnn.py	`78.57% <ø> (ø)`
torchvision/models/detection/mask_rcnn.py	`79.66% <ø> (ø)`
torchvision/models/detection/_utils.py	`82.58% <60.00%> (-0.87%)`	⬇️
torchvision/models/detection/rpn.py	`90.41% <100.00%> (+0.23%)`	⬆️
torchvision/datasets/__init__.py	`100.00% <0.00%> (ø)`
torchvision/datasets/widerface.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d0063f3...0d2d8a8. Read the comment docs.

fmassa

Thanks

datumbox · 2021-01-14T10:45:05Z

The failing tests on ONNX are not related to this PR. The master is broken (see #3251).
The failing clang_format test is again unrelated and it's due to a version change on the formatter that is being investigated.

… on RPN (#3205) Summary: * Introduce small score threshold on rpn * Adding docs and fixing keypoint and mask. * Making value 0.0 by default for BC. * Fixing for onnx. * Update threshold. * Removing non-default threshold from reference scripts. Reviewed By: datumbox Differential Revision: D25954569 fbshipit-source-id: a429a13c7153f0b87c377464d3d062aa28370002 Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Introduce small score threshold on rpn

60318d9

facebook-github-bot added the cla signed label Dec 22, 2020

datumbox force-pushed the feature/thresholded_rpn branch 2 times, most recently from 33cd476 to ae8b92e Compare December 23, 2020 00:27

Adding docs and fixing keypoint and mask.

e0b3405

datumbox force-pushed the feature/thresholded_rpn branch 3 times, most recently from 6585030 to 8c38842 Compare December 23, 2020 13:04

Making value 0.0 by default for BC.

64b33a9

datumbox force-pushed the feature/thresholded_rpn branch 2 times, most recently from 6606123 to 264d4d7 Compare December 23, 2020 14:20

Fixing for onnx.

c42e309

datumbox force-pushed the feature/thresholded_rpn branch from 264d4d7 to c42e309 Compare December 23, 2020 14:25

datumbox mentioned this pull request Dec 23, 2020

Boxes with negative scores in NMS input? #3198

Closed

Update threshold.

1ee020f

datumbox changed the title ~~[WIP] Investigate speed/accuracy tradeoff from the introduction of score thresholds on RPN~~ Investigate speed/accuracy tradeoff from the introduction of score thresholds on RPN Jan 4, 2021

datumbox requested a review from fmassa January 4, 2021 11:09

datumbox changed the title ~~Investigate speed/accuracy tradeoff from the introduction of score thresholds on RPN~~ Improve speed/accuracy of FasterRCNN by introducing a score threshold on RPN Jan 7, 2021

datumbox and others added 2 commits January 8, 2021 10:30

Merge branch 'master' into feature/thresholded_rpn

a1f2796

Removing non-default threshold from reference scripts.

26fc770

fmassa approved these changes Jan 14, 2021

View reviewed changes

Merge branch 'master' into feature/thresholded_rpn

0d2d8a8

datumbox merged commit 8ebfd2f into pytorch:master Jan 14, 2021

datumbox deleted the feature/thresholded_rpn branch January 14, 2021 10:46

jiafatom mentioned this pull request Jan 20, 2021

ONNX tests failing on master #3251

Closed

fmassa mentioned this pull request Jan 27, 2021

Replace functional.sigmoid with torch.sigmoid in RPN #3307

Merged

datumbox changed the title ~~Improve speed/accuracy of FasterRCNN by introducing a score threshold on RPN~~ Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN Feb 4, 2021

fmassa mentioned this pull request Aug 13, 2021

Negative Samples in Faster RCNN training results in NaN RPN_BOX_REG Loss #2144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN #3205

Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN #3205

datumbox commented Dec 22, 2020 •

edited

Loading

datumbox commented Jan 7, 2021

t-vi commented Jan 7, 2021

masahi commented Jan 7, 2021

codecov bot commented Jan 13, 2021 •

edited

Loading

fmassa left a comment

datumbox commented Jan 14, 2021 •

edited

Loading

Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN #3205

Improve speed/accuracy of *RCNN models by introducing a score threshold on RPN #3205

Conversation

datumbox commented Dec 22, 2020 • edited Loading

Effects of Threshold on Inference (pre-trained models)

Effects of Threshold on Training & Accuracy

fasterrcnn_resnet50_fpn

maskrcnn_resnet50_fpn

keypointrcnn_resnet50_fpn

datumbox commented Jan 7, 2021

t-vi commented Jan 7, 2021

masahi commented Jan 7, 2021

codecov bot commented Jan 13, 2021 • edited Loading

Codecov Report

fmassa left a comment

Choose a reason for hiding this comment

datumbox commented Jan 14, 2021 • edited Loading

datumbox commented Dec 22, 2020 •

edited

Loading

codecov bot commented Jan 13, 2021 •

edited

Loading

datumbox commented Jan 14, 2021 •

edited

Loading