Transformer Object Detection

[kieranfraser]

Description

Adding DEtection TRansformer (DETR) to ART to facilitate object detection using Transformers.

• DETR estimator added to art/estimators/object_detection
• requires additional method in art/estimators/pytorch.py which sets MultiheadAttention layer into train/eval mode
• associated tests added
• notebook demonstrating use of DETR estimator detecting objects and under adversarial patch attack added to notebooks/adversarial_patch/ and updated notebooks/ README

Paper: "End-to-End Object Detection with Transformers"

Fixes # (issue)
#1638

Type of change

Please check all relevant options.

• Improvement (non-breaking)
• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

• Test prediction of DETR
• Test loss gradient
• Test applying preprocessing defences
• Test computation of losses (classification, bbox, giou)
• Test DETR with PGD attack

Test Configuration:

• OS
• Python version
• ART version or commit number
• TensorFlow / Keras / PyTorch / MXNet version

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[codecov-commenter]

Codecov Report

Merging #2047 (cb49cdd) into dev_1.15.0 (30e513c) will decrease coverage by 9.59%.
The diff coverage is 73.09%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@              Coverage Diff               @@
##           dev_1.15.0    #2047      +/-   ##
==============================================
- Coverage       75.97%   66.38%   -9.59%     
==============================================
  Files             297      307      +10     
  Lines           26516    27480     +964     
  Branches         4867     5047     +180     
==============================================
- Hits            20145    18243    -1902     
- Misses           5325     8198    +2873     
+ Partials         1046     1039       -7     

Impacted Files	Coverage Δ
art/attacks/evasion/auto_conjugate_gradient.py	86.52% <ø> (+0.94%)	⬆️
art/attacks/evasion/brendel_bethge.py	87.01% <ø> (ø)
.../defences/trainer/ibp_certified_trainer_pytorch.py	75.00% <0.00%> (-3.17%)	⬇️
art/attacks/poisoning/bad_det/bad_det_rma.py	96.42% <33.33%> (ø)
...s/certification/randomized_smoothing/tensorflow.py	89.83% <50.00%> (-1.55%)	⬇️
...cks/poisoning/perturbations/image_perturbations.py	88.46% <55.55%> (ø)
...certification/derandomized_smoothing/tensorflow.py	83.33% <66.66%> (-1.67%)	⬇️
art/estimators/object_detection/pytorch_yolo.py	77.57% <68.67%> (+64.27%)	⬆️
.../object_detection/pytorch_detection_transformer.py	69.93% <69.93%> (ø)
art/estimators/classification/tensorflow.py	82.92% <75.00%> (+6.88%)	⬆️
... and 9 more

... and 77 files with indirect coverage changes

[beat-buesser]

Hi @kieranfraser Thank you very much for your pull request! I have added a few review comments, please let me know what you think.

[beat-buesser]

What is the license of these functions from https://github.com/facebookresearch/detr?

[beat-buesser]

Can we add typing to these functions?

[kieranfraser]

The license is Apache 2.0 which should be okay?

[beat-buesser]

We need to either add a pytest call to run_tests.sh or integrate into one existing or new dedicated testing runs for object detectors to make the tests run automatically.

[kieranfraser]

Tried to add to run_tests.sh, but tests failing in setup with urllib.error.HTTPError: HTTP Error 500: Internal Server Error, which could be due to needing to pull in the pre-trained weights from torch.hub? The PyTorch Faster-RCNN test, which also pulls a pre-trained model from torch.hub, is called from a GitHub workflow (ci-pytorch-object-detectors.yml) - so that might be a better place to call?

[beat-buesser]

Continuing in #2192