Provide checkpoints for MixFormer. (open-mmlab#778)

* add MixFormer in README * add checkpoints * add mixformer in docs * fix metafile * update test results * fix mixformer test pipeline * update model-index.yml
shuyangsun · Nov 25, 2022 · b1679f9 · b1679f9
1 parent 87d55ec
commit b1679f9
Show file tree

Hide file tree

Showing 10 changed files with 43 additions and 27 deletions.
diff --git a/README.md b/README.md
@@ -112,6 +112,7 @@ Supported Methods
 
 - [x] [SiameseRPN++](configs/sot/siamese_rpn) (CVPR 2019)
 - [x] [STARK](configs/sot/stark) (ICCV 2021)
+- [x] [MixFormer](configs/sot/mixformer) (CVPR 2022)
 - [ ] [PrDiMP](https://arxiv.org/abs/2003.12565) (CVPR2020) (WIP)
 
 Supported Datasets

diff --git a/README_zh-CN.md b/README_zh-CN.md
@@ -108,6 +108,7 @@ MMTracking也提供了更详细的[教程](docs/zh_cn/tutorials/)，比如[配
 
 - [x] [SiameseRPN++](configs/sot/siamese_rpn) (CVPR 2019)
 - [x] [STARK](configs/sot/stark) (ICCV 2021)
+- [x] [MixFormer](configs/sot/mixformer) (CVPR 2022)
 - [ ] [PrDiMP](https://arxiv.org/abs/2003.12565) (CVPR2020) (WIP)
 
 支持的数据集：

diff --git a/configs/sot/mixformer/README.md b/configs/sot/mixformer/README.md
@@ -30,18 +30,18 @@ We now provide the evaluation results using model weights released in [official
 
 ### Lasot
 
-|  Method   | Style | Inf time(fps) | Success | Norm precision | Precision |                 Config                  |
-| :-------: | :---: | :-----------: | :-----: | :------------: | :-------: | :-------------------------------------: |
-| MixFormer |   -   |       -       |  69.2   |      78.8      |   74.3    | [config](./mixformer_cvt_500e_lasot.py) |
+|  Method   | Style | Inf time(fps) | Success | Norm precision | Precision |                 Config                  |                                                        Download                                                        |
+| :-------: | :---: | :-----------: | :-----: | :------------: | :-------: | :-------------------------------------: | :--------------------------------------------------------------------------------------------------------------------: |
+| MixFormer |   -   |       -       |  69.0   |      79.6      |   75.2    | [config](./mixformer_cvt_500e_lasot.py) | [model](https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_lasot/mixformer_cvt_500e_lasot.pth) |
 
 ### TrackingNet
 
-|  Method   | Style | Inf time(fps) | Success | Norm precision | Precision |                    Config                     |
-| :-------: | :---: | :-----------: | :-----: | :------------: | :-------: | :-------------------------------------------: |
-| MixFormer |   -   |       -       |  81.1   |      86.6      |   80.0    | [config](./mixformer_cvt_500e_trackingnet.py) |
+|  Method   | Style | Inf time(fps) | Success | Norm precision | Precision |                    Config                     |                                                        Download                                                        |
+| :-------: | :---: | :-----------: | :-----: | :------------: | :-------: | :-------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------: |
+| MixFormer |   -   |       -       |  81.4   |      86.8      |   80.3    | [config](./mixformer_cvt_500e_trackingnet.py) | [model](https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_lasot/mixformer_cvt_500e_lasot.pth) |
 
 ### GOT10k
 
-|  Method   | Style | Inf time(fps) | Average Overlap | Success Rate 0.5 | Success Rate 0.75 |                  Config                  |
-| :-------: | :---: | :-----------: | :-------------: | :--------------: | :---------------: | :--------------------------------------: |
-| MixFormer |   -   |       -       |      72.0       |       82.0       |       68.1        | [config](./mixformer_cvt_500e_got10k.py) |
+|  Method   | Style | Inf time(fps) | Average Overlap | Success Rate 0.5 | Success Rate 0.75 |                  Config                  |                                                         Download                                                         |
+| :-------: | :---: | :-----------: | :-------------: | :--------------: | :---------------: | :--------------------------------------: | :----------------------------------------------------------------------------------------------------------------------: |
+| MixFormer |   -   |       -       |      70.1       |       80.1       |       65.6        | [config](./mixformer_cvt_500e_got10k.py) | [model](https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_got10k/mixformer_cvt_500e_got10k.pth) |
diff --git a/configs/sot/mixformer/metafile.yml b/configs/sot/mixformer/metafile.yml
@@ -1,8 +1,12 @@
 Collections:
   - Name: MixFormer
     Metadata:
-     Architecture:
-        - TransFormer
+      Training Data: GOT10k, LaSOT, TrackingNet, MSCOCO
+      Training Techniques:
+        - AdamW
+      Training Resources: 8x V100 GPUs
+      Architecture:
+        - Convolutional Vision TransFormer
     Paper:
         URL: https://arxiv.org/abs/2203.11082
         Title: "MixFormer: End-to-End Tracking with Iterative Mixed Attention"
@@ -16,9 +20,10 @@ Models:
       - Task: Single Object Tracking
         Dataset: GOT10k
         Metrics:
-          AO: 72.0
-          SR0.5: 82.0
-          SR0.75: 68.1
+          AO: 70.1
+          SR0.5: 80.1
+          SR0.75: 65.6
+    Weights: https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_got10k/mixformer_cvt_500e_got10k.pth
 
   - Name: mixformer_cvt_500e_lasot
     In Collection: MixFormer
@@ -27,9 +32,10 @@ Models:
       - Task: Single Object Tracking
         Dataset: LaSOT
         Metrics:
-          Success: 69.2
-          Norm Precision: 78.8
-          Precision: 74.3
+          Success: 69.0
+          Norm Precision: 79.6
+          Precision: 75.2
+    Weights: https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_lasot/mixformer_cvt_500e_lasot.pth
 
   - Name: mixformer_cvt_500e_trackingnet
     In Collection: MixFormer
@@ -38,6 +44,7 @@ Models:
       - Task: Single Object Tracking
         Dataset: TrackingNet
         Metrics:
-          Success: 81.1
-          Norm Precision: 86.6
-          Precision: 80.0
+          Success: 81.4
+          Norm Precision: 86.8
+          Precision: 80.3
+    Weights: https://download.openmmlab.com/mmtracking/sot/mixformer/mixformer_cvt_500e_lasot/mixformer_cvt_500e_lasot.pth
diff --git a/configs/sot/mixformer/mixformer_cvt_500e_got10k.py b/configs/sot/mixformer/mixformer_cvt_500e_got10k.py
@@ -47,7 +47,7 @@
         search_size=320,
         template_factor=2.0,
         template_size=128,
-        update_interval=[10],
+        update_interval=[25],
         online_size=[2],
         max_score_decay=[0.98],
     ))

diff --git a/configs/sot/mixformer/mixformer_cvt_500e_trackingnet.py b/configs/sot/mixformer/mixformer_cvt_500e_trackingnet.py
@@ -8,7 +8,7 @@
         template_factor=2.0,
         template_size=128,
         update_interval=[25],
-        online_size=[2],
+        online_size=[1],
         max_score_decay=[1.0],
     ))
 

diff --git a/docs/en/model_zoo.md b/docs/en/model_zoo.md
@@ -75,6 +75,10 @@ Please refer to [SiameseRPN++](https://github.com/open-mmlab/mmtracking/blob/mas
 
 Please refer to [STARK](https://github.com/open-mmlab/mmtracking/blob/master/configs/sot/stark) for details.
 
+### MixFormer (CVPR 2022)
+
+Please refer to [MixFormer](https://github.com/open-mmlab/mmtracking/blob/master/configs/sot/mixformer) for details.
+
 ## Baselines of video instance segmentation
 
 ### MaskTrack R-CNN (ICCV 2019)

diff --git a/docs/zh_cn/model_zoo.md b/docs/zh_cn/model_zoo.md
@@ -75,6 +75,10 @@
 
 详情请参考 [STARK](https://github.com/open-mmlab/mmtracking/blob/master/configs/sot/stark)。
 
+### MixFormer (CVPR 2022)
+
+详情请参考 [MixFormer](https://github.com/open-mmlab/mmtracking/blob/master/configs/sot/mixformer)。
+
 ## 视频个例分割基线
 
 ### MaskTrack R-CNN (ICCV 2019)

diff --git a/mmtrack/models/sot/mixformer.py b/mmtrack/models/sot/mixformer.py
@@ -102,9 +102,8 @@ def update_template(self, img, bbox, conf_score):
                 self.online_forget_id = (self.online_forget_id +
                                          1) % self.online_size
 
-            if self.online_size > 1:
-                with torch.no_grad():
-                    self.set_online(self.template, self.online_template)
+            with torch.no_grad():
+                self.set_online(self.template, self.online_template)
 
             self.best_conf_score = -1
             self.best_online_template = self.template
@@ -134,13 +133,12 @@ def track(self, img, bbox):
             out_dict = self.head(template, search)
 
         pred_box = out_dict['pred_bboxes']  # xyxy
-        # Baseline: Take the mean of all pred boxes as the final result
-
         pred_box = self.mapping_bbox_back(pred_box, self.memo.bbox,
                                           resize_factor)
         pred_box = self._bbox_clip(pred_box, H, W, margin=10)
 
         # update template
+        self.best_conf_score = self.best_conf_score * self.max_score_decay
         conf_score = -1.
         if self.head.score_decoder_head is not None:
             # get confidence score (whether the search region is reliable)

diff --git a/model-index.yml b/model-index.yml
@@ -6,6 +6,7 @@ Import:
   - configs/mot/ocsort/metafile.yml
   - configs/sot/siamese_rpn/metafile.yml
   - configs/sot/stark/metafile.yml
+  - configs/sot/mixformer/metafile.yml
   - configs/vid/dff/metafile.yml
   - configs/vid/fgfa/metafile.yml
   - configs/vid/selsa/metafile.yml