[Docs] oclip readme

README.md

@@ -110,6 +110,13 @@ Please see [Quick Run](https://mmocr.readthedocs.io/en/dev-1.x/get_started/quick
 
 Supported algorithms:
 
+<details open>
+<summary>BackBone</summary>
+
+- [x] [oCLIP](configs/backbone/oclip/README.md) (ECCV'2022)
+
+</details>
+
 <details open>
 <summary>Text Detection</summary>
 

README_zh-CN.md

@@ -110,6 +110,13 @@ pip3 install -e .
 
 支持的算法：
 
+<details open>
+<summary>BackBone</summary>
+
+- [x] [oCLIP](configs/backbone/oclip/README.md) (ECCV'2022)
+
+</details>
+
 <details open>
 <summary>文字检测</summary>
 

configs/backbone/oclip/README.md

@@ -0,0 +1,31 @@
+# oCLIP
+
+> [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://arxiv.org/abs/2203.03911)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Recently, Vision-Language Pre-training (VLP) techniques have greatly benefited various vision-language tasks by jointly learning visual and textual representations, which intuitively helps in Optical Character Recognition (OCR) tasks due to the rich visual and textual information in scene text images. However, these methods cannot well cope with OCR tasks because of the difficulty in both instance-level text encoding and image-text pair acquisition (i.e. images and captured texts in them). This paper presents a weakly supervised pre-training method, oCLIP, which can acquire effective scene text representations by jointly learning and aligning visual and textual information. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features, respectively, as well as a visual-textual decoder that models the interaction among textual and visual features for learning effective scene text representations. With the learning of textual features, the pre-trained model can attend texts in images well with character awareness. Besides, these designs enable the learning from weakly annotated texts (i.e. partial texts in images without text bounding boxes) which mitigates the data annotation constraint greatly. Experiments over the weakly annotated images in ICDAR2019-LSVT show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks, respectively. In addition, the proposed method outperforms existing pre-training techniques consistently across multiple public datasets (e.g., +3.2% and +1.3% for Total-Text and CTW1500).
+
+<div align=center>
+<img src="https://user-images.githubusercontent.com/24622904/199475057-aa688422-518d-4d7a-86fc-1be0cc1b5dc6.png"/>
+</div>
+
+## Support Model
+
+|           | DBNet | DBNet++ | FCENet | TextSnake | PSENet | DRRG | Mask R-CNN |
+| :-------: | :---: | :-----: | :----: | :-------: | :----: | :--: | :--------: |
+| ICDAR2015 |   ✓   |    ✓    |   ✓    |           |   ✓    |      |     ✓      |
+|  CTW1500  |       |         |        |     ✓     |   ✓    |  ✓   |     ✓      |
+
+## Citation
+
+```bibtex
+@article{xue2022language,
+  title={Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting},
+  author={Xue, Chuhui and Zhang, Wenqing and Hao, Yu and Lu, Shijian and Torr, Philip and Bai, Song},
+  journal={Proceedings of the European Conference on Computer Vision (ECCV)},
+  year={2022}
+}
+```

configs/backbone/oclip/metafile.yml

@@ -0,0 +1,10 @@
+Collections:
+- Name: oCLIP
+  Metadata:
+    Training Data: SynthText
+    Architecture:
+      - CLIPResNet
+  Paper:
+    URL: https://arxiv.org/abs/2203.03911
+    Title: 'Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting'
+  README: configs/backbone/oclip/README.md

configs/textdet/dbnet/README.md

@@ -20,6 +20,7 @@ Recently, segmentation-based methods are quite popular in scene text detection,
 | :--------------------------------------: | :-------------------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :-----------------------------------------: |
 | [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py) |                      ImageNet                       | ICDAR2015 Train | ICDAR2015 Test |  1200   |    736    |  0.8853   | 0.7583 | 0.8169 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/20220825_221614.log) |
 | [DBNet_r50dcn](/configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py) | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.8784   | 0.8315 | 0.8543 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/20220828_124917.log) |
+| [DBNet_oclip-r50](/configs/textdet/dbnet/dbnet_oclip-resnet50_fpnc_1200e_icdar2015.py) | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test |  1200   |           |           |        |        |          [model](<>) \| [log](<>)           |
 
 ## Citation
 

configs/textdet/dbnet/metafile.yml

@@ -38,3 +38,15 @@ Models:
         Metrics:
           hmean-iou: 0.8543
     Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth
+
+  - Name: dbnet_oclip-resnet50_fpnc_1200e_icdar2015
+    In Collection: DBNet
+    Config: configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou:
+    Weights:

configs/textdet/dbnetpp/README.md

@@ -19,6 +19,7 @@ Recently, segmentation-based scene text detection methods have drawn extensive a
 |                  Method                  |                  Pretrained Model                   |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                  Download                   |
 | :--------------------------------------: | :-------------------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :-----------------------------------------: |
 | [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py) | [Synthtext](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) ([model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth)) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9116   | 0.8291 | 0.8684 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/20220829_230108.log) |
+| [DBNetpp_oclip-r50](/configs/textdet/dbnetpp/dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015.py) |                          -                          | ICDAR2015 Train | ICDAR2015 Test |  1200   |           |           |        |        |          [model](<>) \| [log](<>)           |
 
 ## Citation
 

configs/textdet/dbnetpp/metafile.yml

@@ -26,3 +26,15 @@ Models:
         Metrics:
           hmean-iou: 0.8684
     Weights: https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth
+
+  - Name: dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015
+    In Collection: DBNetpp
+    Config: configs/textdet/dbnetpp/dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou:
+    Weights:

configs/textdet/drrg/README.md

@@ -19,6 +19,7 @@ Arbitrary shape text detection is a challenging task due to the high variety and
 |                            Method                            | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                            Download                            |
 | :----------------------------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------------: |
 | [DRRG](/configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py) |     ImageNet     | CTW1500 Train | CTW1500 Test |  1200   |    640    |  0.8775   | 0.8179 | 0.8467 | [model](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth) \\ [log](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/20220827_105233.log) |
+| [DRRG_oclip-r50](/configs/textdet/drrg/drrg_oclip-resnet50_fpn-unet_1200e_ctw1500.py) |        -         | CTW1500 Train | CTW1500 Test |  1200   |           |           |        |        |                    [model](<>) \\ [log](<>)                    |
 
 ## Citation
 

configs/textdet/drrg/metafile.yml

@@ -25,3 +25,15 @@ Models:
         Metrics:
           hmean-iou: 0.8467
     Weights: https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth
+
+  - Name: drrg_oclip-resnet50_fpn-unet_1200e_ctw1500
+    In Collection: DRRG
+    Config: configs/textdet/drrg/drrg_oclip-resnet50_fpn-unet_1200e_ctw1500.py
+    Metadata:
+      Training Data: CTW1500
+    Results:
+      - Task: Text Detection
+        Dataset: CTW1500
+        Metrics:
+          hmean-iou:
+    Weights:

configs/textdet/fcenet/README.md

@@ -19,12 +19,14 @@ One of the main challenges for arbitrary-shaped text detection is to design a go
 |                       Method                        |     Backbone     | Pretrained Model | Training set  |   Test set   | #epochs |  Test size  | Precision | Recall | Hmean  |                       Download                        |
 | :-------------------------------------------------: | :--------------: | :--------------: | :-----------: | :----------: | :-----: | :---------: | :-------: | :----: | :----: | :---------------------------------------------------: |
 | [FCENet](/configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py) | ResNet50 + DCNv2 |     ImageNet     | CTW1500 Train | CTW1500 Test |  1500   | (736, 1080) |  0.8689   | 0.8296 | 0.8488 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/20220825_221510.log) |
+| [FCENet_oclip-r50](/configs/textdet/fcenet/fcenet_oclip-resnet50-dcnv2_fpn_1500e_ctw1500.py) |  CLIP-ResNet50   |        -         | CTW1500 Train | CTW1500 Test |  1500   |             |           |        |        |               [model](<>) \| [log](<>)                |
 
 ### ICDAR2015
 
 |                          Method                          | Backbone | Pretrained Model | Training set | Test set  | #epochs |  Test size   | Precision | Recall | Hmean  |                          Download                           |
 | :------------------------------------------------------: | :------: | :--------------: | :----------: | :-------: | :-----: | :----------: | :-------: | :----: | :----: | :---------------------------------------------------------: |
 | [FCENet](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py) | ResNet50 |     ImageNet     |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |  0.8243   | 0.8834 | 0.8528 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/20220826_140941.log) |
+| [FCENet_oclip-r50](/configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_icdar2015.py) | ResNet50 |        -         |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |     -     |   -    |   -    |                  [model](<>) \| [log](<>)                   |
 
 ## Citation
 

configs/textdet/fcenet/metafile.yml

@@ -26,6 +26,19 @@ Models:
         Metrics:
           hmean-iou: 0.8488
     Weights: https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth
+
+  - Name: fcenet_oclip-resnet50_fpn_1500e_ctw1500
+    In Collection: FCENet
+    Config: configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_ctw1500.py
+    Metadata:
+      Training Data: CTW1500
+    Results:
+      - Task: Text Detection
+        Dataset: CTW1500
+        Metrics:
+          hmean-iou:
+    Weights:
+
   - Name: fcenet_resnet50_fpn_1500e_icdar2015
     In Collection: FCENet
     Config: configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py
@@ -37,3 +50,15 @@ Models:
         Metrics:
           hmean-iou: 0.8528
     Weights: https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth
+
+  - Name: fcenet_oclip-resnet50_fpn_1500e_icdar2015
+    In Collection: FCENet
+    Config: configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean-iou:
+    Weights:

configs/textdet/maskrcnn/README.md

@@ -19,12 +19,14 @@ We present a conceptually simple, flexible, and general framework for object ins
 |                            Method                            | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                            Download                            |
 | :----------------------------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------------: |
 | [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500.py) |     ImageNet     | CTW1500 Train | CTW1500 Test |   160   |   1600    |  0.7165   | 0.7776 | 0.7458 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/mask-rcnn_resnet50_fpn_160e_ctw1500_20220826_154755-ce68ee8e.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/20220826_154755.log) |
+| [MaskRCNN_oclip-r50](/configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_ctw1500.py) |        -         | CTW1500 Train | CTW1500 Test |   160   |   1600    |           |        |        |                    [model](<>) \| [log](<>)                    |
 
 ### ICDAR2015
 
 |                           Method                           | Pretrained Model |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                           Download                           |
 | :--------------------------------------------------------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :----------------------------------------------------------: |
 | [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py) |     ImageNet     | ICDAR2015 Train | ICDAR2015 Test |   160   |   1920    |  0.8644   | 0.7766 | 0.8182 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/mask-rcnn_resnet50_fpn_160e_icdar2015_20220826_154808-ff5c30bf.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/20220826_154808.log) |
+| [MaskRCNN_oclip-r50](/configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_icdar2015.py) |        -         | ICDAR2015 Train | ICDAR2015 Test |   160   |   1920    |           |        |        |                   [model](<>) \| [log](<>)                   |
 
 ## Citation
 

configs/textdet/maskrcnn/metafile.yml

@@ -28,6 +28,18 @@ Models:
           hmean: 0.7458
     Weights: https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/mask-rcnn_resnet50_fpn_160e_ctw1500_20220826_154755-ce68ee8e.pth
 
+  - Name: mask-rcnn_oclip-resnet50_fpn_160e_ctw1500
+    In Collection: Mask R-CNN
+    Config: configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_ctw1500.py
+    Metadata:
+      Training Data: CTW1500
+    Results:
+      - Task: Text Detection
+        Dataset: CTW1500
+        Metrics:
+          hmean:
+    Weights:
+
   - Name: mask-rcnn_resnet50_fpn_160e_icdar2015
     In Collection: Mask R-CNN
     Config: configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py
@@ -39,3 +51,15 @@ Models:
         Metrics:
           hmean: 0.8182
     Weights: https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/mask-rcnn_resnet50_fpn_160e_icdar2015_20220826_154808-ff5c30bf.pth
+
+  - Name: mask-rcnn_oclip-resnet50_fpn_160e_icdar2015
+    In Collection: Mask R-CNN
+    Config: configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_icdar2015.py
+    Metadata:
+      Training Data: ICDAR2015
+    Results:
+      - Task: Text Detection
+        Dataset: ICDAR2015
+        Metrics:
+          hmean:
+    Weights: