add ABINet [WIP, don't merge yet] #385

safeandnewYH · 2023-06-13T09:26:54Z

Thank you for your contribution to the MindOCR repo.
Before submitting this PR, please make sure:

You have read the Contributing Guidelines on pull requests
Your code builds clean without any errors or warnings
You are using approved terminology
You have added unit tests

Motivation

(Write your motivation for proposed changes here.)

Test Plan

(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)

Related Issues and PRs

(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)

zhtmike

大量文件位置错误，请先修改下~

zhtmike · 2023-06-14T02:36:11Z

configs/rec/abinet/abinet.yaml

@@ -0,0 +1,173 @@
+system:
+  mode: 0 # 0 for graph mode, 1 for pynative mode in MindSpore
+  distribute: true #2023.06.12 modify


clean #2023.06.12 modify

谢谢，已修改。

zhtmike · 2023-06-14T02:36:34Z

configs/rec/abinet/abinet.yaml

+  drop_overflow_update: False
+
+common:
+  character_dict_path: &character_dict_path  #改charset_36.txt mindocr/utils/dict/en_dict.txt


clean #改charset_36.txt

谢谢，已修改。

zhtmike · 2023-06-14T02:37:06Z

configs/rec/abinet/abinet.yaml

+
+model:
+  type: rec
+  pretrained : "/home/gaohan/code/abinet/mindocr_ckpt.ckpt"


path 不要用自己地址，参考crnn config

谢谢，已修改。

数据地址还没有改，为了方便本地调试。

zhtmike · 2023-06-14T02:37:34Z

configs/rec/abinet/abinet.yaml

+  name: RecCTCLabelDecode
+  character_dict_path: *character_dict_path
+  use_space_char: *use_space_char
+#postprocess和metirc需要改


#需要改这种无意义的comment 需要去掉

谢谢，已修改。

zhtmike · 2023-06-14T02:38:06Z

configs/rec/abinet/abinet.yaml

+  opt: adam
+
+
+# loss_scaler:


不需要loss_scaler?

因为没有使用混合精度，所以应该不需要使用loss_scaler。

zhtmike · 2023-06-14T02:59:26Z

mindocr/models/backbones/abinet_utils/utils/ms_utils.py

不属于backbone，请摆在合适位置

谢谢，已修改。

zhtmike · 2023-06-14T02:59:45Z

mindocr/models/backbones/rec_abinet_backbone.py

+        # # {'feature': attn_vecs, 'logits': logits, 'pt_lengths': pt_lengths,
+        # # 'attn_scores': attn_scores, 'loss_weight':self.loss_weight, 'name': 'vision'}
+
+        # all_l_res, all_a_res = [], []


大量comment请删除

谢谢，已修改。

zhtmike · 2023-06-14T03:00:34Z

mindocr/models/backbones/src/PositionAttention.py

src 命名改成 blocks

谢谢，已修改。

zhtmike · 2023-06-14T03:01:26Z

mindocr/models/backbones/src/ms_BaseVision.py

src 命名改成 blocks

谢谢，已修改。

zhtmike · 2023-06-14T03:02:11Z

mindocr/models/backbones/src/ms_loss_new.py

loss 不属于backbone

谢谢，已修改。

safeandnewYH · 2023-06-14T11:58:02Z

主要修改了文件位置和中文注释的问题

Gaohan123 · 2023-06-14T14:07:53Z

mindocr/data/abinet_dataset.py

+
+                return self._next_image(lmdb_idx)
+
+            # 这里的image是PIL.Image类型的，没法用Mindspore直接加载，因此用np转换了一下


这些中文注释请修改

谢谢，已修改。

Gaohan123 · 2023-06-14T14:09:52Z

mindocr/data/abinet_dataset.py

+                    warnings.simplefilter("ignore", UserWarning)  # EXIF warning from TiffPlugin
+                    image = PIL.Image.open(buf).convert(self.convert_mode)
+                if self.is_training and not self._check_image(image):
+                    # logging.info(f'Invalid image is found: {self.name}, {idx}, {label}, {len(label)}')


注释掉的代码块，如果不需要，请删除

谢谢，已修改。

Gaohan123 · 2023-06-14T14:11:38Z

mindocr/models/rec_abinet.py

+            "name": "ABINet_Head",
+        },
+    }
+    model = ABINet_Model_Test(model_config)


这个为什么加test？不如直接叫模型的名字

谢谢，已修改。

Gaohan123 · 2023-06-15T02:05:17Z

configs/rec/abinet/abinet.yaml

+  dataset:
+    type: ABINet_Dataset
+    path: [/home/gaohan/data/abinet_data/train/MJ/MJ_test,'/home/gaohan/data/abinet_data/train/MJ/MJ_train','/home/gaohan/data/abinet_data/train/MJ/MJ_valid','/home/gaohan/data/abinet_data/train/ST']
+    dataset_root: /home/gaohan/data/abinet_data/train/ # Optional, if set, dataset_root will be used as a prefix for data_dir


数据集路径记得改成通用的。自己调试在本地改回去即可

谢谢，已修改。

zhtmike

另外请提供对应的README文件

zhtmike · 2023-06-15T02:01:19Z

mindocr/data/abinet_dataset.py

+__all__ = ["ABINet_Dataset"]
+
+
+class ABINet_Dataset:


ABINet_Dataset 读图内容为 mindocr/data/rec_lmdb_dataset.py 中的LMDBDataset格式，请用LMDBDataset 代替。label转换，图片normalization 等皆用现有的transform内容代替。可参考CRNN

如果是指ABINet_Dataset类的话，因为在__getitem__方法中做了特殊处理，因此没法直接用LMDBDataset 代替，不过应该有可以简化代码的地方，后续可以进行修改。

重合的地方可以继承复用，做了特殊处理可以覆盖

zhtmike · 2023-06-15T02:02:37Z

mindocr/data/transforms/abinet_transforms.py

+__all__ = ["ABINet_Transforms"]
+
+
+class ABINet_Transforms(object):


class命名无需下划线

谢谢，已修改。

zhtmike · 2023-06-15T02:04:31Z

mindocr/models/backbones/rec_abinet_backbone.py

@@ -0,0 +1,4626 @@
+from __future__ import absolute_import, division


本身仅支持python3.7+，不需要 from __future__ import absolute_import, division

谢谢，已修改。

zhtmike · 2023-06-15T02:07:16Z

mindocr/models/backbones/rec_abinet_backbone.py

+_default_tfmer_cfg = dict(
+    d_model=512, nhead=8, d_inner=2048, dropout=0.1, activation="relu"  # 1024
+)
+__all__ = [


只需要 export ABINetIter_Backbone 和 abinet_backbone，其他不要再__all__里面。 ABINetIter_Backbone 去除下划线

谢谢，已修改。

zhtmike · 2023-06-15T02:11:05Z

mindocr/models/backbones/rec_abinet_backbone.py

大量内容和mindspore.nn.transformer及mindspore.nn （例如mindspore.nn.MultiheadAttention）极为相似，请直接import mindspore对应module

开发过程中，发现mindspore.nn.transformer中的内容不能直接使用，比如TransformerEncoder和TransformerDecoder，现有功能不满足模型的需求。因此在mindspore.transformer的基础上进行了改动以满足模型需求，因此虽然相似但是没法直接import。MultiheadAttention也许可以import，我忘了当初有没有改这个了，如果发现没有修改可以import。

嗯请复用code，使用继承等方式修改需要改的地方，避免大量copy paste

好的，后续我查一下代码，可以精简的地方简化一下。

zhtmike · 2023-06-15T02:14:58Z

mindocr/models/necks/abinet_neck.py

+import mindspore as ms
+from mindspore import nn
+
+from ..backbones.rec_abinet_backbone import Model, PositionalEncoding


同样不应该引用backbone内容

同上述问题

zhtmike · 2023-06-15T02:16:03Z

mindocr/models/rec_abinet.py

+    }
+    model = ABINet_Model(model_config)
+
+    # load pretrained weights


pretrain那段需保留，请按crnn方式修改

谢谢，已修改。

zhtmike · 2023-06-15T02:16:16Z

mindocr/models/rec_abinet.py

+
+default_cfgs = {
+    # 'abinet':
+    "crnn_resnet34": _cfg(


内容不对

因为现在还没有上传abinet的ckpt到mindspore上，所以此处只是做了个标记，后续会更改url的内容。

zhtmike · 2023-06-15T02:17:21Z

mindocr/utils/dict/charset_36.txt

@@ -0,0 +1,36 @@
+0	a
+1	b


这字典跟default字典重叠，不需要额外提供，把character_dict_path设成none即可，参考crnn

此模型最好需要一个这样格式的dict，虽然默认的dict包含更多字符，但还需额外处理。上面有问题提到可以把需要复用的代码放到一个文件夹里，这个dict也可以作为此模型专用的字典放置进去。

谢谢，已修改。

zhtmike · 2023-06-15T02:17:49Z

requirements.txt

@@ -16,3 +16,4 @@ xml-python>=0.4.3
 packaging
 Cython
 lanms>=1.0.2
+pandas


哪里需要用到pandas?

谢谢，已修改。

safeandnewYH · 2023-06-16T06:41:47Z

根据意见已修改，README暂时还未添加。

hqkate · 2023-06-16T07:11:34Z

configs/rec/abinet/abinet.yaml

+
+model:
+  type: rec
+  pretrained : "/tmp_rec/pretrain/mindocr_ckpt.ckpt"


建议补充说明下这个pretrained模型怎么准备

谢谢，后续在README中会添加。

hqkate · 2023-06-16T07:14:55Z

mindocr/data/abinet_dataset.py

+__all__ = ["ABINet_Dataset"]
+
+
+class ABINet_Dataset:


重合的地方可以继承复用，做了特殊处理可以覆盖

hqkate · 2023-06-16T07:16:53Z

mindocr/losses/abinet_loss.py

+
+        if i == 2:
+            pt_logits = output["logits"]
+            weight = 1.0


if条件底下的 weight = 1.0 是不是可以删掉

谢谢，已修改。

hqkate · 2023-06-16T07:19:44Z

mindocr/models/backbones/rec_abinet_backbone.py

+    # load pretrained weights
+    if pretrained:
+        raise NotImplementedError
+


NotImplementedError 可以加一下message哈，可参考rec_resnet.py

谢谢，已修改。

hqkate · 2023-06-16T07:23:49Z

mindocr/models/backbones/rec_abinet_backbone.py

+        self.loss_weight = 1.0
+        self.out_channels = 512
+        self.backbone = ResTranformer()
+        mode = "nearest"


mode 变量只在48行用到，可直接在48行 mode='nearest'

谢谢，已修改。

hqkate · 2023-06-16T07:25:54Z

mindocr/models/necks/abinet_neck.py

+        zeroo = ms.ops.Zeros()
+        zeros = zeroo((96, 26, 512), ms.float32)
+        zeros = zeros.transpose(1, 0, 2)
+        qeury = self.pos_encoder(zeros)


变量名是否改成 query 更准确?

谢谢，已修改。

safeandnewYH · 2023-06-17T10:47:10Z

已删掉abinet_dataset

safeandnewYH · 2023-06-17T16:07:11Z

已尽量复用数据变换部分，Transformer部分留下了TransformerEncoder、TransformerDecoder、TransformerEncoderLayer、TransformerDecoderLayer部分，因为这四部分相对于mindspore.nn.transformer做了修改。

SamitHuang · 2023-06-27T14:44:58Z

configs/rec/abinet/README.md

+
+| **Model** | **Context** | **Avg Accuracy** | **Train T.** | **FPS** | **Recipe** | **Download** |
+| :-----: | :-----------: | :--------------: | :----------: | :--------: | :--------: |:----------: |
+| ABINet      | D910x8-MS1.9-G | 92.23%    | 20076 s/epoch       | 99 | [yaml](https://github.com/safeandnewYH/ABINet_Figs/blob/main/abinet.yaml) | [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en_34aeaf96.ckpt)


权重文件命名不符合规范，请更正：
abinet_resnet45_en-34aeaf96.ckpt
en后是-，不是_.
另外，yaml的文件名应与 ckpt保持一致：abinet_resnet45_en.yaml

已修改，谢谢。

SamitHuang · 2023-06-27T14:45:17Z

configs/rec/abinet/README_CN.md

+
+| **Model** | **Context** | **Avg Accuracy** | **Train T.** | **FPS** | **Recipe** | **Download** |
+| :-----: | :-----------: | :--------------: | :----------: | :--------: | :--------: |:----------: |
+| ABINet      | D910x8-MS1.9-G | 92.23%    | 20076 s/epoch       | 99 | [yaml](https://github.com/safeandnewYH/ABINet_Figs/blob/main/abinet.yaml) | [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en_34aeaf96.ckpt)


已修改，谢谢。

SamitHuang · 2023-06-27T15:29:53Z

tools/benchmarking/multi_dataset_eval.py

@@ -43,6 +43,9 @@
 import os
 import sys

+import mindspore as ms
+
+ms.set_context(device_id=5)


device_id=5 删除

已修改，谢谢。

zhtmike · 2023-06-28T03:56:15Z

configs/rec/abinet/abinet.yaml

ABINet 是使用了预训练模型了吗，我看config没有？

已添加，谢谢。

怎么产生预训练模型和如何使用，README需要补充下

预训练模型从原作者repo中得到，会在README补充如何使用。

zhtmike · 2023-06-29T06:43:34Z

configs/rec/abinet/README_CN.md

+
+在这里，我们使用 `train/` 文件夹下的数据集进行训练，我们使用 `evaluation/` 下的数据集来评估模型的准确性。
+
+**Train:**


数据集大小需要补充下，参考CRNN

已补充，谢谢。

zhtmike · 2023-06-29T07:37:34Z

tools/benchmarking/multi_dataset_eval.py

@@ -86,7 +88,7 @@ def main(cfg):
            )
        else:
            device_id = cfg.system.get("device_id", 0)
-            ms.set_context(device_id=device_id)
+            # ms.set_context(device_id=device_id)


这两个修改要改回去

已修改，谢谢。

zhtmike · 2023-06-29T07:43:30Z

mindocr/losses/abinet_loss.py

+        super().__init__()
+        self.ce = SoftCrossEntropyLoss()
+        self.bce = nn.BCELoss(reduction="mean")
+        self.losses = ms.Parameter([])


self.losses 还有 last_losses 似乎没用到，否则可删去

已修改，谢谢。

safeandnewYH · 2023-06-30T03:53:59Z

已按照意见修改。

HaoyangLee · 2023-06-30T09:26:47Z

configs/rec/abinet/README.md

+
+| **Model** | **Context** | **Avg Accuracy** | **Train T.** | **FPS** | **Recipe** | **Download** |
+| :-----: | :-----------: | :--------------: | :----------: | :--------: | :--------: |:----------: |
+| ABINet      | D910x8-MS1.9-G | 92.53%    | 22993 s/epoch       | 86 | [yaml](https://github.com/safeandnewYH/ABINet_Figs/blob/main/abinet_resnet45_en.yaml) | [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en-41e4bbd0.ckpt)


中文readme同步修改

已修改，谢谢。

Songyuanwei · 2023-07-03T08:43:08Z

mindocr/data/transforms/rec_abinet_transforms.py

+    CVRescale,
+)
+
+_logger = Logger("mindocr")


这个地方需要修改一下， Logger这个函数已经去掉了，所以需要删掉from mindocr.utils.logger import Logger，参考其他脚本，使用import logging, _logger = logging.getLogger(__name__)打印信息

HaoyangLee · 2023-06-30T09:32:34Z

configs/rec/abinet/README.md

+
+| **Model** | **Context** | **Avg Accuracy** | **Train T.** | **FPS** | **Recipe** | **Download** |
+| :-----: | :-----------: | :--------------: | :----------: | :--------: | :--------: |:----------: |
+| ABINet      | D910x8-MS1.9-G | 92.53%    | 22993 s/epoch       | 86 | [yaml](https://github.com/safeandnewYH/ABINet_Figs/blob/main/abinet_resnet45_en.yaml) | [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en-41e4bbd0.ckpt)


原值是单卡FPS，此处填写八卡FPS，约等于单卡FPS乘以8。中文readme同步修改

hadipash · 2023-07-07T10:02:35Z

mindocr/data/transforms/rec_abinet_transforms.py

+import cv2
+import numpy as np
+import PIL
+import six


Do not use six! six is needed for backward compatibility with python 2.x. Our minimum supported version of python is 3.7!

HaoyangLee requested review from SamitHuang, zhtmike, HaoyangLee and hqkate June 14, 2023 02:23

zhtmike requested changes Jun 14, 2023

View reviewed changes

safeandnewYH force-pushed the main branch 4 times, most recently from e94f7a1 to a69c5d5 Compare June 14, 2023 11:40

Gaohan123 reviewed Jun 14, 2023

View reviewed changes

safeandnewYH force-pushed the main branch from a69c5d5 to f311008 Compare June 14, 2023 14:50

Gaohan123 reviewed Jun 15, 2023

View reviewed changes

zhtmike requested changes Jun 15, 2023

View reviewed changes

safeandnewYH force-pushed the main branch 3 times, most recently from 88491b2 to d0026eb Compare June 15, 2023 11:01

hqkate reviewed Jun 16, 2023

View reviewed changes

safeandnewYH force-pushed the main branch from b41a6d3 to 6d79f24 Compare June 17, 2023 05:38

safeandnewYH force-pushed the main branch 2 times, most recently from 4cd3f1c to f90d7f7 Compare June 17, 2023 14:52

zhtmike requested a review from hadipash June 19, 2023 08:44

safeandnewYH force-pushed the main branch 3 times, most recently from ab07941 to ea5e2ba Compare June 19, 2023 14:31

abinet added 3 commits June 26, 2023 14:12

Add ABINet 0626

e7751a4

Add ABINet 0626

92b9e8e

Add ABINet 0626

7215233

SamitHuang reviewed Jun 27, 2023

View reviewed changes

zhtmike reviewed Jun 28, 2023

View reviewed changes

Add ABINet 0628

dcdf056

zhtmike reviewed Jun 29, 2023

View reviewed changes

safeandnewYH force-pushed the main branch from a6ffdd2 to d5eafdb Compare June 30, 2023 03:14

abinet and others added 8 commits June 30, 2023 03:14

Add ABINet 0629

c8c2133

Add ABINet 0629

d5eafdb

Merge branch 'main' into main

e88dea7

Add ABINet 0630

c8153db

Add ABINet 0630

34ae07d

Add ABINet 0630

13ed298

Update ABINet README.md

274beac

Update ABINet README_CN.md

aa3823a

HaoyangLee reviewed Jun 30, 2023

View reviewed changes

safeandnewYH added 2 commits June 30, 2023 17:47

Update ABINet README.md

51456a3

Update ABINet README_CN.md

92fd1d0

Songyuanwei reviewed Jul 3, 2023

View reviewed changes

HaoyangLee reviewed Jul 5, 2023

View reviewed changes

HaoyangLee and others added 4 commits July 7, 2023 11:08

Update README.md

2323870

Update README_CN.md

b3f30d1

Update rec_abinet_transforms.py

56b669e

Merge branch 'main' into main

3b0c61e

HaoyangLee merged commit 96fff90 into mindspore-lab:main Jul 7, 2023

hadipash reviewed Jul 7, 2023

View reviewed changes

kk928290341 mentioned this pull request Jul 24, 2024

[630][Model] ABINet模型支持 #231

Closed


		return self._next_image(lmdb_idx)

		# 这里的image是PIL.Image类型的，没法用Mindspore直接加载，因此用np转换了一下

		__all__ = ["ABINet_Transforms"]


		class ABINet_Transforms(object):

		@@ -0,0 +1,4626 @@
		from __future__ import absolute_import, division


		在这里，我们使用 `train/` 文件夹下的数据集进行训练，我们使用 `evaluation/` 下的数据集来评估模型的准确性。

		Train:

	\| ABINet \| D910x8-MS1.9-G \| 92.53% \| 22993 s/epoch \| 86 \| [yaml](https://github.com/safeandnewYH/ABINet_Figs/blob/main/abinet_resnet45_en.yaml) \| [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en-41e4bbd0.ckpt)
	\| ABINet \| D910x8-MS1.9-G \| 92.53% \| 22993 s/epoch \| 86 \| [yaml](abinet_resnet45_en.yaml) \| [ckpt](https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_resnet45_en-41e4bbd0.ckpt)

add ABINet [WIP, don't merge yet] #385

add ABINet [WIP, don't merge yet] #385

Uh oh!

Conversation

safeandnewYH commented Jun 13, 2023

Motivation

Test Plan

Related Issues and PRs

Uh oh!

zhtmike left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

safeandnewYH commented Jun 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhtmike left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

zhtmike Jun 15, 2023 •

edited

Loading