Add Seq2Seq Chinese model support #289

zhtmike · 2023-05-15T09:39:48Z

Add Chinese model configure file
Support pretrained weight with backbone only. Use remove_prefix=True to load pretrained model from MindOCR
Support relative path of dictionary in modelArt
Add Documentation of training custom dataset
Merge the documentation of Chinese dataset from Add Chinese CRNN support #280

Thank you for your contribution to the MindOCR repo.
Before submitting this PR, please make sure:

You have read the Contributing Guidelines on pull requests
Your code builds clean without any errors or warnings
You are using approved terminology
You have added unit tests

Motivation

(Write your motivation for proposed changes here.)

Test Plan

(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)

Related Issues and PRs

(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)

SamitHuang

Looks good overall! Some improvements are needed before merge as mentioned in comments.

SamitHuang · 2023-05-17T17:42:41Z

docs/cn/tutorials/training_recognition_custom_dataset_CN.md

+## 字典准备
+
+为训练中、英文等不同语种的识别网络，用户需配置对应的字典。只有存在于字典中的字符会被模型正确预测。MindOCR现提供中、英两种字典，其中
+- `英文字典`：包括大小写英文、数字和标点符号。存放于`mindocr/utils/dict/en_dict.txt`


目前已训练的crnn英文识别模型(trained on MJ+ST) 所用的字典是默认的小写字母+数字，未用到这个字典？
若在自定义数据集上改用这个英文字典，其他超参如num_classes, use_space_char 也要说明如何调整。

已修改为新的字典，num_classes, use_space_char 说明已更新。

SamitHuang · 2023-05-17T17:44:14Z

docs/cn/tutorials/training_recognition_custom_dataset_CN.md

+- `英文字典`：包括大小写英文、数字和标点符号。存放于`mindocr/utils/dict/en_dict.txt`
+- `中文字典`：包括常用中文字符、大小写英文、数字和标点符号。存放于`mindocr/utils/dict/ch_dict.txt`
+
+目前MindOCR暂未提供自定义字典配置。该功能将在新版本中推出。


可否通过指定character_dict_path，并修改num_classes，来使用自定义字典并微调训练？

已提供修改字典字符教学。

SamitHuang · 2023-05-17T17:47:32Z

docs/cn/tutorials/training_recognition_custom_dataset_CN.md

+```yaml
+...
+common:
+  character_dict_path: &character_dict_path mindocr/utils/dict/en_dict.txt


此处character_dict_path跟crnn_resnet34.yaml中的配置不一样，num_claases需同步调整，不然跑训练可能会出错。

这个初始配置为crnn_resnet34_ch.yaml，在PR #280 , num_claases应该正确

SamitHuang · 2023-05-17T17:49:04Z

mindocr/models/backbones/builder.py

@@ -53,7 +52,12 @@ def build_backbone(name, **kwargs):
    if 'pretrained' in kwargs:
        pretrained = kwargs['pretrained']
        if not isinstance(pretrained, bool):
-            load_model(backbone, pretrained)
+            if remove_prefix:


可否用 load_model中已有的auto_mapping选项来自动映射？

应该不行，auto_mapping会找最相近的matches，但如果想在backbone load minocr 模型的话需要把前缀backbone.去掉，差异会有9个字符

SamitHuang · 2023-05-17T17:50:46Z

mindocr/models/backbones/rec_resnet.py


 @register_backbone
 def rec_resnet34(pretrained: bool = True, **kwargs):
    model = RecResNet(in_channels=3, layers=34, **kwargs)

-    # load pretrained weights
-    if pretrained:
+    if pretrained is True:
        raise NotImplementedError


建议给出更详细的报错信息，如 Pretrained checkpoint for rec_resnet34 does not exist.

zhtmike · 2023-05-18T09:12:45Z

有两个问题

英文字典和中文字典不统一。英文字典含有空格，中文字典不包含空格，什么时候用use_space_char会有混淆。
num_classes应该自动产生，否则要根据字典字符数量、use_space_char 还有模型类别决定，用户用会比较困难去理解

SamitHuang · 2023-05-18T09:18:15Z

字典文件里，不应该包含空格。统一用 use_space_char来指定添加空格支持。
num_classes 确实可计算出来（如设为空 / Null)，自动计算。

zhtmike · 2023-05-18T09:29:47Z

字典文件里，不应该包含空格。统一用 use_space_char来指定添加空格支持。

num_classes 确实可计算出来（如设为空 / Null)，自动计算。

已在英文字典去除空格。2 涉及改动较大，建议另开一个PR

HaoyangLee · 2023-05-18T10:01:58Z

docs/cn/tutorials/training_recognition_custom_dataset_CN.md

+word_1657.png	你好
+word_1814.png	cathay
+```
+*注意*：请将图片名和标签以 \tag 作为分隔，避免使用空格或其他分隔符。


此处应为\tab或\t

已修改。

HaoyangLee · 2023-05-18T10:02:17Z

docs/en/tutorials/training_recognition_custom_dataset.md

+word_1657.png	你好
+word_1814.png	cathay
+```
+*Note*: Please separate image names and labels using \tag, and avoid using spaces or other delimiters.


此处应为\tab或\t

已修改。

HaoyangLee · 2023-05-18T10:07:43Z

configs/rec/rare/README_CN.md

line19参考文献的引用应修改为："#references" -> "#参考文献"

已修改。

HaoyangLee · 2023-05-18T10:29:35Z

configs/rec/rare/rare_resnet34_ch.yaml

+  backbone:
+    name: rec_resnet34
+    pretrained: https://download.mindspore.cn/toolkits/mindocr/rare/rare_resnet34-309dc63e.ckpt
+    remove_prefix: True


此处建议添加remove_prefix参数的注释，不然用户有点难理解

已加注释

HaoyangLee · 2023-05-18T10:43:37Z

docs/cn/datasets/chinese_text_recognition_CN.md

docs/下面tutorials/和datasets/中更新的文档是否需要在外部文档（如主页README，或RARE模型的REAMDE）给一个链接引过来？否则用户找不到这几个文档

RARE模型的README已提供连接，等下CRNN 的PR也会指过去。

zhtmike requested review from SamitHuang and hqkate May 15, 2023 09:39

zhtmike force-pushed the seq2seq branch 4 times, most recently from 5cb632c to 9120f86 Compare May 17, 2023 06:50

seq2seq support chinese and add documentation

1b1e9ff

zhtmike force-pushed the seq2seq branch from 93c23a9 to 1b1e9ff Compare May 17, 2023 06:55

zhtmike changed the title ~~Seq2Seq Chinese model configure file~~ Add Seq2Seq Chinese model support May 17, 2023

zhtmike marked this pull request as ready for review May 17, 2023 07:13

zhtmike requested a review from HaoyangLee May 17, 2023 07:13

fix url and export function

c32039b

SamitHuang reviewed May 17, 2023

View reviewed changes

Update tutorial and detail error message

08bc228

zhtmike added 2 commits May 18, 2023 17:27

remove space in english dictionary

33d1bfe

fix number

a00e445

SamitHuang approved these changes May 18, 2023

View reviewed changes

HaoyangLee reviewed May 18, 2023

View reviewed changes

fix typo

7dade83

HaoyangLee approved these changes May 18, 2023

View reviewed changes

HaoyangLee merged commit 17a03bf into mindspore-lab:main May 18, 2023

zhtmike deleted the seq2seq branch May 24, 2023 05:53

Add Seq2Seq Chinese model support #289

Add Seq2Seq Chinese model support #289

Uh oh!

Conversation

zhtmike commented May 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Test Plan

Related Issues and PRs

Uh oh!

SamitHuang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhtmike commented May 18, 2023

Uh oh!

SamitHuang commented May 18, 2023

Uh oh!

zhtmike commented May 18, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhtmike commented May 15, 2023 •

edited

Loading