Skip to content

use default ser ckpt in vi_layoutxlm_ser #789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 30, 2024

Conversation

hongziqi
Copy link
Contributor

@hongziqi hongziqi commented Dec 26, 2024

Thank you for your contribution to the MindOCR repo.
Before submitting this PR, please make sure:

Motivation

  1. update register_model: vi_layoutxlm_ser to use pretrain ckpt
  2. fix bug by inferring ser in layoutxlm_ser:
    [ERROR] ANALYZER(2921322,ffffa6185010,python):2024-12-30-08:59:24.113.844 [mindspore/ccsrc/pipeline/jit/ps/static_analysis/evaluator.cc:713] Run] Primitive: <StandardPrimEvaluator_list_getitem> infer failed, failed info: list_getitem evaluator index should be in range[-4, 4), but got 4.

Before Update(would not load ckpt in mindocr/mindocr/models/kie_layoutxlm.py

  1. infer with vi_layoutxlm:
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg

image

  1. infer with layoutxlm, the following bugs have occurred:
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg --ser_algorithm LAYOUTXLM

image

After Update(use the best ckpt link)

  1. infer with vi_layoutxlm:
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg

image

  1. infer with layoutxlm:
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg --ser_algorithm LAYOUTXLM

image

  1. eval with vi_layoutxlm:
python tools/eval.py --config configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yaml

image

  1. eval with layoutxlm:
python tools/eval.py --config configs/kie/layoutxlm/ser_layoutxlm_xfund_zh.yaml

image

  1. train with vi_layoutxlm:
python tools/train.py --config configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yaml

image
image

  1. train with layoutxlm:
python tools/train.py --config configs/kie/layoutxlm/ser_layoutxlm_xfund_zh.yaml

image
image

Test Plan

test for

# download mindocr
git clone -b feat-update-kie-ckpt-link https://github.com/hongziqi/mindocr.git
# build mindocr wheel package and install mindocr
python setup.py bdist_wheel
pip install dist/mindocr-0.4.0-py3-none-any.whl

# infer with VI_LAYOUTXLM
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg

# infer with LAYOUTXLM
python tools/infer/text/predict_ser.py --rec_algorithm CRNN_CH --det_algorithm DB_PPOCRv3 --image_dir configs/kie/vi_layoutxlm/example.jpg --ser_algorithm LAYOUTXLM

# download dataset
wget https://download.mindspore.cn/toolkits/mindocr/vi-layoutxlm/XFUND.tar && tar -xf XFUND.tar
# train or eval (need to update the path in yaml, please refer to the file: configs/kie/vi_layoutxlm/README_CN.md)
python tools/train.py(eval.py) --config configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yaml
python tools/train.py(eval.py) --config configs/kie/layoutxlm/ser_layoutxlm_xfund_zh.yaml

@hongziqi hongziqi force-pushed the feat-update-kie-ckpt-link branch 5 times, most recently from 3dc446c to 51a64dc Compare December 30, 2024 01:31
@hongziqi hongziqi changed the title update default ser ckpt link use default ser ckpt in vi_layoutxlm_ser Dec 30, 2024
load_pretrained(model, default_cfg)

return model


@register_model
def vi_layoutxlm_ser(pretrained: bool = True, use_visual_backbone: bool = False, use_float16: bool = False, **kwargs):
def vi_layoutxlm_ser(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

训练脚本是不是也要适配

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不影响训练脚本,训练配置的是backbone的pretrained参数,这里并没有改动backbone网络的参数。

@hongziqi hongziqi force-pushed the feat-update-kie-ckpt-link branch from f2159db to ad34340 Compare December 30, 2024 02:49
@CaitinZhao CaitinZhao merged commit 62716ec into mindspore-lab:main Dec 30, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants