Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support MultiScaleDeformableAttn with cambricon MLU backend #2396

Merged
merged 27 commits into from
Nov 16, 2022
Merged

[Feature] Support MultiScaleDeformableAttn with cambricon MLU backend #2396

merged 27 commits into from
Nov 16, 2022

Conversation

ZShaopeng
Copy link
Contributor

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

The motivation of the PR is to support running MultiScaleDeformAttn with Cambricon MLU backend.

It includes five parts:

1. Add ms_deform_attn_mlu_kernel.mlu src code as MLU kernel.
2. Add ms_deform_attn_mlu.cpp to support launching kernel in PyTorch.
3. Modify common_mlu_helper.hpp to add common function.
4. Refactor multi_scale_deform_attn.py to support MLU backend.
5. Refactor test_ms_deformable_attn.py to support testing MultiScaleDeformAttn with MLU backend.

Modification

  • MLU src code
    Add MLU src code of MultiScaleDeformAttn in directory:
    mmcv/ops/csrc/common/mlu/ms_deform_attn_mlu_kernel.mlu,
    mmcv/ops/csrc/common/mlu/common_mlu_helper.hpp.
  • PyTorch adaptation
    Adapt MultiScaleDeformAttn for PyTorch in directory:
    mmcv/ops/csrc/pytorch/mlu/ms_deform_attn_mlu.cpp,
    mmcv/ops/multi_scale_deform_attn.py.
  • Unit test
    Support test MultiScaleDeformAttn with various backends in test_ms_deformable_attn.py.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Before PR:

  • I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
  • Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
  • Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
  • New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
  • CLA has been signed and all committers have signed the CLA in this PR.

@CLAassistant
Copy link

CLAassistant commented Nov 8, 2022

CLA assistant check
All committers have signed the CLA.

@ZShaopeng ZShaopeng changed the title [Feature] Support MsDeformAttnForward with cambricon MLU backend [Feature] Support MsDeformAttn with cambricon MLU backend Nov 8, 2022
@grimoire
Copy link
Member

grimoire commented Nov 9, 2022

Unit test test_gradient_numerical gradcheck failed.

@zhouzaida zhouzaida added the MLU label Nov 9, 2022
@zhouzaida zhouzaida changed the title [Feature] Support MsDeformAttn with cambricon MLU backend [WIP] Support MsDeformAttn with cambricon MLU backend Nov 11, 2022
@zhouzaida zhouzaida changed the title [WIP] Support MsDeformAttn with cambricon MLU backend [Feature] Support MsDeformAttn with cambricon MLU backend Nov 15, 2022
Copy link
Member

@grimoire grimoire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhouzaida zhouzaida changed the title [Feature] Support MsDeformAttn with cambricon MLU backend [Feature] Support MultiScaleDeformableAttn with cambricon MLU backend Nov 16, 2022
@zhouzaida zhouzaida merged commit 1244266 into open-mmlab:master Nov 16, 2022
@ZShaopeng ZShaopeng deleted the ms_deform_attn branch February 8, 2023 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants