Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x #2215

Merged
merged 23 commits into from
Dec 1, 2022
Merged

[Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x #2215

merged 23 commits into from
Dec 1, 2022

Conversation

MengzhangLI
Copy link
Contributor

@MengzhangLI MengzhangLI commented Oct 20, 2022 •
edited
Loading

Add MaskFormer(NeurIPS'2021) in dev-1.x.

image

Paper: https://arxiv.org/abs/2107.06278
Repo: https://github.com/facebookresearch/MaskFormer

  • Align Inference Metric
  • Align Training Metric
  • Add Unit Test

Results about Aligning Inference metric

(1) Using maskformer_R50_bs16_160k, the inference metric has been aligned.

image

image

Results about Aligning training metric

WIP.

Li-Qingyun
Li-Qingyun reviewed Oct 21, 2022
View reviewed changes
configs/maskformer/maskformer_r50_512x512_80k_ade20k.py Outdated
Comment on lines 49 to 76
transformer_decoder=dict(
type='DetrTransformerDecoder',
return_intermediate=True,
num_layers=6,
transformerlayers=dict(
type='DetrTransformerDecoderLayer',
attn_cfgs=dict(
type='MultiheadAttention',
embed_dims=256,
num_heads=8,
attn_drop=0.1,
proj_drop=0.1,
dropout_layer=None,
batch_first=False),
ffn_cfgs=dict(
embed_dims=256,
feedforward_channels=2048,
num_fcs=2,
act_cfg=dict(type='ReLU', inplace=True),
ffn_drop=0.1,
dropout_layer=None,
add_identity=True),
# the following parameter was not used,
# just make current api happy
feedforward_channels=2048,
operation_order=('self_attn', 'norm', 'cross_attn', 'norm',
'ffn', 'norm')),
init_cfg=None),
Copy link
Contributor

@Li-Qingyun Li-Qingyun Oct 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Mr. Li, my name is Qingyun and I participated in refactoring the DETR-like algorithm of mmdet.
We have deconstructed the registry mechanism to build DetrTransformerDecoder in the new implementation in open-mmlab/mmdetection#8763 (The PR has been merged into refactor-detr branch of official mmdetection repo, and will be merge to 3.x when other DETRs are refactored).
I would like to remind you that the logic of building decoder and the config may need to be modified.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Mr. Li, my name is Qingyun and I participated in refactoring the DETR-like algorithm of mmdet. We have deconstructed the registry mechanism to build DetrTransformerDecoder in the new implementation in open-mmlab/mmdetection#8763 (The PR has been merged into refactor-detr branch of official mmdetection repo, and will be merge to 3.x when other DETRs are refactored). I would like to remind you that the logic of building decoder and the config may need to be modified.

Thank you qingyun, very useful information for us. ;)

@MengzhangLI MengzhangLI changed the title [Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x [WIP][Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x Oct 26, 2022
@MeowZheng MeowZheng added this to the 2.0.0rc2 milestone Oct 31, 2022
@MeowZheng MeowZheng added the 1.x Related issue of 1.x version label Nov 1, 2022
MeowZheng
MeowZheng reviewed Nov 1, 2022
View reviewed changes
Copy link
Collaborator

@MeowZheng MeowZheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might check both of 2 params
https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/configs/ade20k-150/swin/maskformer_swin_tiny_bs16_160k.yaml#L21-L22

@MeowZheng MeowZheng deleted the branch open-mmlab:dev-1.x November 2, 2022 10:34
@MeowZheng MeowZheng closed this Nov 2, 2022
@MeowZheng MeowZheng reopened this Nov 2, 2022
@MengzhangLI MengzhangLI changed the title [WIP][Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x [Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x Nov 15, 2022
@codecov
Copy link

codecov bot commented Nov 29, 2022 •
edited
Loading

Codecov Report

Base: 83.18% // Head: 83.29% // Increases project coverage by +0.11% 🎉

Coverage data is based on head (1cb5359) compared to base (925faea).
Patch coverage: 93.84% of modified lines in pull request are covered.

Additional details and impacted files 
@@             Coverage Diff             @@
##           dev-1.x    #2215      +/-   ##
===========================================
+ Coverage    83.18%   83.29%   +0.11%     
===========================================
  Files          141      142       +1     
  Lines         8003     8068      +65     
  Branches      1200     1205       +5     
===========================================
+ Hits          6657     6720      +63     
- Misses        1160     1161       +1     
- Partials       186      187       +1
Flag Coverage Δ
unittests 83.29% <93.84%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/models/decode_heads/maskformer_head.py 93.75% <93.75%> (ø)
mmseg/models/decode_heads/__init__.py 100.00% <100.00%> (ø)
mmseg/models/backbones/timm_backbone.py 45.83% <0.00%> (+8.33%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@MengzhangLI MengzhangLI force-pushed the Add_MaskFormer_2.0 branch 2 times, most recently from cc564e1 to 341a7dd Compare December 1, 2022 02:40
MeowZheng
MeowZheng reviewed Dec 1, 2022
View reviewed changes
mmseg/models/decode_heads/maskformer_head.py Show resolved Hide resolved
requirements/mminstall.txt Outdated Show resolved Hide resolved
requirements/runtime.txt Outdated Show resolved Hide resolved
@MeowZheng MeowZheng merged commit 933e4d3 into open-mmlab:dev-1.x Dec 1, 2022
@MengzhangLI MengzhangLI mentioned this pull request Dec 2, 2022
Closed
4 tasks
@Li-Qingyun Li-Qingyun mentioned this pull request Dec 19, 2022
Merged
MeowZheng pushed a commit to MeowZheng/mmsegmentation that referenced this pull request Dec 30, 2022
@MengzhangLI @MeowZheng
[Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x (open-mmlab#2215
d487dd8 
)

* [Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x

* add mmdet try except logic

* refactor config files

* add readme

* fix config

* update models & logs

* add MMDET installation and fix info

* fix comments

* fix

* fix config norm optimizer setting

* update models & logs & unittest

* add docstring of MaskFormerHead

* wait for mmdet 3.0.0rc4

* replace seg_mask with seg_logits & add docstring for batch_input_shape

* use mmdet3.0.0rc4

* fix readme and modify config comments

* add mmdet installation in pr_stage_test.yml

* update mmcv version in pr_stage_test.yml

* add mmdet in build_cpu of pr_stage_test.yml

* modify mmdet& mmcv installation in merge_stage_test.yml

* fix typo

* update test.yml

* update test.yml
@MengzhangLI MengzhangLI deleted the Add_MaskFormer_2.0 branch January 9, 2023 14:31
aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023
@chavinlo @patrickvonplaten
Create convert_vae_pt_to_diffusers.py (open-mmlab#2215)
717a956 
* Create convert_vae_pt_to_diffusers.py

Just a simple script to convert VAE.pt files to diffusers format
Tested with: https://huggingface.co/WarriorMama777/OrangeMixs/blob/main/VAEs/orangemix.vae.pt

* Update convert_vae_pt_to_diffusers.py

Forgot to add the function call

* make style

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: chavinlo <example@example.com>
nahidnazifi87 pushed a commit to nahidnazifi87/mmsegmentation_playground that referenced this pull request Apr 5, 2024
@MengzhangLI
[Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x (open-mmlab#2215
f9e398b 
)

* [Feature] Support MaskFormer(NeurIPS'2021) in MMSeg 1.x

* add mmdet try except logic

* refactor config files

* add readme

* fix config

* update models & logs

* add MMDET installation and fix info

* fix comments

* fix

* fix config norm optimizer setting

* update models & logs & unittest

* add docstring of MaskFormerHead

* wait for mmdet 3.0.0rc4

* replace seg_mask with seg_logits & add docstring for batch_input_shape

* use mmdet3.0.0rc4

* fix readme and modify config comments

* add mmdet installation in pr_stage_test.yml

* update mmcv version in pr_stage_test.yml

* add mmdet in build_cpu of pr_stage_test.yml

* modify mmdet& mmcv installation in merge_stage_test.yml

* fix typo

* update test.yml

* update test.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Li-Qingyun Li-Qingyun Li-Qingyun left review comments

@MeowZheng MeowZheng MeowZheng approved these changes

Assignees

@MengzhangLI MengzhangLI

Labels
1.x Related issue of 1.x version
Projects
None yet
Milestone
1.0.0rc2
Development

Successfully merging this pull request may close these issues.
3 participants
@MengzhangLI @MeowZheng @Li-Qingyun