Skip to content

AWQModifier fast resolve mappings, better logging, MoE support #1444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jun 5, 2025

Conversation

brian-dellabetta
Copy link
Collaborator

@brian-dellabetta brian-dellabetta commented May 16, 2025

SUMMARY:
In AWQ, resolving mappings can take a while because it is traversing the entire model tree, rather than just the parent, to find the balance layers. This scopes the search to just the parent module. For MoE models, the previous implementation only found a single layer for each regex string provided in mappings. This updates that to find as many as it can, which is necessary for mappings like

AWQMapping(
    "re:.*post_attention_layernorm$",
    ["re:.*mlp.experts.*.gate_proj$", "re:.*mlp.experts.*.up_proj$"],
)

which have multiple gate_proj and up_proj layers, one for each expert.

gsm8k results with Qwen/Qwen3-30B-A3B MoE model after AWQ W4A16 Symmetric:

Tasks Version Filter n-shot Metric Value Stderr
gsm8k 3 flexible-extract 5 exact_match 0.3813 ± 0.0134
strict-match 5 exact_match 0.8810 ± 0.0089

TEST PLAN:

  • working with Qwen/Qwen3-30B-A3B with same set of mappings used in AutoAWQ. Example included in this PR in examples/awq/qwen3_moe_example.py. Ran successfully in ~2 hours on a single H100 with ~70GB of 80GB used (additional memory needed during saving)
  • Same wikitext PPL (14.0814) as on main for meta-llama/Llama-3.2-3B-Instruct

Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-fast-resolve-mappings branch 2 times, most recently from 83333a6 to 63012b4 Compare May 21, 2025 20:27
This was referenced May 22, 2025
@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-fast-resolve-mappings branch from 6749f77 to 661454f Compare May 29, 2025 16:19
@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-fast-resolve-mappings branch from ba26683 to 0ec8e0e Compare June 2, 2025 15:25
@brian-dellabetta brian-dellabetta marked this pull request as ready for review June 3, 2025 16:33
@brian-dellabetta brian-dellabetta changed the title AWQModifier fast resolve mappings, better logging AWQModifier fast resolve mappings, better logging, MoE support Jun 3, 2025
@brian-dellabetta brian-dellabetta added the ready When a PR is ready for review label Jun 3, 2025
@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-fast-resolve-mappings branch from f3d9f10 to ac26dbf Compare June 3, 2025 17:24
@brian-dellabetta brian-dellabetta requested review from dsikka, kylesayrs, rahul-tuli and shanjiaz and removed request for dsikka and kylesayrs June 3, 2025 17:30
rahul-tuli
rahul-tuli previously approved these changes Jun 3, 2025
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
kylesayrs
kylesayrs previously approved these changes Jun 5, 2025
Copy link
Collaborator

@kylesayrs kylesayrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many nits, otherwise looks good

Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
@brian-dellabetta brian-dellabetta dismissed stale reviews from kylesayrs and rahul-tuli via 96d5f59 June 5, 2025 16:52
@brian-dellabetta brian-dellabetta merged commit ceffa64 into main Jun 5, 2025
11 checks passed
@brian-dellabetta brian-dellabetta deleted the bdellabe/awq-fast-resolve-mappings branch June 5, 2025 18:00
@Chao-Xue
Copy link

https://github.com/vllm-project/llm-compressor/blob/ceffa644072b1d440df3d99b0f98f6416a05bf2f/examples/awq/qwen3_moe_example.py#L55~L56
It looks like re:.*mlp.shared_expert_gate$ is included in the ignore list, but Qwen3-MoE doesn’t seem to have a shared_expert_gate module. Probably not an issue tho

@brian-dellabetta
Copy link
Collaborator Author

@Chao-Xue , yes that is for other qwen MoE architectures where there is a shared expert that we ignore. not all qwen MoE models have that though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready When a PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants