[model] make llama4 compatible with pure dense layers #17315
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
Signed-off-by: Lucia Fang <fanglu@fb.com>
luccafong
force-pushed
the
llama-dense-layers
branch
from
ea54eaf to
ae67aa8
Compare
…7315) Signed-off-by: Lucia Fang <fanglu@fb.com>
|
Looks like this code fix has not been pushed into the docker hub yet. I’m using vLLM docker image v0.8.5.post1 and still getting the same divide by zero error while deploying Llama-Guard-4-12B model. Any tentative date when this fix will be available on docker hub ? |
|
You can use commit-specific images: https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-images |
…7315) Signed-off-by: Lucia Fang <fanglu@fb.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
…7315) Signed-off-by: Lucia Fang <fanglu@fb.com>
Syncing midstream NM fork to Upstream tag of [v0.8.5.post1](https://github.com/vllm-project/vllm/tree/v0.8.5.post1) + cherry pick of vllm-project@be633fb needed for benchmarks + [CP](neuralmagic/nm-vllm-ent@1fe447d) for compressed tensor bump + [CP](vllm-project#17677) for lora on AMD + [CP](vllm-project#17315) for llama4 w/ pure dense layers ``` commit 31c73ba (HEAD -> upstream-v0.8.5, nm-fork/upstream-v0.8.5) Author: Chauncey <chaunceyjiang@gmail.com> Date: Wed Apr 30 15:11:04 2025 +0800 [Bugfix] Fix AttributeError: 'State' object has no attribute 'engine_client' (vllm-project#17434) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> commit f8db0bd Author: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Date: Fri May 2 14:01:38 2025 -0400 [BugFix][Attention] Fix sliding window attention in V1 giving incorrect results (vllm-project#17574) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> commit e335c34 Author: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Date: Fri May 2 04:07:03 2025 -0400 [BugFix] Fix Memory Leak (vllm-project#17567) Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> commit cc463fe Merge: 1e358ff ba41cc9 Author: Selbi Nuryyeva <selbi@redhat.com> Date: Tue Apr 29 12:34:57 2025 -0400 Merge branch 'tag-upstream-v0.8.5' into upstream-v0.8.5 commit ba41cc9 (tag: v0.8.5, tag-upstream-v0.8.5) Author: Michael Goin <mgoin64@gmail.com> Date: Mon Apr 28 16:20:24 2025 -0600 [Model] Add tuned triton fused_moe configs for Qwen3Moe (vllm-project#17328) Signed-off-by: mgoin <mgoin64@gmail.com> commit dcbac4c Author: Simon Mo <simon.mo@hey.com> Date: Mon Apr 28 14:12:01 2025 -0700 [Model] Qwen3 Dense FP8 Compat Fixes (vllm-project#17318) Signed-off-by: simon-mo <xmo@berkeley.edu> [...] ``` Commands ``` git fetch upstream git checkout -b upstream-v0.8.5 git merge upstream/v0.8.5 git cherry-pick be633fb ``` TEST PLAN accept sync: https://github.com/neuralmagic/nm-cicd/actions/runs/14841223552 related PR in cicd: neuralmagic/nm-cicd#99 release workflow: https://github.com/neuralmagic/nm-cicd/actions/runs/14845693864
…7315) Signed-off-by: Lucia Fang <fanglu@fb.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
…7315) Signed-off-by: Lucia Fang <fanglu@fb.com> Signed-off-by: minpeter <kali2005611@gmail.com>
make llama4 compatible with pure dense layers when interleave_moe_layer_step == 0 (no moe layers)