[Model] PP support for Mamba-like models #10992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

tlrmchlsmth merged 31 commits into vllm-project:main from mzusman:mamba_jamba_pp

Dec 11, 2024

Contributor

mzusman commented Dec 8, 2024 •

edited by github-actions bot

Loading

Adds PP support for Mamba-like models (Jamba, Mamba, MambaFalcon, etc..)

CC @tlrmchlsmth

mzusman requested review from WoosukKwon, alexm-redhat, comaniac, njhill, robertgshaw2-redhat, youkaichao and ywang96 as code owners

December 8, 2024 15:52

github-actions bot commented Dec 8, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

mergify bot commented Dec 8, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added needs-rebase and removed needs-rebase labels

mzusman added 10 commits

December 8, 2024 17:58


          Add PP support to Jamba

90d0ff3

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          GPU Runner adaptations

d30f741

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Add tests and adaptations

47e961b

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Add PP to mamba

e7e3ea4

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Introduce LayerBlockType

ebbf7e3

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Fix jamba

5514aa6

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Fix MS for Mamba

f9fcff4

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Format

2b5b525

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Format

01c7d93

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Format

50f2a8c

Signed-off-by: mzusman <mor.zusmann@gmail.com>

mzusman force-pushed the mamba_jamba_pp branch from e9f48d0 to 50f2a8c Compare

December 8, 2024 16:03

mzusman added 3 commits

December 8, 2024 18:06


          Remove comment

314c9af

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Add is_pp_missing_parameter to mamba weight loading

3347f3f

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Revert Mamba MS fix, will fix it in future PR

f21ff9a

Signed-off-by: mzusman <mor.zusmann@gmail.com>

DarkLight1337 removed request for WoosukKwon, alexm-redhat, comaniac, njhill and youkaichao

December 8, 2024 22:32

mzusman added 2 commits

December 9, 2024 18:00


          Fix get_num_layers_by_block_type to always return num of layers

f354468

regardless of given block_type

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Format

529ade6

Signed-off-by: mzusman <mor.zusmann@gmail.com>

tlrmchlsmth reviewed

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved

mzusman added 4 commits

December 9, 2024 18:22


          Add comment for workaround

e7f6338

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Fix comment

6d863ae

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Add IsHybrid interface to return 0 layers on non-hybrid models asking

d9ef20d

about non-attention layers

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Typo

029d710

Signed-off-by: mzusman <mor.zusmann@gmail.com>

mergify bot commented Dec 9, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge remote-tracking branch 'github/main' into mamba_jamba_pp

fbf4ccd

Signed-off-by: mzusman <mor.zusmann@gmail.com>

mergify bot removed the needs-rebase label

mzusman added 2 commits

December 9, 2024 19:37


          Format

3bc3823

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Fix mamba logic

0f70131

Signed-off-by: mzusman <mor.zusmann@gmail.com>

mergify bot commented Dec 10, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge remote-tracking branch 'github/main' into mamba_jamba_pp

704f29a

Signed-off-by: mzusman <mor.zusmann@gmail.com>

mergify bot removed the needs-rebase label


          Small fix to logic

fcce742

Signed-off-by: mzusman <mor.zusmann@gmail.com>

tlrmchlsmth reviewed

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved


          FIx logic

e3d8343

Signed-off-by: mzusman <mor.zusmann@gmail.com>

tlrmchlsmth reviewed

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved

mzusman added 2 commits

December 10, 2024 19:47


          Fixing according to comment

6d6b3d4

Signed-off-by: mzusman <mor.zusmann@gmail.com>


          Format

c034ffe

Signed-off-by: mzusman <mor.zusmann@gmail.com>

tlrmchlsmth approved these changes

View reviewed changes

Member

tlrmchlsmth left a comment

Thank you!


          Fix comment

75ff1ef

Signed-off-by: mzusman <mor.zusmann@gmail.com>

tlrmchlsmth merged commit ffa48c9 into vllm-project:main

55 checks passed

fabianlim mentioned this pull request

Add Bamba Model #10909

Merged

6 tasks

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request


          [Model] PP support for Mamba-like models (vllm-project#10992)

04ea41e

Signed-off-by: mzusman <mor.zusmann@gmail.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation ready