[Model] Add PLaMo2 #14323

Alnusjaponica · 2025-03-06T03:10:33Z

This PR adds support for PLaMo2, specifically PLaMo2 1B and PLaMo2 8B. PLaMo2 is a hybrid mamba2 architecture featuring sliding window attention.

In this PR, we have created the inference architecture for the PLaMo2 model.

Please note that some functionalities such as PP/TP, quantization, and LoRA are not yet supported in this PR. We plan to address these features in subsequent follow-up PRs.

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Co-Authored-By: Kento Nozawa <nzw0301@preferred.jp> Co-Authored-By: Hiroaki Mikami <mhiroaki@preferred.jp>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Co-authored-by: Calvin Metzger <metzger@preferred.jp>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

github-actions · 2025-03-06T03:10:44Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica · 2025-03-06T03:25:44Z

This is my first time submitting a PR to this repository, and I just joined the Slack. I believe I do not have permission to unblock additional CIs, so I would appreciate it if you could add me to the Buildkite org. If I have missed anything or if there are areas for improvement, please do let me know.

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Signed-off-by: shemmi <shemmi@preferred.jp>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

tests/models/decoder_only/language/test_hybrid.py

tests/models/registry.py

vllm/model_executor/models/registry.py

DarkLight1337 · 2025-03-19T15:12:24Z

@tlrmchlsmth can you do a quick pass to check that the implementation of this model fits our architecture? No need to test for correctness since they are the model vendor

tests/models/decoder_only/language/test_hybrid.py

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica · 2025-04-07T11:21:24Z

@DarkLight1337 Thanks for your review. I've updated the tests.
@tlrmchlsmth Now that the public model is updated and I’ve also updated this PR according to the new changes, could you take another look?

DarkLight1337

Sorry for forgetting about this, should be good to go now!

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica · 2025-04-15T05:31:10Z

Thanks for your approval!
I found some tests fails after I merged the main branch, so I am trying to fix them. I would appreciated it if you could take another look once the test passes.

Alnusjaponica · 2025-04-15T07:57:54Z

It seems unrelated part of the tests are still failing. I'll wait main branch to be fixed.

DarkLight1337 · 2025-04-15T08:56:15Z

Can you update models tests in

vllm/.buildkite/test-pipeline.yaml

Line 396 in 70e7ed8

- VLLM_USE_V1=0 pytest -v -s models/test_initialization.py -k 'not llama4'

to run your model separate from the rest of the models? There is a known memory leak issue for Llama 4 in those tests so perhaps that's also the case for your model.

Alnusjaponica · 2025-04-15T12:57:23Z

Sure, it passed in my environment, so I thought it was not relevant either. Let me see if other tests pass with the latest main branch at first.

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica · 2025-04-16T02:02:37Z

@DarkLight1337
Separating plamo2 passes the Basic Model Test, so might be the case for our models. Is there anything I can do to fix this?

DarkLight1337 · 2025-04-16T02:31:16Z

The problem doesn't appear in V1 so let's just merge this.

Alnusjaponica · 2025-04-16T04:17:39Z

Thanks for the approval! Next, I'll I will submit a follow-up PR to provide support for TP/PP, etc.

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Signed-off-by: shemmi <shemmi@preferred.jp> Co-authored-by: Kento Nozawa <nzw0301@preferred.jp> Co-authored-by: Hiroaki Mikami <mhiroaki@preferred.jp> Co-authored-by: Calvin Metzger <metzger@preferred.jp>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Signed-off-by: shemmi <shemmi@preferred.jp> Co-authored-by: Kento Nozawa <nzw0301@preferred.jp> Co-authored-by: Hiroaki Mikami <mhiroaki@preferred.jp> Co-authored-by: Calvin Metzger <metzger@preferred.jp> Signed-off-by: Yang Wang <elainewy@meta.com>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Signed-off-by: shemmi <shemmi@preferred.jp> Co-authored-by: Kento Nozawa <nzw0301@preferred.jp> Co-authored-by: Hiroaki Mikami <mhiroaki@preferred.jp> Co-authored-by: Calvin Metzger <metzger@preferred.jp>

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Signed-off-by: shemmi <shemmi@preferred.jp> Co-authored-by: Kento Nozawa <nzw0301@preferred.jp> Co-authored-by: Hiroaki Mikami <mhiroaki@preferred.jp> Co-authored-by: Calvin Metzger <metzger@preferred.jp> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

Alnusjaponica and others added 5 commits March 4, 2025 22:09

Add PLaMo2 model at v0.6.3.post1

b7fb14d

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Co-Authored-By: Kento Nozawa <nzw0301@preferred.jp> Co-Authored-By: Hiroaki Mikami <mhiroaki@preferred.jp>

Follow-up to the latest based on Jamba implementaion

e58e384

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com> Co-authored-by: Calvin Metzger <metzger@preferred.jp>

Modify interfaces

b0f101e

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Add workaround to use IsHybrid interface

a783e31

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Update dependencies for test

5ffec2c

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

mergify bot added the ci/build label Mar 6, 2025

Alnusjaponica added 2 commits March 6, 2025 12:12

Add test for plamo2 model

49dd3b0

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Modify code comment

68d3bed

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica force-pushed the add-plamo2 branch from 514cff5 to 68d3bed Compare March 6, 2025 03:13

Resolve mypy error

bee8035

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Alnusjaponica marked this pull request as ready for review March 6, 2025 03:28

Alnusjaponica requested review from DarkLight1337 and ywang96 as code owners March 6, 2025 03:28

Add plamo to test_registry

1058777

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

DarkLight1337 requested a review from tlrmchlsmth March 6, 2025 04:23

Alnusjaponica and others added 5 commits March 10, 2025 12:23

Merge branch 'main' into add-plamo2

80a9abb

pip-compile

e86e46f

Signed-off-by: shemmi <shemmi@preferred.jp>

pip-compile

7659755

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Add workarounds to hundle the difference in config assumptions

e394371

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Make workaround simple

9d7efcc

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

nzw0301 mentioned this pull request Mar 12, 2025

[New Model]: pfnet/plamo-2-8b #14214

Open

1 task

Alnusjaponica added 3 commits March 17, 2025 15:57

Merge branch 'main' into add-plamo2

b0b222e

Merge branch 'main' into add-plamo2

121ab1d

yapf

f4a6ac1

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

DarkLight1337 reviewed Mar 19, 2025

View reviewed changes

tests/models/decoder_only/language/test_hybrid.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Mar 19, 2025

View reviewed changes

tests/models/registry.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Mar 19, 2025

View reviewed changes

vllm/model_executor/models/registry.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Apr 7, 2025

View reviewed changes

tests/models/decoder_only/language/test_hybrid.py Outdated Show resolved Hide resolved

Update object names to plamo2-prefixed in the tests

f43d02a

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

Merge branch 'main' into add-plamo2

2f3bed1

DarkLight1337 approved these changes Apr 15, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) April 15, 2025 04:12

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 15, 2025

Fix Plamo2ForCausalLM class name

7b41a18

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

auto-merge was automatically disabled April 15, 2025 05:26
Head branch was pushed to by a user without write access

Merge branch 'main' into add-plamo2

f5bf80a

Alnusjaponica added 2 commits April 15, 2025 21:58

Merge branch 'main' into add-plamo2

313e050

Split plamo2 initialization test for debugging purpose

0c8fb36

Signed-off-by: Shinichi Hemmi <50256998+Alnusjaponica@users.noreply.github.com>

vllm-bot merged commit 3badb02 into vllm-project:main Apr 16, 2025
63 of 69 checks passed

tlrmchlsmth mentioned this pull request Apr 24, 2025

[RFC]: Native support for Mamba, SSM, and hybrid transformer models in vLLM V1 #17140

Open

9 tasks

ckhordiasma mentioned this pull request May 14, 2025

nm vllm ent 0.8.5 sync red-hat-data-services/vllm#139

Merged

Alnusjaponica mentioned this pull request Jun 16, 2025

[Model] Support TP/PP/mamba2 kernel for PLaMo2 #19674

Open

4 tasks

seemethere mentioned this pull request Jun 25, 2025

requirements: Remove mamba_ssm from test.txt #20047

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Model] Add PLaMo2 #14323

[Model] Add PLaMo2 #14323

Uh oh!

Alnusjaponica commented Mar 6, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

Alnusjaponica commented Mar 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Mar 19, 2025

Uh oh!

Uh oh!

Alnusjaponica commented Apr 7, 2025 •

edited

Loading

Uh oh!

DarkLight1337 left a comment

Uh oh!

Alnusjaponica commented Apr 15, 2025

Uh oh!

Alnusjaponica commented Apr 15, 2025

Uh oh!

DarkLight1337 commented Apr 15, 2025

Uh oh!

Alnusjaponica commented Apr 15, 2025 •

edited

Loading

Uh oh!

Alnusjaponica commented Apr 16, 2025

Uh oh!

DarkLight1337 commented Apr 16, 2025

Uh oh!

Uh oh!

Alnusjaponica commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

[Model] Add PLaMo2 #14323

[Model] Add PLaMo2 #14323

Uh oh!

Conversation

Alnusjaponica commented Mar 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2025

Uh oh!

Alnusjaponica commented Mar 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Mar 19, 2025

Uh oh!

Uh oh!

Alnusjaponica commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Alnusjaponica commented Apr 15, 2025

Uh oh!

Alnusjaponica commented Apr 15, 2025

Uh oh!

DarkLight1337 commented Apr 15, 2025

Uh oh!

Alnusjaponica commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alnusjaponica commented Apr 16, 2025

Uh oh!

DarkLight1337 commented Apr 16, 2025

Uh oh!

Uh oh!

Alnusjaponica commented Apr 16, 2025

Uh oh!

Uh oh!

Alnusjaponica commented Mar 6, 2025 •

edited by github-actions bot

Loading

Alnusjaponica commented Apr 7, 2025 •

edited

Loading

Alnusjaponica commented Apr 15, 2025 •

edited

Loading