[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

unaidedelf8777 · 2025-04-03T00:44:53Z

Adds outlines as a guided decoding backend for V1, and updates the integration for V0.

The aim of this is three fold:

Remove the dependency on outlines, and only use outlines_core
performance gains for V0 using the write_mask_into method on Guide to write a bitmask in-place for use in logits masking.
outlines backend for V1

Because the dependency on outlines will be removed, support for grammar based decoding with the outlines backend will also be removed (CFG classes reside in the outlines package)

cc @aarnphm

github-actions · 2025-04-03T00:45:05Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

unaidedelf8777 · 2025-04-03T00:49:54Z

NOTE: Can't be merged until next version of outlines_core is released.

russellb · 2025-04-07T20:04:36Z

Thank you for the PR! I will review it this week.

aarnphm

I reviewed the v0 code path. One ask is to add tests for this for disabling cache path.

And we should update the requirements/common.txt to the lowest version of outlines-core supported.

vllm/model_executor/guided_decoding/outlines_logits_processors.py

mergify · 2025-04-11T01:24:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @unaidedelf8777.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

unaidedelf8777 · 2025-04-13T17:54:51Z

@russellb @aarnphm All code is done. If you guys approve of it I’ll go ahead and clean up all the linter complaints, and then it should be ready.

Also outlines-core update has been pushed to pypi and pinned here (v0.2.9)

aarnphm

First round of review. A few things needs to be addressed here. but great progress so far.

vllm/model_executor/guided_decoding/outlines_decoding.py

vllm/model_executor/guided_decoding/outlines_logits_processors.py

vllm/v1/structured_output/backend_outlines.py

mergify · 2025-04-18T23:44:25Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @unaidedelf8777.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

vllm/envs.py

tests/entrypoints/llm/test_guided_generate.py

russellb · 2025-06-02T14:29:48Z

tests/model_executor/test_guided_processors.py

+                                    reasoner=None,
+                                    vocab_size=32000)


Can you explain why all of the changes were needed to this file? I don't think it has changed a while, so I'm surprised to see changes needed for the V0 code we're trying to change as little as possible.

The old integration would silent fail in this test when an arbitrary sequence of token ids was passed in since it would just default to the initial state of the DFA when the sequence's prior state could not be found in the self._fsm_state defaultdict. In contrast the new version which uses the Guide object will error in this case as opposed to defaulting, so we need to make sure we are passing in sequences which have their sequential dependency (Guide has seen all tokens in the sequence prior to the most recent token) satisfied.

russellb · 2025-06-02T14:38:44Z

CI also needs to pass on here. If you think any of the failures aren't the fault of your PR, I have some notes here on what to do next: #18782

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 · 2025-06-03T19:15:59Z

@russellb the two failing tests seem to be un-related. Both concern parts of the codebase which have not been modified, and only started failing after I updated the base branch of the PR, leading me to believe it is an upstream issue. Further, on this CI run from last night roughly when I updated the base branch the same tests are failing.

russellb

Thanks for the updates. Can you add outlines testing to the V1 tests here?

https://github.com/vllm-project/vllm/blob/main/tests/v1/entrypoints/llm/test_struct_output_generate.py

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

unaidedelf8777 · 2025-06-22T15:04:15Z

@russellb sorry about the hold up, I had to go to Mexico for a family situation last week. V1 tests are added and passing. The failing speculative decoding and quantization tests are currently failing on main according to this ci run. The LoRA test which fails seems to be unrelated to anything modified here.

aarnphm

Thanks for all the hard work and perseverance! The CI failure is not relevant and will be fixed separately.

but still will wait for @russellb to take a look once he's back

unaidedelf8777 requested review from mgoin, russellb, WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 3, 2025 00:44

unaidedelf8777 changed the title ~~[V0][V1][Core] Add outlines integration for V1, and update V0 integration.~~ [V0][V1][Core] Add outlines integration for V1, and update V0 integration. [DO NOT MERGE] Apr 3, 2025

mergify bot added the v1 label Apr 3, 2025

unaidedelf8777 marked this pull request as draft April 3, 2025 00:46

unaidedelf8777 changed the title ~~[V0][V1][Core] Add outlines integration for V1, and update V0 integration. [DO NOT MERGE]~~ [V0][V1][Core] Add outlines integration for V1, and update V0 integration. Apr 3, 2025

hmellor mentioned this pull request Apr 3, 2025

Removes duplicate outlines processors #6900

Closed

mergify bot added the structured-output label Apr 4, 2025

aarnphm approved these changes Apr 7, 2025

View reviewed changes

vllm/model_executor/guided_decoding/outlines_logits_processors.py Outdated Show resolved Hide resolved

mergify bot added tpu Related to Google TPUs ci/build and removed tpu Related to Google TPUs labels Apr 9, 2025

mergify bot added needs-rebase and removed needs-rebase labels Apr 11, 2025

unaidedelf8777 marked this pull request as ready for review April 13, 2025 17:55

aarnphm requested changes Apr 14, 2025

View reviewed changes

mergify bot added needs-rebase and removed needs-rebase labels Apr 18, 2025

russellb requested changes Jun 2, 2025

View reviewed changes

unaidedelf8777 added 6 commits June 2, 2025 17:42

fix

d4063e4

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix failing entrypoint test

186eede

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix pre-commit

cb6e9f0

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

c8548c4

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

Merge branch 'main' into update-outlines-integration

731cf7b

re-run ci

e0b1eea

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

russellb requested changes Jun 5, 2025

View reviewed changes

unaidedelf8777 added 17 commits June 7, 2025 17:37

update tests

1587dd8

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

75c7a46

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix failing test

b07d315

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit fix

8ad4aa3

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix

8dc48d5

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix

981b322

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix

d4ab074

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix

b9861c3

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

79bb600

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

96b649b

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

disable grammar and structured_tag tests for outlines

0254844

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix final test

f839448

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

pre-commit

f431efe

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix re_replacement_seq regex for mistral

96d2489

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

regex

c0be222

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

make replacement_seq regex catch larger.

54c6075

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

fix mistral handling

b363dda

Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>

aarnphm requested a review from russellb June 23, 2025 19:54

aarnphm approved these changes Jun 23, 2025

View reviewed changes

Uh oh!

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

Are you sure you want to change the base?

[V0][V1][Core] Add outlines integration for V1, and update V0 integration. #15975

Uh oh!

Conversation

unaidedelf8777 commented Apr 3, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 3, 2025

Uh oh!

unaidedelf8777 commented Apr 3, 2025

Uh oh!

russellb commented Apr 7, 2025

Uh oh!

aarnphm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Apr 11, 2025

Uh oh!

unaidedelf8777 commented Apr 13, 2025

Uh oh!

aarnphm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Apr 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

russellb Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

unaidedelf8777 Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

russellb commented Jun 2, 2025

Uh oh!

unaidedelf8777 commented Jun 3, 2025

Uh oh!

russellb left a comment

Choose a reason for hiding this comment

Uh oh!

unaidedelf8777 commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aarnphm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

unaidedelf8777 commented Apr 3, 2025 •

edited by github-actions bot

Loading

unaidedelf8777 commented Jun 22, 2025 •

edited

Loading

aarnphm left a comment •

edited

Loading