[Bugfix] Ensure special tokens are properly filtered out for guided structured output with MistralTokenizer

[gcalmettes]

This PR improves the support of the vllm MistralTokenizer with the guided structured output functionalities.

In particular:

• the version of lm-format-enforcer is bumped to the latest version, so it includes the support of the vllm MistralTokenizer (feat: add support for vLLM Mistral Tokenizer noamgat/lm-format-enforcer#142)
• the vllm MistralTokenizer methods leveraged by the structured output libraries are now properly populated so that the list of possible tokens generated based on the chosen grammar/constraints (e.g.: json output) correctly filters out the special tokens. Note that this problem was present before, but masked by the fact that the special tokens were automatically filtered out of the conversion from ids to string: the recent vllm support for automatic mistral tool parsing allowed to reveal the bug (also see this comment #10333). As a benefit, this should also make the structured output generation by Mistral models more reliable.

cc: @patrickvonplaten

Example: code that would break without this PR but passes with it

"""
vllm openai server started with the following arguments:
    --model=mistralai/Pixtral-12B-2409
    --guided-decoding-backend=lm-format-enforcer 
    --enable-auto-tool-choice 
    --tool-call-parser=mistral 
    --tokenizer-mode=mistral
"""

from openai import OpenAI
import json
from pydantic import BaseModel

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none",
)

######################################
# Example 1: using `client.beta.chat.completions.parse`
######################################

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

completion = client.beta.chat.completions.parse(
    model="mistralai/Pixtral-12B-2409",
    messages=[
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
    ],
    response_format=CalendarEvent,
    max_completion_tokens=10000,
)

event = completion.choices[0].message.parsed
print(event.__dict__)

######################################
# Example 2: using `client.chat.completions.create`
######################################

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    steps: list[Step]
    final_answer: str

completion = client.chat.completions.create(
    model="mistralai/Pixtral-12B-2409",
    messages=[
        {"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step."},
        {"role": "user", "content": "how can I solve 8x + 7 = -23"}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "toto",
            "schema": MathReasoning.model_json_schema()
        }
    },
    max_completion_tokens=10000,
)

math_reasoning = completion.choices[0].message.content
print(json.loads(json.dumps(math_reasoning, indent=2)))

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[gcalmettes]

@DarkLight1337 it's a follow up of #10333

[DarkLight1337]

cc @simon-mo

[gcalmettes]

@DarkLight1337 unfortunately, we did not get lucky and got timed-out on the Async Engine, Inputs, Utils, Worker Test test. Any chance you restart this one ? 🙏