[Bug]: Issues with max_model_len and config_format mistral

[tjohnson31415]

Your current environment

The output of python collect_env.py

🐛 Describe the bug

We have observed crashes and bad results when using the mistralai/Mistral-Small-3.1-24B-Instruct-2503 model due to some bug in how --max-model-len is handled.

For this model, the params.json does not have any configuration for the max sequence length. The default value used is 128000 REF. But, the HF config in config.json has "max_position_embeddings": 131072.

So we run the OpenAI server with:

VLLM_USE_V1=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 --tokenizer-mode mistral  --config-format mistral --load-format mistral --max-num-seqs 4 --max-model-len 131072

The server then boots with the expected warning (though it is a little misleading since we aren't using config.json and 128000 is the default value not from the config):

WARNING 05-06 22:11:51 [config.py:3178] User-specified max_model_len (131072) is greater than the derived max_model_len (max_position_embeddings=128000 or model_max_length=None in model's config.json). This may lead to incorrect model outputs or CUDA errors. Make sure the value is correct and within the model context size.

If we send a request with a context length > 128000 and <131072, eg.

curl -s -X POST   -H "Content-Type: application/json"   "http://localhost:8000/v1/completions"   --data-binary @- << _EOF
  {
   "model": "mistralai/Mistral-Small-3.1-24B-Instruct-2503",
   "prompt": "$(seq -s ' ' 1 23500)",
   "max_generated_tokens": 8
  }
_EOF

then we get a crash with a bunch of CUDA errors like:

/home/vllm/.cache/vllm/torch_compile_cache/b3db66fc05/rank_0_0/inductor_cache/pi/cpiayc7qwveqsg53w53ihzygo5qggdq4bzprcwvrqyu5eljbgt6z.py:37: unknown: block: [21452,0,0], thread: [53,0,0] Assertion `index out of bounds: 0 <= tl.broadcast_to(tmp10, [XBLOCK]) < 128000` failed.
/home/vllm/.cache/vllm/torch_compile_cache/b3db66fc05/rank_0_0/inductor_cache/pi/cpiayc7qwveqsg53w53ihzygo5qggdq4bzprcwvrqyu5eljbgt6z.py:37: unknown: block: [21452,0,0], thread: [54,0,0] Assertion `index out of bounds: 0 <= tl.broadcast_to(tmp10, [XBLOCK]) < 128000` failed.
/home/vllm/.cache/vllm/torch_compile_cache/b3db66fc05/rank_0_0/inductor_cache/pi/cpiayc7qwveqsg53w53ihzygo5qggdq4bzprcwvrqyu5eljbgt6z.py:37: unknown: block: [21452,0,0], thread: [55,0,0] Assertion `index out of bounds: 0 <= tl.broadcast_to(tmp10, [XBLOCK]) < 128000` failed.
...
ERROR 05-06 22:15:06 [core.py:402]   File "/workspace/my-vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1218, in execute_model
ERROR 05-06 22:15:06 [core.py:402]     valid_sampled_token_ids = sampled_token_ids.tolist()
ERROR 05-06 22:15:06 [core.py:402]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-06 22:15:06 [core.py:402] RuntimeError: CUDA error: device-side assert triggered

(Note that similar errors were reported in #17348 but that was issue was found to be a bad GPU)

If we run the server with --enforce-eager, the out of bounds error from torch compilation is avoided and the request succeeds, but the results are gibberish, eg. "text":"2 7687 7 2139 AL27"

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[princepride]

@DarkLight1337 I've also been able to reproduce this bug. Anyone working on this? I can try if not.

[DarkLight1337]

If @tjohnson31415 isn't already, you can work on it. Thanks!

[tjohnson31415]

Thanks for jumping on this!

A couple of other points that I found from my investigation:

1. Note that with VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ... --max-model-len 131072 the scheduler is configured to allow a longer sequence length, but the model configuration will still have the lower value. I think the model config should be updated in this case, or switch users to use --hf-overrides to increase the max model length (like what is done with the rope configurations now). Note that this affects other models too.
2. _get_and_verify_max_len will choose the lowest value from the configuration REF. This causes a problem when setting both max_position_embeddings and max_seq_len in the derived HF config if their values differ, wich would happen even if params.json had one of the values. I haven't fully checked, but I think all Mistral models use a Llama based text model; if so, setting only max_position_embeddings may be better. I see that the PR makes sure to set them to the same value, which fixes the main bugginess here!

[princepride]

Hi @tjohnson31415,

Thanks a lot for the detailed feedback and further insights from your investigation! I really appreciate you digging deeper into this.

Regarding your points:

1. Scheduler vs. Model Configuration Discrepancy:
   You've raised a very valid point about the scheduler being configured for the user-specified max_model_len while the underlying model's internal configuration might not be directly updated in the same way merely by setting --max-model-len along with VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. My PR Fix: Correct max_model_len derivation from config.json for Mistral format #17777 primarily focused on ensuring that the max_model_len derived from config.json (especially when params.json is missing or incomplete for Mistral format) is consistent and correctly used, which resolves the crash and output corruption.
   Your suggestion to either:

   • More explicitly update the model's internal config based on --max-model-len, or
   • Standardize on using --hf-overrides for such modifications (akin to RoPE settings)

   is excellent for ensuring greater robustness and clarity. I believe that changes related to --hf-overrides would be more suitable for a separate feature issue and subsequent PR. This might indeed be a good direction for a follow-up improvement to ensure all parts of the system are fully aligned with the user's intent for max_model_len. I agree this could benefit other models too.

2. _get_and_verify_max_len Behavior and Mistral Configs:
   Thanks for noticing that the PR ensures the relevant length parameters (like max_position_embeddings and max_seq_len in the derived HF config) are set to the same value! That was indeed key to fixing the immediate bug.
   Your observation about Mistral models potentially relying primarily on max_position_embeddings due to their Llama-based architecture is also a very insightful point. While my PR aims for consistency by setting them equally when deriving the config, further specialization or simplification for Mistral-like models based on this could be explored in the future.

The main goal of my PR was to address the critical bug causing crashes and incorrect outputs with the specified model when max_model_len exceeded the implicitly derived length. I'm glad it seems to have tackled that "main bugginess."

These additional points are very helpful for future enhancements and ensuring the long-sequence handling is as robust and user-friendly as possible. Thanks again!