Skip to content

Commit

Permalink
[Bugfix] Better error message for MLPSpeculator when `num_speculative…
Browse files Browse the repository at this point in the history
…_tokens` is set too high (vllm-project#5894)

Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
  • Loading branch information
tdoublep authored and prashantgupta24 committed Jul 1, 2024
1 parent af2e655 commit a2327fa
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions vllm/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -956,9 +956,9 @@ def maybe_create_spec_config(
# Verify provided value doesn't exceed the maximum
# supported by the draft model.
raise ValueError(
"Expected both speculative_model and "
"num_speculative_tokens to be provided, but found "
f"{speculative_model=} and {num_speculative_tokens=}.")
"This speculative model supports a maximum of "
f"num_speculative_tokens={n_predict}, but "
f"{num_speculative_tokens=} was provided.")

draft_model_config.max_model_len = (
SpeculativeConfig._maybe_override_draft_max_model_len(
Expand Down

0 comments on commit a2327fa

Please sign in to comment.