[Misc] Add penalties sampling parameters to serve tool #25974
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for frequency_penalty, presence_penalty, and repetition_penalty sampling parameters to the vllm bench serve tool. The implementation is correct in passing these parameters to the backend. However, there is a lack of client-side validation for the values of these new parameters. I've added a comment to suggest adding validation to improve user experience and prevent benchmark failures due to invalid inputs.
| sampling_group.add_argument( | ||
| "--frequency-penalty", | ||
| type=float, | ||
| default=None, | ||
| help="Frequency penalty sampling parameter. Only has effect on " | ||
| "openai-compatible backends.", | ||
| ) | ||
| sampling_group.add_argument( | ||
| "--presence-penalty", | ||
| type=float, | ||
| default=None, | ||
| help="Presence penalty sampling parameter. Only has effect on " | ||
| "openai-compatible backends.", | ||
| ) | ||
| sampling_group.add_argument( | ||
| "--repetition-penalty", | ||
| type=float, | ||
| default=None, | ||
| help="Repetition penalty sampling parameter. Only has effect on " | ||
| "openai-compatible backends.", | ||
| ) |
There was a problem hiding this comment.
The newly added penalty parameters (frequency_penalty, presence_penalty, repetition_penalty) are parsed as floats without any validation. The OpenAI API, and vLLM's implementation of it, has specific valid ranges for these parameters:
frequency_penalty: between -2.0 and 2.0.presence_penalty: between -2.0 and 2.0.repetition_penalty: must be a positive float.
Passing values outside these ranges will cause requests to fail at the server level, which could be confusing for users running benchmarks. It would be better to add client-side validation for these parameters to provide immediate and clear feedback on invalid inputs. This could be done using a custom type function with argparse.
github-actions
bot
added
the
ready
Head branch was pushed to by a user without write access
southfreebird
force-pushed
the
feature/add-frequency-penalties-to-serve
branch
2 times, most recently
from
a9082b2 to
fe78868
Compare
Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com>
southfreebird
force-pushed
the
feature/add-frequency-penalties-to-serve
branch
from
fe78868 to
7199ec1
Compare
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com> Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com> Signed-off-by: Karan Goel <3261985+karan@users.noreply.github.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com>
…25974) Signed-off-by: Sergei Skvortsov <sergeyskv@nebius.com> Co-authored-by: Sergei Skvortsov <sergeyskv@nebius.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
2 participants
Purpose
Adding the frequency_penalty, presence_penalty, and repetition_penalty sampling parameters to the serve tool. It allows enabling them for performance measurement.
Test Plan
Example for
frequency_penalty:Test Result