[UX] sampling with vllm

[sindhuvahinis]

Description

This PR

• Makes greedy the default for vllm, [before this PR random sampling was the default for vllm]
• Introduces num_beams should enable beam search. [Even for the beam search, the generated sequence will be 1. It will be greater than 1 only if n > 1. n in equivalent to num_return_sequences in huggingface]
• Makes default value of max_new_tokens to be 30. [Before this PR, it was 16]

[siddvenk]

why 30? is this the same default for other backends as well?

[sindhuvahinis]

Yes, this is the same default for other backends in our handlers.

[siddvenk]

is do_sample also an alias for sampling with some default temperature in other backends? if so, are the default values for temperature when do_sample is set in parity across backends?

[lanking520]

+1, we probably want to define the do sample behavior

[sindhuvahinis]

Yes, do_sample means sampling here. But the default for temperature for greedy in other backends is 1.0.

And vllm seems to do sampling only when temperature is less than 1.0. I guess, they go for the logic that settign temp=0, means disabling the randomness, so they choose greedy when temp is set to 0.

vLLM chooses Sampling as their default method for choosing next token. But all other frameworks has Greedy as their default. So to unify our handlers behavior, here we are trying to make the sampling method in parity. But yes, default temp value would not be the same as other backends.

[siddvenk]

I see. It can be in another PR, or it might not be feasible at all, but if the default sampling behavior is different across backends when do_sample=True, that could cause confusion when users switch between backends. I wonder if it's possible (or woth it) to try to unify the behavior when do_sample=True across all backends (meaning same temperature/other params set to same value)

[sindhuvahinis]

Agreed, Will write an internal quip doc for it.

[davidthomas426]

vllm implements greedy sampling differently internally, but it just uses temperature of 0 as the API instead of adding a separate parameter for it.

• https://github.com/vllm-project/vllm/blob/fb96c1e98c05ffa35dd48416f68e88edb2f9eb34/vllm/sampling_params.py#L235-L236
• https://github.com/vllm-project/vllm/blob/fb96c1e98c05ffa35dd48416f68e88edb2f9eb34/vllm/model_executor/layers/sampler.py#L396-L448