OpenAI server with min_p

[generalsvr]

How to set properly min_p sampling when using vllm.entrypoints.openai.api_server ? Is it supported?

[esmeetu]

Hi, @generalsvr. vLLM currently supports min_p sampling. You can request api server with min_p parameter. For my testing, 0.05 is a proper value, but it depends on your model.

[generalsvr]

Ok I changed openai python client lib to support min_p arg. Works better. But when I try to set stream=True I get this error:

  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/routing.py", line 69, in app
    await response(scope, receive, send)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
    async with anyio.create_task_group() as task_group:
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/responses.py", line 273, in wrap
    await func()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/starlette/responses.py", line 262, in stream_response
    async for chunk in self.body_iterator:
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 599, in completion_stream_generator
    response_json = create_stream_response_json(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 548, in create_stream_response_json
    response = CompletionStreamResponse(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/pydantic/main.py", line 164, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 1 validation error for CompletionStreamResponse
usage
  Field required [type=missing, input_value={'id': 'cmpl-0b8b9c924028...e, finish_reason=None)]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
INFO 12-27 14:22:13 async_llm_engine.py:111] Finished request cmpl-0b8b9c92402848109fb4983f8758b9b5.

[esmeetu]

@generalsvr It seems you installed a higher version of pydantic. Please be sure to use version 1.10.x.