[Bugfix] make test_openai_schema.py pass

[davidxia]

by filtering out test cases to POST /tokenize endpoint that are known to fail with a reply of HTTP 501 Not Implemented.

Enable the test in CI.

FIX #18162

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[davidxia]

I couldn't find an easy way to modify the generated OpenAPI or Swagger UI to not have "type": "file". So instead I decided to skip those test cases.

[DarkLight1337]

cc @tarukumar @hmellor

[davidxia]

Is it expected that POST /v1/chat/completions times out? It times out for me locally on an A100 GPU too.

(verbose_name='POST /v1/chat/completions') SUBFAIL entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/chat/completions] - schemathesis.exceptions.CheckFailed:

1. Response timeout

The server failed to respond within the specified limit of 10000.00ms

Reproduce with:

    curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": "\u00f6\u001f\u00d2?\u00fa", "role": "tool", "tool_call_id": ""}, {"content": [{"text": "", "type": "text"}, {"text": "", "type": "text"}, {"text": "", "type": "text", "": null}], "role": "developer"}]}' --insecure http://localhost:34335/v1/chat/completions

Falsifying example: test_openapi_stateless(
    case=,
)

[hmellor]

I don't think so, was the GPU busy during those 10 seconds? And if you increase the timout will the test eventually pass?

[davidxia]

I don't think so, was the GPU busy during those 10 seconds?

I'm using a dedicated GCE VM with no other workloads on it. So I don't think so.

And if you increase the timout will the test eventually pass?

yes, see commit d84a35e

[hmellor]

I'm using a dedicated GCE VM with no other workloads on it. So I don't think so.

Sorry I meant is vLLM doing things during those 10 seconds. Sounds like it just needed more time

[davidxia]

Per-route request timeouts are documented here. Without this I get

schemathesis.exceptions.CheckFailed...The server failed to respond within the specified limit of 30000.00ms

$ pytest tests/entrypoints/openai/test_openai_schema.py -v
/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
=========================================================== test session starts ===========================================================
platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 -- /home/dxia/src/github.com/vllm-project/vllm/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/dxia/src/github.com/vllm-project/vllm/.hypothesis/examples'))
rootdir: /home/dxia/src/github.com/vllm-project/vllm
configfile: pyproject.toml
plugins: mock-3.14.0, anyio-4.6.2.post1, timeout-2.3.1, subtests-0.14.1, shard-0.1.2, buildkite-test-collector-0.1.9, asyncio-0.24.0, forked-1.6.0, hypothesis-6.131.0, schemathesis-3.39.15, rerunfailures-14.0
asyncio: mode=Mode.STRICT, default_loop_scope=None
collected 1 item                                                                                                                          
Running 1 items in this shard: tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless

tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless 
tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/chat/completions] (verbose_name='POST /v1/chat/completions') SUBFAIL [ 50%]
tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless PASSED                                                       [100%]

================================================================ FAILURES =================================================================
____________________________________ test_openapi_stateless (verbose_name='POST /v1/chat/completions') ____________________________________
  + Exception Group Traceback (most recent call last):
  |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/lazy.py", line 370, in run_subtest
  |     sub_test(**fixtures)
  |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py", line 83, in test_openapi_stateless
  |     def test_function(*args: Any, **kwargs: Any) -> Any:
  |                ^^^
  |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 1839, in wrapped_test
  |     raise the_error_hypothesis_found
  | hypothesis.errors.FlakyFailure: Hypothesis test_openapi_stateless(case=) produces unreliable results: Falsified on the first call but did not on a subsequent one (1 sub-exception)
  | Falsifying example: test_openapi_stateless(
  |     case=,
  | )
  | Failed to reproduce exception. Expected: 
  | case = Case(body={'messages': [{'content': 'K', 'name': ";\x83\U000164edC\U00049bcc,\U0010bb3eõæÃ\x9evUàk?\U000f8fd6\n'Ðß'\U0...09, 'Ó': -1.0091098906725308e+16}], '\x1e=8\U00051cc9Ý': {'e': [None], '\U0003bf24ЧÇ': []}}], '': [-1.192092896e-07]})
  | 
  |     @schema.include(method="POST", path="/v1/chat/completions").parametrize()
  |     @schema.override(headers={"Content-Type": "application/json"})
  |     @settings(deadline=30000)
  |     def test_openapi_stateless(case: schemathesis.Case):
  |         key = (
  |             case.operation.method.upper(),
  |             case.operation.path,
  |         )
  |         timeout = {
  |             ("POST", "/v1/chat/completions"): 30,
  |         }.get(key, DEFAULT_TIMEOUT_SECONDS)
  |     
  |         #No need to verify SSL certificate for localhost
  | >       case.call_and_validate(verify=False, timeout=timeout)
  | 
  | tests/entrypoints/openai/test_openai_schema.py:106: 
  | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  | .venv/lib/python3.12/site-packages/schemathesis/models.py:418: in call
  |     response = self.operation.schema.transport.send(
  | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  | 
  | self = <schemathesis.transports.RequestsTransport object at 0x7992cd34cb30>
  | case = Case(body={'messages': [{'content': 'K', 'name': ";\x83\U000164edC\U00049bcc,\U0010bb3eõæÃ\x9evUàk?\U000f8fd6\n'Ðß'\U0...09, 'Ó': -1.0091098906725308e+16}], '\x1e=8\U00051cc9Ý': {'e': [None], '\U0003bf24ЧÇ': []}}], '': [-1.192092896e-07]})
  | session = <requests.sessions.Session object at 0x7992c7aa0380>, base_url = None
  | headers = None, params = None, cookies = None
  | kwargs = {'timeout': 30, 'verify': False}
  | requests = <module 'requests' from '/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/requests/__init__.py'>
  | ReadTimeoutError = <class 'urllib3.exceptions.ReadTimeoutError'>
  | 
  |     def send(
  |         self,
  |         case: Case,
  |         *,
  |         session: requests.Session | None = None,
  |         base_url: str | None = None,
  |         headers: dict[str, Any] | None = None,
  |         params: dict[str, Any] | None = None,
  |         cookies: dict[str, Any] | None = None,
  |         **kwargs: Any,
  |     ) -> requests.Response:
  |         import requests
  |         from urllib3.exceptions import ReadTimeoutError
  |     
  |         data = self.serialize_case(case, base_url=base_url, headers=headers, params=params, cookies=cookies)
  |         data.update(kwargs)
  |         data.setdefault("timeout", DEFAULT_RESPONSE_TIMEOUT / 1000)
  |         if session is None:
  |             validate_vanilla_requests_kwargs(data)
  |             session = requests.Session()
  |             close_session = True
  |         else:
  |             close_session = False
  |         verify = data.get("verify", True)
  |         try:
  |             with case.operation.schema.ratelimit():
  |                 response = session.request(**data)  # type: ignore
  |         except (requests.Timeout, requests.ConnectionError) as exc:
  |             if isinstance(exc, requests.ConnectionError):
  |                 if not isinstance(exc.args[0], ReadTimeoutError):
  |                     raise
  |                 req = requests.Request(
  |                     method=data["method"].upper(),
  |                     url=data["url"],
  |                     headers=data["headers"],
  |                     files=data.get("files"),
  |                     data=data.get("data") or {},
  |                     json=data.get("json"),
  |                     params=data.get("params") or {},
  |                     auth=data.get("auth"),
  |                     cookies=data["cookies"],
  |                     hooks=data.get("hooks"),
  |                 )
  |                 request = session.prepare_request(req)
  |             else:
  |                 request = cast(requests.PreparedRequest, exc.request)
  |             timeout = 1000 * data["timeout"]  # It is defined and not empty, since the exception happened
  |             code_message = case._get_code_message(case.operation.schema.code_sample_style, request, verify=verify)
  |             message = f"The server failed to respond within the specified limit of {timeout:.2f}ms"
  | >           raise get_timeout_error(case.operation.verbose_name, timeout)(
  |                 f"\n\n1. {failures.RequestTimeout.title}\n\n{message}\n\n{code_message}",
  |                 context=failures.RequestTimeout(message=message, timeout=timeout),
  |             ) from None
  | E           schemathesis.exceptions.CheckFailed: 
  | E           
  | E           1. Response timeout
  | E           
  | E           The server failed to respond within the specified limit of 30000.00ms
  | E           
  | E           Reproduce with: 
  | E           
  | E               curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": "K", "name": ";\u0083\ud819\udcedC\ud8e6\udfcc,\udbee\udf3e\u00f5\u00e6\u00c3\u009evU\u00e0k?\udba3\udfd6\n'"'"'\u00d0\u00df'"'"'\ud894\udf99", "role": "function", "\u00f1": [[-1.7976931348623157e+308], -15141, false], "\u00e3\ud97c\udf74x\u0084": {}, "\udace\udc0f": [{"\udbec\udc3b\u0017 ": [-1.7976931348623157e+308], "\u009ae\u00c4L\"\u0011\u009c\u00f5\u0014\u00de": [null], "": [[true], []]}, [[]]], "\u0082ti*\u00d1\u00d4\u00b6CAq": [], "\udb60\ude86\u0085(\u00b5\udb4d\udc0c\u0007\u0097\u00ec\u0082\u00e6\udbef\uddb0\u0015": [[-1.7976931348623157e+308], -15141, false], "\u00ce\u00e9K\u00b3\ud97d\udfccN\u00e3\u00caMlm\u007f": [{"\b\udb28\udf9f\u00ab\u00ea\u001emS\u0098u\udbc2\uddbc\u0086\udaeb\udd9e\u0004\ud94f\uddee\u0094\u008b2\uda24\udc2d\u0083\ud868\ude84\u0084": -2.225073858507203e-309, "\u00d3": -1.0091098906725308e+16}], "\u001e=8\ud907\udcc9\u00dd": {"e": [null], "\ud8af\udf24\u00d0\u00a7\u00c7": []}}], "": [-1.192092896e-07]}' --insecure http://localhost:57477/v1/chat/completions
  | 
  | .venv/lib/python3.12/site-packages/schemathesis/transports/__init__.py:202: CheckFailed
  | 
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 1087, in _execute_once_for_engine
    |     result = self.execute_once(data)
    |              ^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 1024, in execute_once
    |     result = self.test_runner(data, run)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 729, in default_executor
    |     return function(data)
    |            ^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 996, in run
    |     return test(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py", line 83, in test_openapi_stateless
    |     def test_function(*args: Any, **kwargs: Any) -> Any:
    |                ^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/hypothesis/core.py", line 906, in test
    |     result = self.test(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py", line 89, in collecting_wrapper
    |     wrapped_test = hypothesis.seed(seed)(wrapped_test)
    |         ^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py", line 86, in collecting_wrapper
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py", line 85, in test_function
    |     return test(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/tests/entrypoints/openai/test_openai_schema.py", line 106, in test_openapi_stateless
    |     case.call_and_validate(verify=False, timeout=timeout)
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/models.py", line 590, in call_and_validate
    |     response = self.call(base_url, session, headers, **kwargs)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/models.py", line 418, in call
    |     response = self.operation.schema.transport.send(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/transports/__init__.py", line 202, in send
    |     raise get_timeout_error(case.operation.verbose_name, timeout)(
    | schemathesis.exceptions.CheckFailed: 
    | 
    | 1. Response timeout
    | 
    | The server failed to respond within the specified limit of 30000.00ms
    | 
    | Reproduce with: 
    | 
    |     curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": "K", "name": ";\u0083\ud819\udcedC\ud8e6\udfcc,\udbee\udf3e\u00f5\u00e6\u00c3\u009evU\u00e0k?\udba3\udfd6\n'"'"'\u00d0\u00df'"'"'\ud894\udf99", "role": "function", "\u00f1": [[-1.7976931348623157e+308], -15141, false], "\u00e3\ud97c\udf74x\u0084": {}, "\udace\udc0f": [{"\udbec\udc3b\u0017 ": [-1.7976931348623157e+308], "\u009ae\u00c4L\"\u0011\u009c\u00f5\u0014\u00de": [null], "": [[true], []]}, [[]]], "\u0082ti*\u00d1\u00d4\u00b6CAq": [], "\udb60\ude86\u0085(\u00b5\udb4d\udc0c\u0007\u0097\u00ec\u0082\u00e6\udbef\uddb0\u0015": [[-1.7976931348623157e+308], -15141, false], "\u00ce\u00e9K\u00b3\ud97d\udfccN\u00e3\u00caMlm\u007f": [{"\b\udb28\udf9f\u00ab\u00ea\u001emS\u0098u\udbc2\uddbc\u0086\udaeb\udd9e\u0004\ud94f\uddee\u0094\u008b2\uda24\udc2d\u0083\ud868\ude84\u0084": -2.225073858507203e-309, "\u00d3": -1.0091098906725308e+16}], "\u001e=8\ud907\udcc9\u00dd": {"e": [null], "\ud8af\udf24\u00d0\u00a7\u00c7": []}}], "": [-1.192092896e-07]}' --insecure http://localhost:57477/v1/chat/completions
    | 
    +------------------------------------
---------------------------------------------------------- Captured stdout setup ----------------------------------------------------------
INFO 05-20 12:42:15 [__init__.py:30] Available plugins for group vllm.general_plugins:
INFO 05-20 12:42:15 [__init__.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-20 12:42:15 [__init__.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 05-20 12:42:15 [__init__.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 05-20 12:42:15 [__init__.py:44] plugin lora_filesystem_resolver loaded.
INFO 05-20 12:42:27 [weight_utils.py:291] Using model weights format ['*.safetensors']
INFO 05-20 12:42:27 [weight_utils.py:341] No model.safetensors.index.json found in remote.
INFO 05-20 12:42:32 [__init__.py:248] Automatically detected platform cuda.
INFO 05-20 12:42:37 [__init__.py:30] Available plugins for group vllm.general_plugins:
INFO 05-20 12:42:37 [__init__.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-20 12:42:37 [__init__.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 05-20 12:42:37 [__init__.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 05-20 12:42:37 [__init__.py:44] plugin lora_filesystem_resolver loaded.
INFO 05-20 12:42:39 [api_server.py:1289] vLLM API server version 0.8.5.dev708+g451da4bcb
INFO 05-20 12:42:39 [cli_args.py:300] non-default args: {'port': 57477, 'task': 'generate', 'trust_remote_code': True, 'seed': 0, 'max_model_len': 2048, 'enforce_eager': True, 'limit_mm_per_prompt': {'image': 2}, 'max_num_seqs': 5}
INFO 05-20 12:42:51 [config.py:2112] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 05-20 12:42:51 [cuda.py:87] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 05-20 12:42:56 [__init__.py:248] Automatically detected platform cuda.
INFO 05-20 12:43:01 [core.py:427] Waiting for init message from front-end.
INFO 05-20 12:43:01 [core.py:61] Initializing a V1 LLM engine (v0.8.5.dev708+g451da4bcb) with config: model='HuggingFaceTB/SmolVLM-256M-Instruct', speculative_config=None, tokenizer='HuggingFaceTB/SmolVLM-256M-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=HuggingFaceTB/SmolVLM-256M-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [], "max_capture_size": 0}
INFO 05-20 12:43:01 [__init__.py:30] Available plugins for group vllm.general_plugins:
INFO 05-20 12:43:01 [__init__.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-20 12:43:01 [__init__.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 05-20 12:43:01 [__init__.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 05-20 12:43:01 [__init__.py:44] plugin lora_filesystem_resolver loaded.
WARNING 05-20 12:43:01 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7dfa41c164b0>
INFO 05-20 12:43:02 [parallel_state.py:1079] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 05-20 12:43:05 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 05-20 12:43:05 [gpu_model_runner.py:1503] Starting to load model HuggingFaceTB/SmolVLM-256M-Instruct...
INFO 05-20 12:43:05 [cuda.py:216] Using Flash Attention backend on V1 engine.
INFO 05-20 12:43:06 [weight_utils.py:291] Using model weights format ['*.safetensors']
INFO 05-20 12:43:06 [weight_utils.py:341] No model.safetensors.index.json found in remote.
INFO 05-20 12:43:06 [default_loader.py:279] Loading weights took 0.24 seconds
INFO 05-20 12:43:06 [gpu_model_runner.py:1521] Model loading took 0.4899 GiB and 0.755285 seconds
INFO 05-20 12:43:06 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 2 image items of the maximum feature size.
INFO 05-20 12:43:08 [kv_cache_utils.py:637] GPU KV cache size: 1,567,632 tokens
INFO 05-20 12:43:08 [kv_cache_utils.py:640] Maximum concurrency for 2,048 tokens per request: 765.45x
INFO 05-20 12:43:08 [core.py:163] init engine (profile, create kv cache, warmup model) took 1.55 seconds
INFO 05-20 12:43:08 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 97977
INFO 05-20 12:43:09 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:57477
INFO 05-20 12:43:09 [launcher.py:28] Available routes are:
INFO 05-20 12:43:09 [launcher.py:36] Route: /openapi.json, Methods: HEAD, GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /docs, Methods: HEAD, GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /redoc, Methods: HEAD, GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /health, Methods: GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /load, Methods: GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /ping, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /ping, Methods: GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /version, Methods: GET
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /pooling, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /classify, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /score, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /rerank, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /invocations, Methods: POST
INFO 05-20 12:43:09 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     127.0.0.1:53264 - "GET /health HTTP/1.1" 200 OK
INFO:     127.0.0.1:53272 - "GET /openapi.json HTTP/1.1" 200 OK
---------------------------------------------------------- Captured stderr setup ----------------------------------------------------------
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  4.58it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  4.58it/s]

INFO:     Started server process [219237]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
------------------------------------------------------------ Captured log call ------------------------------------------------------------

============================================================ warnings summary =============================================================
.venv/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    ref_error: type[Exception] = jsonschema.RefResolutionError,

.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `method` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `endpoint` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `tag` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `operation_id` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless
  /home/dxia/src/github.com/vllm-project/vllm/vllm/engine/arg_utils.py:58: DeprecationWarning: Passing a JSON argument as a string containing comma separated key=value pairs is deprecated. This will be removed in v0.10.0. Please use a JSON string instead.
    return cast(T, nullable_kvs(val))

tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/specs/openapi/references.py:11: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    from jsonschema.exceptions import RefResolutionError

tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/specs/openapi/references.py:51: DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
    class InliningResolver(jsonschema.RefResolver):

tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless
  /home/dxia/src/github.com/vllm-project/vllm/.venv/lib/python3.12/site-packages/schemathesis/specs/openapi/schemas.py:89: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    SCHEMA_PARSING_ERRORS = (KeyError, AttributeError, jsonschema.exceptions.RefResolutionError)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= short test summary info =========================================================
(verbose_name='POST /v1/chat/completions') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/chat/completions] - hypothesis.errors.FlakyFailure: Hypothesis test_openapi_stateless(case=) produces unreliable results: Falsified on the first call but ...
=========================================== 1 failed, 1 passed, 9 warnings in 203.01s (0:03:23) ===========================================

[hmellor]

It'd be nice to add a comment explaining that this specific method+path requires a longer timeout.

[davidxia]

yup, done

[davidxia]

Sorry I meant is vLLM doing things during those 10 seconds.

Oh, I'm not sure. How can I check? Even when I run this test for only that endpoint with

@schema.include(method="POST", path="/v1/chat/completions").parametrize()
...
definitely test_openapi_stateless(case: schemathesis.Case):

it still times out without the changes in d84a35e.

[davidxia]

Ready for another review. cc @hmellor

[hmellor]

LGTM, just a nit about adding a comment and moving some magic numbers to globals

[hmellor]

It'd be nice to add a comment explaining that this specific method+path requires a longer timeout.

[davidxia]

@hmellor thanks for the review and suggestions. updated!

[hmellor]

LGTM, thanks for doing this!

[DarkLight1337]

Please merge from main to fix CI

[davidxia]

Please merge from main to fix CI

thanks, done. @hmellor you'll probably need to enable auto-merge again