hindsight_api.engine.providers.openai_compatible_llm: Provider returned empty message content

[RyanOvO]

Background:
I deployed the qwen3.5-27B model on the ascend server based on the vllm-ascend Docker image, with a context length of 64K, as follows:

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True
# To reduce memory fragmentation and avoid out of memory
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=512
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export TASK_QUEUE_ENABLE=1

ASCEND_RT_VISIBLE_DEVICES=5,6 \
vllm serve /home/developer/Qwen3.5-27B-w8a8-mtp \
--host 0.0.0.0 \
--port 1025 \
--data-parallel-size 1 \
--tensor-parallel-size 2 \
--seed 1024 \
--quantization ascend \
--served-model-name qwen3p5-llm \
--max-num-seqs 32 \
--max-model-len 65536 \
--max-num-batched-tokens 8096 \
--trust-remote-code \
--gpu-memory-utilization 0.92 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--no-enable-prefix-caching \
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_cpu_binding":true}' \
--enforce-eager \

But the current error is as follows:

My configuration:

     HINDSIGHT_API_DATABASE_URL: 'xxxx'
      HINDSIGHT_API_LLM_PROVIDER: openai
      HINDSIGHT_API_LLM_API_KEY: 123456
      HINDSIGHT_API_LLM_BASE_URL: 'http://xxx/v1'
      HINDSIGHT_API_LLM_MODEL: 'qwen3p5-model'

Is it related to the context length of the model? Or is there any key configuration item missing in hindsight?

[benfrank241]

Thanks for the detailed setup — this isn't a context-length problem, and you're not missing a required config field. The give-away is in the log lines:

scope=verification ... Provider returned empty message content
(openai/qwen3p5-llm, scope=verification, finish_reason=length, has_tool_calls=False, refusal=False)

Two things are happening:

1. It's the startup verification probe, not your real workload. On boot Hindsight makes one tiny "Say 'ok'" call with max_tokens=100 just to confirm the endpoint is reachable. It failed 3×, so you see LLM connection verification failed ... Server will start but LLM-dependent operations may fail. This step is best-effort — the server still comes up.

2. finish_reason=length + empty content is the reasoning parser eating the whole budget. You're serving with --reasoning-parser qwen3, so vLLM splits the model's chain-of-thought into a separate reasoning_content field and only puts the final answer in content. With a 100-token cap, Qwen3.5 spends the entire budget "thinking" and hits the length limit before it ever emits any content — so content comes back empty. Hindsight currently reads content (it doesn't fall back to reasoning_content), hence the error.

We're tracking the server-side fix for this in #2630 (the verification probe shouldn't be defeatable by a reasoning model). In the meantime, any of these will get you unblocked:

Recommended — serve the model plain for Hindsight. Drop --reasoning-parser qwen3, --enable-auto-tool-choice, and --tool-call-parser qwen3_coder from the vLLM command for this endpoint. Hindsight uses JSON-mode (response_format), not function-calling, so the tool-call flags aren't needed, and it already strips inline <think>…</think> blocks from the response itself. Returning the thinking inline in content is exactly the shape Hindsight expects, and it removes the empty-content failure mode for real retain/recall calls too — not just verification. If the model supports it, disabling thinking (enable_thinking=false / /no_think) is even cleaner.

Quick unblock — skip the probe. Set HINDSIGHT_API_SKIP_LLM_VERIFICATION=true. That silences the boot-time check, but note it only hides the symptom: if you keep the reasoning parser, real retain/recall calls can still return empty content whenever the model over-thinks relative to the token budget. So I'd combine this with the change above rather than rely on it alone.

One more thing to double-check: your posted config has HINDSIGHT_API_LLM_MODEL: 'qwen3p5-model', but the logs show the model resolving as qwen3p5-llm (which matches your --served-model-name qwen3p5-llm). Make sure the configured model name matches the served name.

If you drop the reasoning/tool-call parsers and still see an error, can you share a log line from an actual retain (scope won't say verification)? That'll tell us whether anything else needs adjusting. Thanks for trying Hindsight on Ascend/vLLM! 🙏

[RyanOvO]

i fixed it with there:

 HINDSIGHT_API_LLM_PROVIDER: openai
      HINDSIGHT_API_LLM_API_KEY: '123456'
      HINDSIGHT_API_LLM_BASE_URL: xxxx/v1'
      HINDSIGHT_API_LLM_MODEL: 'qwen3p5-llm'
      HINDSIGHT_API_LLM_MAX_TOKENS: 32768
      HINDSIGHT_API_LLM_EXTRA_BODY: '{"chat_template_kwargs": {"enable_thinking": false}}'
      HINDSIGHT_API_SKIP_LLM_VERIFICATION: 'true'
      HINDSIGHT_API_RETAIN_CHUNK_SIZE: 8192
      HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: 16384