Skip to content

feat(serve): OpenAI client parity on the multi-slot path — tools, reasoning_effort, sampling, usage, tool-iteration reuse - #609

Open
alpineQ wants to merge 3 commits into
warpfront:betafrom
alpineQ:feat/multislot-openai-parity
Open

feat(serve): OpenAI client parity on the multi-slot path — tools, reasoning_effort, sampling, usage, tool-iteration reuse#609
alpineQ wants to merge 3 commits into
warpfront:betafrom
alpineQ:feat/multislot-openai-parity

Conversation

@alpineQ

@alpineQ alpineQ commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #603. Makes the multi-slot path serve real OpenAI agent clients. Verified end-to-end against a production agent gateway (~10k-token system prompt, 31 tool schemas, per-chat title generation): tool calls round-trip (including two parallel calls in one turn), tool iterations reuse the session KV, interleaved title calls no longer evict conversations, and clients finally see real usage numbers.

  • Prompt: JinjaChatFrame::render_messages with the model's chat template and the request's tools — the same rendering the daemon path uses; the hand-rolled ChatML frame stays as the no-template fallback. Typed Message / ToolCall mapping from OpenAI messages (assistant tool_calls echo, tool role, tool_call_id).
  • Emission: the slots loop drives Qwen35Emit (begin / observe / finish) — the same EosFilter + think/tool-routing + post-hoc-grammar aggregate the daemon drives, so both backends classify output identically. Tool grammar is gated by qwen35_grammar_on exactly like the daemon (withheld tools ⇒ matcher inactive; the Hermes-JSON envelope matcher otherwise rejects qwen3.6's native XML at the second body token). Slots-specific config: no stop sequences yet, and max_think = 0 — a think force-close needs take_forced token injection into the slot's KV, which the engine does not support. Both are listed as known gaps rather than silently ignored.
  • Tool-result iterations: SessionTable::find_reentry matches an EQUAL convo key (same user turns — a tool round), and the client sends a continuation_suffix_tool_results suffix rendered in the official template's <tool_response> shape. Same strict-KV-extension construction as continuation_suffix: nothing is re-rendered, so the DeltaNet state stays valid. Without it, every agent iteration cold-prefilled the conversation, because find_continuation requires the key to be one user turn ahead.
  • Sampling: SubmitRequest carries temperature / top_p / top_k / seed; admit installs them into the slot's SlotSampleParams — per SP4's own session-state design. Absent fields mean greedy, the previous behaviour.
  • Usage: Event::Accepted { reused, prefill } / Event::Done { generated }; the completion's done envelope carries cached_tokens / prefill_tokens / tokens, which completion_usage already projects. Closes the "usage.completion_tokens is always 0" gap from feat(serve): multi-slot batched attention — 3–4 concurrent agents on one GPU (SP1–SP7) #573's self-review and makes prefix-cache hits client-visible via prompt_tokens_details.cached_tokens.
  • reasoning_effort: none / reasoning_budget_tokens: 0 / max_think_tokens: 1enable_thinking = false rendering, matching the daemon's contract. Before this, a reasoning_effort: "none", max_tokens: 128 structured call burned its whole budget inside <think> and returned empty content.

Deliberately not included: response_format (#604) and GET /props (#605) — both are RFCs awaiting direction.

Stacked on #607 and #608 — review those first; this branch contains their commits.

Which crate(s) does this touch?

  • crates/hipfire-runtime (serve protocol event fields, continuation_suffix_tool_results, SessionTable::find_reentry)
  • crates/hipfire-arch-qwen35 (serve_engine: counter emission, per-request sample params, reentry match)
  • examples / daemon (hipfire-cli serve slots path)

Test plan

  • cargo build --release --workspace --features deltanet clean
  • cargo test --lib --workspace --features deltanet passes (incl. new find_reentry and serve-protocol tests)
  • End-to-end against an OpenAI agent gateway on gfx1100 24 GB, qwen3.6-35b-a3b.mq4r, 2 × 16384: titles parse, finish_reason = tool_calls with correct arguments, a tool iteration reuses the session — engine trace continuation HIT — reused 375 of 423 tokens, response usage.prompt_tokens_details.cached_tokens: 375; chat switches and interleaved title calls reuse 10k+ contexts with 30–50-token prefills
  • speed.arch-fast passes in the gate below
  • Change gate run and telemetry pasted below

scripts/serve_concurrency_gate.sh not run as acceptance evidence here. Its speedup floor (1.30×) was locked on a 128 GB gfx1151 box; on this 24 GB host the gate cannot produce a comparison, because the beta side it would be compared against is exactly what #602 fixes — beta's engine OOMs before the server accepts a request. Worth a re-measure on hardware where both arms can run.

change_gate telemetry
**change_gate: FAIL**

host gfx=`gfx1100` rocm=`6.18.37` models_dir=`/home/alpineq/.hipfire/models` · `80a572c824dcbbcdddac1cc5f6d1a7e91d7c4dd6`..`e59c08b79224f7e71aa98bc08e70dd19d2953579` dirty · est=16.55min actual=55.9s

### Routes RUN
| route                   | status | duration |
| ----------------------- | ------ | -------- |
| serve.agentic.a3b-fast  | fail   | 1.1s     |
| serve.battery.qwen35-4b | fail   | 0.1s     |
| serve.battery.qwen35-9b | fail   | 0.1s     |
| speed.arch-fast         | pass   | 19.7s    |
| unit.arch-qwen35        | pass   | 1.6s     |
| unit.diff-check         | pass   | 0.0s     |
| unit.env-docs           | fail   | 0.2s     |
| unit.hipfire-cli        | pass   | 3.6s     |
| unit.hipfire-config     | pass   | 0.1s     |
| unit.hipfire-runtime    | pass   | 10.7s    |
| unit.no-gpu-control     | fail   | 18.7s    |

### Routes NOT RUN
| route | reason |
| ----- | ------ |
| —     | —      |

_Blocked or excluded routes mean coverage is incomplete — this report is not an admission that unrun surfaces are safe._
Pre-existing failures on this host, verified against a clean `upstream/beta` worktree and NOT introduced here:
  • unit.env-docs — 173 findings, every one in a crate this diff does not touch (quantize, gemma4, deepseek4, redline, ...). No finding names a file in this PR.
  • unit.no-gpu-controlhipfire-tui's bounded_run_stdout_fill_is_not_falsely_timed_out fails identically on pristine beta (timing-sensitive dashboard test).
  • serve.agentic.a3b-fastzero tokens emitted on pristine beta too.
  • serve.battery.qwen35-4b / -9b — the route argv asks --thinking med with max_tokens 180/300, which serve_harness.py rejects before touching the GPU (the harness defect PR fix(serve): emit Qwen reasoning content #572 documented).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant