Repository navigation
feat: Post-completion incremental warm-up snapshot for the next turn's history prefix #1144
Description
Part of epic #1148.
Problem / Background
With the history-boundary snapshot in place (see #1143), turn N+1 gets a hit covering the history rendered up to turn N's user message, but the previous assistant reply still has to be prefilled on every turn. For a 31B model that is avoidable work sitting on the foreground request's critical path.
This applies to the entire snapshot-only class (all supports_snapshot_reuse() families, including Gemma 4 and the Qwen 3.5 GatedDeltaNet hybrid). For the recurrent-state families in particular, partial restore is impossible, so #1143 plus this warm-up is their complete reuse story.
The boundary snapshot from the previous turn plus a small delta-prefill of the re-rendered reply produces exactly the state the next turn needs, and nothing about that computation has to happen while the user is waiting.
Proposed Solution
After a healthy finish, enqueue a background warm-up job:
- Restore the boundary snapshot captured during the finished request's prefill.
- Delta-prefill the assistant reply as re-rendered history (typically tens of tokens: the reply retokenized in history form, post thinking-strip).
- Store the resulting snapshot keyed to the next turn's expected history prefix.
In steady state each warm-up only prefills the newest exchange, so the incremental cost stays bounded regardless of conversation length.
Depends on #1143 (the warm-up job restores that snapshot as its starting point).
Acceptance Criteria
- Turn N+1 against
gemma-4-31b-it-4bitprefills only the new user message:cached_tokenscovers system + all prior turns + previous reply (deferred to the epic-level verification run: only small checkpoints were exercised on this branch) - The same holds for
qwen3.5-0.8b-4bitwithenable_thinking=false(divergence class (a), the empty<think>\n\n</think>\n\ninjection): turn 2 cached 194 of 227 with the warm-up versus 150 of 227 without. The 4b default-thinking variant was not usable as a fixture at this size because the 0.8b model never closes its<think>block within any token budget tried, socontentcomes back empty and there is no reply text to echo - Warm-up does not block or delay foreground requests: under two concurrent clients driving 12 back-to-back requests in 2.6s,
snapshot_warmups_runstayed pinned at 0 with 10 queue skips and zero foreground errors, then reached 2 once the load stopped - Measured through the production
/v1/chat/completionspath with counter-based verification via/v1/cache/stats(snapshot_warmups_run/snapshot_warmups_skippedadded for exactly this) - Warm-up failures are non-fatal: every path in
run_next_prompt_cache_warmupreturns silently, the job carries no response channel, and the conversation keeps the feat: Donate a history-boundary snapshot during prefill for snapshot-only families #1143 boundary snapshot
Technical Considerations
- The re-rendered history form of the reply differs from the generated form in three verified ways (epic epic: Multi-turn prompt cache never hits for snapshot-only families (Gemma 4, Qwen 3.5, SSM hybrids) #1148: generation-prompt-only scaffolds, thinking stripped from history, retokenization drift such as falcon-h1's 120 sampled vs 118 re-tokenized tokens). That is exactly why the delta must be prefilled from the re-rendered template output, not from the generated token ids.
- Scheduling: the warm-up job should ride the existing batch scheduler at background priority, or run in idle gaps; it must yield to foreground admission.
- Interaction with capacity: each warm-up inserts a longer snapshot for the same session; the session-chain supersede work in feat: Snapshot capacity hygiene: session-chain supersede and configurable budget #1146 keeps this from doubling resident bytes per turn.
Activity
inureyes commentedon Aug 14, 2026
Epic-level verification on
gemma-4-31b-it-4bitis done, closing the acceptance item this issue deferred to the epic run. Three-turn chat through the production/v1/chat/completionspath on maina206f089(Apple M1 Ultra, default 512 MiB snapshot budget, counter-based via/v1/cache/stats, ~100-token system prompt so the boundary clearsmin_prefix_tokens=32):cached_tokens/ promptsnapshot_hitsdeltaThe uncached remainder each turn is the new user message plus the template boundary, which is the designed behavior: the snapshot covers system plus all prior turns plus the previous reply.
qwen3.5-4b-4bit(default thinking) passed the same scenario with turn 2 at 94/124 and turn 3 at 119/147, and thellama-3.2-1b-instructdense control stayed on the trie path with every snapshot counter at 0. Two observations for the record: the 512 MiB default budget did produce 3 LRU evictions across the run (one 31B snapshot is 230-280 MB and a conversation holds two chains), but the load-bearing entry survived every turn, so the pressure is real yet not hit-breaking at this depth; and with a short first turn (35-token prompt, no system message) the boundary falls under the 32-token floor and turn 2 stays cold withreject_snapshot_divergedcorrectly reporting the geometry (31/95), which is the designed floor rather than a defect.chore(release): v0.5.1
fix(server): resolve warm-up kwargs through the shared helper
feat(server): warm the next turn's history prefix in the background (l…
fix(kimi_k3): close the review findings on the XTML chat path
fix(kimi_k3): close the review findings on the XTML chat path
fix(kimi_k3): close the review findings on the XTML chat path
fix(server): extend reasoning re-echo to responses, messages, warm-up (…