Skip to content

feat: Post-completion incremental warm-up snapshot for the next turn's history prefix #1144

Closed

Description

@inureyes

Part of epic #1148.

Problem / Background

With the history-boundary snapshot in place (see #1143), turn N+1 gets a hit covering the history rendered up to turn N's user message, but the previous assistant reply still has to be prefilled on every turn. For a 31B model that is avoidable work sitting on the foreground request's critical path.

This applies to the entire snapshot-only class (all supports_snapshot_reuse() families, including Gemma 4 and the Qwen 3.5 GatedDeltaNet hybrid). For the recurrent-state families in particular, partial restore is impossible, so #1143 plus this warm-up is their complete reuse story.

The boundary snapshot from the previous turn plus a small delta-prefill of the re-rendered reply produces exactly the state the next turn needs, and nothing about that computation has to happen while the user is waiting.

Proposed Solution

After a healthy finish, enqueue a background warm-up job:

  1. Restore the boundary snapshot captured during the finished request's prefill.
  2. Delta-prefill the assistant reply as re-rendered history (typically tens of tokens: the reply retokenized in history form, post thinking-strip).
  3. Store the resulting snapshot keyed to the next turn's expected history prefix.

In steady state each warm-up only prefills the newest exchange, so the incremental cost stays bounded regardless of conversation length.

Depends on #1143 (the warm-up job restores that snapshot as its starting point).

Acceptance Criteria

  • Turn N+1 against gemma-4-31b-it-4bit prefills only the new user message: cached_tokens covers system + all prior turns + previous reply (deferred to the epic-level verification run: only small checkpoints were exercised on this branch)
  • The same holds for qwen3.5-0.8b-4bit with enable_thinking=false (divergence class (a), the empty <think>\n\n</think>\n\n injection): turn 2 cached 194 of 227 with the warm-up versus 150 of 227 without. The 4b default-thinking variant was not usable as a fixture at this size because the 0.8b model never closes its <think> block within any token budget tried, so content comes back empty and there is no reply text to echo
  • Warm-up does not block or delay foreground requests: under two concurrent clients driving 12 back-to-back requests in 2.6s, snapshot_warmups_run stayed pinned at 0 with 10 queue skips and zero foreground errors, then reached 2 once the load stopped
  • Measured through the production /v1/chat/completions path with counter-based verification via /v1/cache/stats (snapshot_warmups_run / snapshot_warmups_skipped added for exactly this)
  • Warm-up failures are non-fatal: every path in run_next_prompt_cache_warmup returns silently, the job carries no response channel, and the conversation keeps the feat: Donate a history-boundary snapshot during prefill for snapshot-only families #1143 boundary snapshot

Technical Considerations

Activity

  1. added
    type:enhancementNew features, capabilities, or significant additions
    area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)
    and removed on Aug 14, 2026
  2. inureyes commented on Aug 14, 2026

    @inureyes
    MemberAuthor

    Epic-level verification on gemma-4-31b-it-4bit is done, closing the acceptance item this issue deferred to the epic run. Three-turn chat through the production /v1/chat/completions path on main a206f089 (Apple M1 Ultra, default 512 MiB snapshot budget, counter-based via /v1/cache/stats, ~100-token system prompt so the boundary clears min_prefix_tokens=32):

    Turn cached_tokens / prompt uncached remainder snapshot_hits delta
    1 0 / 103 103 (cold) +1 (warm-up internal)
    2 147 / 171 24 +2
    3 187 / 210 23 +1

    The uncached remainder each turn is the new user message plus the template boundary, which is the designed behavior: the snapshot covers system plus all prior turns plus the previous reply. qwen3.5-4b-4bit (default thinking) passed the same scenario with turn 2 at 94/124 and turn 3 at 119/147, and the llama-3.2-1b-instruct dense control stayed on the trie path with every snapshot counter at 0. Two observations for the record: the 512 MiB default budget did produce 3 LRU evictions across the run (one 31B snapshot is 230-280 MB and a conversation holds two chains), but the load-bearing entry survived every turn, so the pressure is real yet not hit-breaking at this depth; and with a short first turn (35-token prompt, no system message) the boundary falls under the 32-token floor and turn 2 stays cold with reject_snapshot_diverged correctly reporting the geometry (31/95), which is the designed floor rather than a defect.

  3. added a commit that references this issue on Aug 20, 2026
    8822b5a
  4. self-assigned this
    on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)priority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions