Repository navigation
feat: gate boundary snapshots by model capability - #1507
Merged
Merged
Conversation
Publish the loaded model snapshot-reuse capability from the worker before the provider is marked ready, then pass it through AppState into every chat-template preparation path that can build prompt-cache context. Dense-KV models now skip the extra history-boundary render entirely while snapshot-reuse families keep the existing boundary invariant. The manual MLXCEL_DISABLE_BOUNDARY_SNAPSHOT kill switch remains a stricter gate on top of the automatic capability check. Validated with targeted chat_request history-boundary tests, cache-on preserve_thinking tests, and ModelProvider single-stream fixture tests. Refs #1153
Keep HTTP request preparation fail-open before the worker publishes loaded=true so a request queued during startup cannot miss the first history-boundary snapshot on a snapshot-reuse model. Once the worker is loaded, the route follows the published snapshot-reuse capability and dense-KV models skip the history-boundary render. Refs #1153
inureyes
added
type:performance
Record the snapshot capability gate implementation, validation coverage, and remaining real-checkpoint qualification limits while the PR is still open so the report is included in the merge. Refs #1153
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gate prompt-cache history-boundary rendering on the loaded model snapshot-reuse capability so dense-KV deployments do not pay a second template render and tokenizer pass that the scheduler cannot consume.
What changed
ModelProvidersnapshot-reuse capability flag from batch, legacy, and XLA workers before the provider is marked loaded.AppStateinto chat, Responses, Anthropic, and prompt-inspection preparation paths.loaded=true, so startup-queued requests cannot miss the first boundary snapshot for a snapshot-capable model.preserve_thinkingcache-on defaulting independent from snapshot reuse, while requiring snapshot capability before producinghistory_promptafter readiness.Test plan
cargo test --lib server::chat_request::tests::history_render -- --nocapturecargo test --lib server::chat_request::tests::prompt_cache_on -- --nocapturecargo test --lib pre_reserved_single_stream_enqueue_does_not_double_reserve -- --nocapturecargo test --lib scheduler_paths_do_not_create_single_stream_reservations -- --nocapturecargo clippy --lib --tests -- -D warningsValidation limits
models/checkpoint directory is present; this PR validates the dispatch gate and prefix invariant with unit tests, while hardware/checkpoint parity remains a qualified-machine check.Closes #1153