Repository navigation
fix(inference): align context, batching, RoPE, and runtime override semantics #1450
Description
Parent
Problem
Several common b10621 options affect model arithmetic or context management independently of the GGML hardware backend: context size, logical/physical batch size, kept prompt tokens, SWA cache sizing, RoPE scaling/base/frequency, YaRN controls, escape processing, tensor checks, and model metadata overrides. mlxcel currently shares some spellings but does not establish equivalent defaults or effects.
Scope
- Audit and align
--ctx-size,--batch-size,--ubatch-size,--keep, and--swa-fullvalue domains, defaults, and truncation behavior. - Implement compatible RoPE scaling, scale, frequency base/scale, and YaRN parameters only for model families whose Rust implementation can honor them.
- Align escape/no-escape processing at CLI prompt boundaries without changing JSON request escaping.
- Classify performance-timing, tensor-check, metadata-override, and operation-offload flags as supported, not applicable, or deferred.
- Align warmup/no-warmup startup behavior and prove whether the empty warmup run allocates the same caches and surfaces the same startup failures.
- Reject model-family or checkpoint combinations that cannot safely apply a requested runtime override.
Acceptance criteria
- CLI and canonical environment precedence tests cover every common runtime option and sentinel. (RoPE/YaRN group:
src/cli/rope_args_tests.rs,tests/llama_compat_manifest.rs. The remaining runtime options in this shard were audited and their divergences recorded incompat/llama-server/b10621/runtime-and-context.jsonrather than claimed.) - Resolved context and batch settings are observable in startup diagnostics and native props. (
/propsnow reportsn_ctx,n_batch,n_ubatch,n_batch_decode,n_kv_max; startup logs the resolved per-slot context, slot count and prefill chunk.) - Truncation/keep/SWA tests cover full context, sliding windows, rotating cache, and MoE/VLM model families where applicable. (Not implemented.
--keep,--swa-full,--context-shift,n_keepandn_discardstaydeferredwith their observable divergences recorded; the retained prefix is still the fixed 4-token attention sink and sliding-window families build their ownRotatingKVCachewith no server-side reach into it.) - RoPE/YaRN overrides are applied before cache allocation and are rejected for unsupported architectures. (Applied inside model construction, which precedes
make_caches; an override that reaches no RoPE seam refuses to serve, verified by an applications counter rather than a hand-maintained architecture list.) - No arithmetic-affecting option is accepted without a behavioral or numerical test. (Teacher-forced logit traces over 8192 positions, plus two per-process integration tests and 24 unit tests.)
- Warmup enablement, disablement, latency boundary, and startup-failure propagation are covered with a real checkpoint. (Out of this shard:
--warmupis owned bymodel-source.json/ fix(server): translate model-source flags and reject unsupported GGUF semantics #1434, so this chain cannot edit its entry.)
Numerical validation
Use the teacher-forced logit-trace procedure from docs/benchmarks.md at the configured prompt/decode widths and report disagreement at decided positions for every new scaling or batching path.
Activity
fix(compat): forbid a supported b10621 entry from carrying a divergence
test(server): add pinned b10621 compatibility manifest and gates
inureyes commentedon Aug 27, 2026
Reopening because the work this issue contracted is not finished, and leaving it closed breaks the
llama-compat manifestgate for every pull request in the repository.PR #1464 merged and closed this issue, but thirteen of the entries it owns in
compat/llama-server/b10621/runtime-and-context.jsonare stillstate: deferredwithissue: 1450:--batch-size,--context-shift,--keep,--parallel,--swa-full,--ubatch-size,--yarn-attn-factor,--yarn-beta-fast,--yarn-beta-slow,--yarn-ext-factor,--yarn-orig-ctx,field:n_discardandfield:n_keep. Only--ctx-size,--predictand the four RoPE entries were flipped.scripts/ci/check_llama_compat_manifest.py --check-issues-openfails on exactly that state witherror: deferred issue #1450 is CLOSED; flip its manifest entries or reopen the issue, and the CI job runs it on every PR, so #1463 and #1465 are both blocked by it as well.Reopening is the second of the two remedies the gate names, and is the reversible one. Close this again once the remaining thirteen entries are flipped out of
deferred, or once they are repointed at whichever issue actually owns the rest of the work.Filed from chain A of epic #1431 while unblocking #1465; no file in
runtime-and-context.jsonwas touched.inureyes commentedon Aug 27, 2026
Reopening to unblock the repository gate, and because the manifest says this issue is not finished.
The
llama-compat manifestjob requires everydeferredentry's implementation issue to be open. Closing this issue while entries undercompat/llama-server/b10621/runtime-and-context.jsonstill point at it turned that job red for every open pull request in the repository, not only for this issue's own PR. That is the failure the gate's own error message names, and reopening is one of the two remedies it offers.The reopening is not bookkeeping. The entries that remain
deferredunder this issue are in its stated scope and are not yet delivered; each now carries a structureddivergencedescribing exactly what still differs from llama-server b10621. The work landed so far is real and merged, so the label moves tostatus:in-progressrather than back tostatus:ready.Two ways to close this honestly, to be decided when the epic reaches its consolidation pass: finish the remaining entries here, or split them into a follow-up issue, repoint those entries at it, and add that issue to the shard's owner set in
pin.json.feat(server): explicitly reject b10621 control-vector options (#1489)
feat(server): close the b10621 context, batching and YaRN runtime gaps (
feat(server): apply b10621 RoPE runtime overrides before model constr…
fix(compat): reconcile manifest ownership and fix trace perplexity