update: offer prompt lookup on the server without slowing plain decode

[inureyes]

Summary

mlxcel serve / mlxcel-server --draft-kind prompt-lookup (no draft model) run prompt-lookup verify rounds per sequence inside the batch scheduler, at most one per tick, while ticks on which no row proposes stay on the lookahead pipeline exactly as without the flag.

• Dispatch: SpeculativeDispatch::PromptLookup { config, max_batch } resolves before the draft-model early return; a draft model with it is InvalidKind; --draft-block-size sets max_draft; --spec-type none drops it. is_kind_specific() stays false, so the burst path is never entered and the scheduler reads prompt_lookup_config(). New --prompt-lookup-max-batch on both server binaries, classified read-only in the runtime settings schema. mlxcel generate --prompt-lookup and its --draft-kind rejection are unchanged.
• Shared round: the verify round moved out of DirectEngine::speculate into Engine::verify_round (engine/verify_round.rs), with Engine::verify_rounds_unsupported as the one eligibility rule and Engine::warm_up_verify_widths as the one warmup; the CLI and the scheduler both run them. ADR 0009 amended.
• Scheduler (scheduler/prompt_lookup.rs): drafter on SequenceInfo, primed at prefill completion; proposing rows verify alone, the rest take the batched step. Switch rule: ask before the step n+1 prime; a proposal commits step n (through try_eval) with no prime, retracts the asks and makes the next tick synchronous; re-prime only when nobody proposed and every gated-in drafter pipelines_plain_rounds(). The module docs state invariants I1 to I8 (append ownership, admission, cancellation, errors), and each scheduler test pins one.
• Gate: a row verifies only while its governor grants a budget, the active batch is at most --prompt-lookup-max-batch, no unified KV budget is set, and the proposal fits under max_tokens and the context-bound stop. A gated-out row calls no draft_block, so its governor records no miss.
• Warmup: the first eligible prefill (normally the startup warmup request) runs one verify forward per block width and unwinds it, as the CLI does.
• Ineligible requests (multimodal, structured output, logprobs, feedback or extended sampler chains, state a trim cannot roll back such as Gemma 3's sliding-window cache) decode plainly with a debug log naming why.
• scripts/bench_serving_concurrency.py --prompt-style plain|copy.

Provisional values for the end-of-run measurement

• DEFAULT_PROMPT_LOOKUP_MAX_BATCH = 2 (server/speculative_dispatch.rs): one extra forward per proposing row at most doubles a step at B <= 2. Measurement (c) sets the final value.
• PROMPT_LOOKUP_PAGED_CONTEXT_LIMIT = None (scheduler/prompt_lookup.rs), no limit, pending measurement (d).

Changes during review

• Review (2cd467a): server-side verify-width warmup (early requests paid each width's first-use kernel cost mid-decode); the n-gram index is built at the first lookup, so rows the gate declines no longer pay it at prefill.
• Security (a64431e): proposals capped at the context bound (a prompt ending just under --max-kv-size could make a verify forward append up to 63 positions the stop then discarded); the same bound in the warmup; an unchecked slice in the ask path replaced, since a scheduler panic stops all serving.
• Finalizer (953b03b): scheduler tests for a stop token inside an accepted block and a cancel during a round.
• Local CUDA gate (9f5b5a8): prompt_lookup_max_batch classified in the runtime settings schema (server_config_schema_classifies_all_112_fields).

Test plan

All on GB10 under gpu-lock, --profile test-fast --features cuda, each CUDA test with -- --test-threads=1.

• New: speculative_dispatch_tests (5), engine::verify_round_tests (3), scheduler_prompt_lookup_tests (11, a cycle-model dense Llama, KV offset checked against committed tokens after every tick; proposal_free_ticks_stay_pipelined fails on main)
• fmt; clippy -D warnings (root and mlxcel-core); mlxcel-core engine::, speculative::, drafter::, session, sampling, generate::; root server::batch, server::speculative_dispatch, server::in_process, server::runtime_settings, engine_probe, backend::, cli_input, commands::, mlxcel-server bin; cli_help_consistency, dead_doc_pointers, model_loader_stdout_guard
• make verify-test-cuda: 13,868 passed; the only failures are the pre-existing SSM bf16 parity tests (chore: post-epic cleanup of compat paged kernels, the model-owned paged flag, and the CUDA SSM bf16 parity failures #2230) and the schema test fixed in 9f5b5a8
• mlxcel generate --prompt-lookup on Llama-3.2-1B, greedy, 160 tokens: identical to the build before this PR
• Server pairs, release build, MLXCEL_SDPA_DETERMINISTIC=1, /completion greedy, flag on vs off: Qwen3-1.7B at 140 tokens (auto, dense, paged) and Llama-3.2-1B at 160 tokens (auto, paged) identical for copy and plain prompts, concurrent copy + plain equal their solo outputs, draft_kind prompt-lookup with draft_n_accepted > 0; mlxcel serve identical; Gemma-3-1B identical with prompt lookup declined (non-trimmable sliding-window cache, as on the CLI). At 160 tokens the Qwen3 copy reply diverges once at about token 150 on an exact tie (top two logprobs both -1.625 on the flag-off server)
• mlxcel-engine-parity -n 400 on Qwen3-1.7B and Llama-3.2-1B: unchanged from update: pipeline penalty and DRY requests in the engine's B=1 client #2256
• Speed (a) to (d): measured after merge together with perf: pipeline penalty and DRY requests in the engine's B=1 client #2229; results and the final constants land in a follow-up docs PR

Closes #2255

[inureyes]

Implementation Review Summary

Intent

Offer prompt lookup on mlxcel serve / mlxcel-server without slowing plain decode: proposal-free ticks stay pipelined, and verify rounds are gated by the governor and --prompt-lookup-max-batch.

Findings Addressed

• The server never warmed the verify widths the CLI warms, so early requests paid first-use kernel costs mid-decode. The warmup loop is now Engine::warm_up_verify_widths; the CLI delegates to it, and the scheduler runs it once on the first eligible prefill, which the startup warmup request covers (MEDIUM, 2cd467a)
• Rows the max-batch gate keeps out still built the full n-gram index at prefill. The index is now built on the first lookup (LOW, 2cd467a)

Remaining Items

• Rule 4 (stay synchronous until every gated-in drafter has made three plain asks) is kept. It is in the issue, lookahead is per tick at any B, and dropping it trades three synchronous ticks for a commit-without-prime tick plus a delayed proposal. Measurements (a) and (c) should confirm it (INFO)
• Rule 3 also retracts the empty asks of non-proposing rows, so stats.rounds undercounts. This matches the CLI and is not reported to clients (LOW, cosmetic)
• Proposals are capped by max_tokens only, not by the context bound. Same as the CLI (LOW)
• No scheduler-level test for a verify-round eval failure, a cancel mid-round, or EOS inside an accepted block (LOW)

Verification

• All stated requirements implemented (the speed measurements remain with the orchestrator)
• No placeholder/mock code remaining (paged limit is an intentionally provisional None)
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass: mlxcel-core engine::, speculative::, drafter::; root server::batch, server::speculative_dispatch, commands::generate, cli_help_consistency; clippy and fmt. proposal_free_ticks_stay_pipelined fails when lookahead_params is forced back to main's behavior

[inureyes]

Security and performance review (fixes in a64431e).

Fixed:

• LOW: proposals were capped by max_tokens only. With --max-kv-size and context shift off, a self-copying prompt ending just under the bound made the verify forward append up to max_draft - 1 positions past the plain path's KV peak, all discarded when the bound finished the row. ContextBound::verify_room now caps the budget so the last forwarded position is at most the finishing token. The verify-width warmup skips a prefill without room for the widest block. Dense caches have no fixed capacity and the paged budget was already checked, so this was wasted compute and transient growth, not an overflow.
• LOW (hardening): ask_prompt_lookup sliced generated_tokens[observed..] unchecked. Not reachable today, but the scheduler has no catch_unwind, so a future break would stop serving. A drafter that runs ahead of its row is now dropped, and the row decodes plainly.

Open:

• MEDIUM: on paged storage, verify forwards for long copy-heavy prompts take the gather path. PROMPT_LOOKUP_PAGED_CONTEXT_LIMIT stays None until measurement (d) decides. It is bounded by the max-batch gate (default 2) and the governor.
• LOW: the lazily built n-gram index costs about 120 B per prompt token (about 15 MB at 128K) plus a one-time build on the scheduler thread. It uses SipHash keys and is small next to the KV for that context.

Verified, no change: every new device read goes through try_eval (#822), failed rounds unwind and fail only their own row, request-controlled edge values are safe (empty prompt rejected, length 1, max_tokens 0 or 1), the request speculative.* keys stay inert so clients cannot enable this per request, and drafters see only their own request's tokens.

server::batch 524 pass, clippy and fmt clean.