Skip to content

update: set prompt-lookup defaults from the GB10 measurement - #2265

Merged
inureyes merged 1 commit into
mainfrom
update/issue-2255-measured-defaults
Oct 10, 2026
Merged

inureyes merged 1 commit into
mainfrom
update/issue-2255-measured-defaults

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

End-of-run measurement for #2229 (PR #2256) and #2255 (PR #2262) on GB10, and the two values #2255 left provisional, now set from it. Report: docs/benchmark_results/server-prompt-lookup-gb10-2026-10-11.md; drivers, raw output and analysis script under docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/.

  • PROMPT_LOOKUP_PAGED_CONTEXT_LIMIT: None to Some(4096). A paged sequence stops running verify rounds past 4096 tokens of context and decodes plainly.
  • DEFAULT_PROMPT_LOOKUP_MAX_BATCH: stays 2; the doc comment and the --prompt-lookup-max-batch help now cite the measurement instead of calling it provisional.
  • docs/CONTINUOUS_BATCHING.md and the CHANGELOG entry state the measured gains and the cost on low-copy chat, and point to perf: stop server prompt-lookup rounds that cost more than they emit #2264.

Results

#2229, penalty requests on the CLI path (--repetition-penalty 1.1, 5 rounds with a null arm): against main before #2229 the engine client now decodes 23 % (Qwen3-1.7B, 256 tokens), 16 % (Qwen3, 8192), 31 % (Llama-3.2-1B, 256) and 23 % (Llama, 8192) faster, every round above every round. Against the pre-epic CxxGenerator (4c44e317) it is equal on Qwen3 at 256 tokens (inside the null range) and 1.5 to 2.7 % faster in the other three cells.

#2255, server prompt lookup (Qwen3-1.7B, 256 tokens):

Criterion Result
(a) plain chat, B=1, flag on vs off, no worse than -1.0 % Not met. -3.49 % (-3.73..-3.22) over 8 chat-template questions x 5 rounds, 25 to 45 % of proposed tokens accepted. Tracked in #2264.
(b) copy-heavy, B=1 +53 to +67 %, 4.2 tokens per verify
(c) concurrency 4 and 8, default gate on matches off in every workload; a gate of 8 loses 13 % on the chat-template copy workload at concurrency 4, so the default stays 2
(d) paged, long prompts on / off ratio for copy and open question: 2.69x / 1.27x at 1024, 2.79x / 1.64x at 2048, 1.42x / 1.12x at 4096, 1.15x / 0.62x at 8192; dense at 8192: 2.10x / 0.99x

The flag is opt-in, so decode without it is unchanged.

Test plan

  • clippy --profile test-fast --features cuda --lib --tests --bins -D warnings
  • --lib server::batch (526), server::speculative_dispatch (17), --test cli_help_consistency (27), --test dead_doc_pointers, Apache header check, all under gpu-lock with --test-threads=1
  • scripts/ci/check_binary_assets.py fails on main's two undeclared tarballs from test(rocm): verify the unified engine on gfx1151 after the engine epic #2250 (docs/benchmark_results/data/rocm-unified-engine-gfx1151-2026-10-08/); this PR adds no binary file

Refs #2229, #2255, #2264

Record the end-of-run measurement for #2229 and #2255 and settle the two values #2255 left provisional. PROMPT_LOOKUP_PAGED_CONTEXT_LIMIT becomes 4096: on paged storage the flag still gains at 4096 tokens (1.42x copy, 1.12x open question) and loses at 8192 (0.62x on the open question). DEFAULT_PROMPT_LOOKUP_MAX_BATCH stays 2: a gate of 8 loses 13 percent on the chat-template copy workload at concurrency 4.

The measurement also shows the flag costs 3.5 percent on chat replies that copy little, which misses #2255's -1.0 percent criterion; the docs and changelog say so, and #2264 tracks gating those rounds off.
@inureyes inureyes added type:performance Performance improvements area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:medium Medium priority status:review Under review status:done Completed and removed status:review Under review labels Oct 10, 2026
@inureyes
inureyes merged commit 8b263d6 into main Oct 10, 2026
25 of 27 checks passed
@inureyes
inureyes deleted the update/issue-2255-measured-defaults branch October 10, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:medium Medium priority status:done Completed type:performance Performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant