Skip to content

update: set prompt-lookup defaults from the GB10 measurement #2265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 1 commit into from
Oct 10, 2026
Merged

update: set prompt-lookup defaults from the GB10 measurement #2265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Added

- `mlxcel serve` and `mlxcel-server` accept `--draft-kind prompt-lookup` with no drafter checkpoint: each eligible request runs prompt-lookup verify rounds inside the batch scheduler, at most one per tick, while proposal-free ticks stay on the lookahead pipeline (#2255). `--draft-block-size` sets the largest proposal and the new `--prompt-lookup-max-batch` (default 2) is the largest active decode batch at which verify rounds run. Finished requests report the rounds in `timings` with `draft_kind: "prompt-lookup"`. `scripts/bench_serving_concurrency.py` gains `--prompt-style plain|copy`. `mlxcel generate --prompt-lookup` runs the same `Engine::verify_round`.
- `mlxcel serve` and `mlxcel-server` accept `--draft-kind prompt-lookup` with no drafter checkpoint: each eligible request runs prompt-lookup verify rounds inside the batch scheduler, at most one per tick, while proposal-free ticks stay on the lookahead pipeline (#2255). `--draft-block-size` sets the largest proposal and the new `--prompt-lookup-max-batch` (default 2) is the largest active decode batch at which verify rounds run; a paged sequence stops verifying past 4096 tokens of context. On GB10 the flag gains 53 to 67 % at B=1 on replies that copy the prompt and costs 3.5 % on chat replies that copy little (#2264 tracks gating those rounds off). Finished requests report the rounds in `timings` with `draft_kind: "prompt-lookup"`. `scripts/bench_serving_concurrency.py` gains `--prompt-style plain|copy`. `mlxcel generate --prompt-lookup` runs the same `Engine::verify_round`.

### Deprecated

Expand Down
17 changes: 14 additions & 3 deletions docs/CONTINUOUS_BATCHING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -665,9 +665,20 @@ proposals.

Each proposing row costs one extra forward per tick, so verify rounds run only
while the active decode batch has at most `--prompt-lookup-max-batch` rows
(default 2, provisional until the concurrency measurement of #2255 sets it).
Above that every row decodes plainly, its drafter records no miss, and it
resumes when the batch shrinks. Verify rounds also stay off under a unified KV
(default 2; at a gate of 8 one chat-template workload lost 13 % at concurrency
4 on GB10). Above that every row decodes plainly, its drafter records no miss,
and it resumes when the batch shrinks. On paged storage a sequence stops
verifying once its context passes 4096 tokens, where the gathered verify
forward starts to cost more than it saves (0.62x at 8192 tokens on a 59 %
acceptance prompt); dense storage has no such limit.

The flag gains on output that copies its prompt (+53 to +67 % at B=1 on GB10
for a repeat-the-paragraph prompt, up to 2.8x with a long document in front)
but costs 3.5 % on chat whose reply copies little (25 to 45 % of proposed
tokens accepted), because each proposal also ends the lookahead for a tick.
Turn it on for workloads that quote or edit their input. Measurements:
[`docs/benchmark_results/server-prompt-lookup-gb10-2026-10-11.md`](benchmark_results/server-prompt-lookup-gb10-2026-10-11.md);
server-side gating for low-acceptance replies is #2264. Verify rounds also stay off under a unified KV
budget (`--kv-unified`). Requests with images, audio or video, structured
output, per-token logprobs, mirostat, adaptive-p or the extended sampler chain
(dynamic temperature, `min_keep`), and models whose state a cache trim cannot
Expand Down
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-off-r1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5275, "decode_tok_s": 166.94, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 1, "completion_tokens": 256, "decode_s": 1.529, "decode_tok_s": 166.77, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 2, "completion_tokens": 256, "decode_s": 1.5297, "decode_tok_s": 166.69, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 3, "completion_tokens": 256, "decode_s": 1.5075, "decode_tok_s": 169.16, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 4, "completion_tokens": 256, "decode_s": 1.5737, "decode_tok_s": 162.04, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6152, "decode_tok_s": 157.87, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 6, "completion_tokens": 256, "decode_s": 1.623, "decode_tok_s": 157.11, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6148, "decode_tok_s": 157.92, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-off-r2.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5692, "decode_tok_s": 162.5, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 1, "completion_tokens": 256, "decode_s": 1.6108, "decode_tok_s": 158.31, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 2, "completion_tokens": 256, "decode_s": 1.6059, "decode_tok_s": 158.79, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 3, "completion_tokens": 256, "decode_s": 1.546, "decode_tok_s": 164.94, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 4, "completion_tokens": 256, "decode_s": 1.5787, "decode_tok_s": 161.52, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6129, "decode_tok_s": 158.1, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 6, "completion_tokens": 256, "decode_s": 1.628, "decode_tok_s": 156.63, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 7, "completion_tokens": 256, "decode_s": 1.5853, "decode_tok_s": 160.85, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-off-r3.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5004, "decode_tok_s": 169.96, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 1, "completion_tokens": 256, "decode_s": 1.4855, "decode_tok_s": 171.66, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 2, "completion_tokens": 256, "decode_s": 1.4942, "decode_tok_s": 170.66, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 3, "completion_tokens": 256, "decode_s": 1.4921, "decode_tok_s": 170.9, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 4, "completion_tokens": 256, "decode_s": 1.5574, "decode_tok_s": 163.74, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 5, "completion_tokens": 256, "decode_s": 1.5026, "decode_tok_s": 169.71, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 6, "completion_tokens": 256, "decode_s": 1.4903, "decode_tok_s": 171.11, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 7, "completion_tokens": 256, "decode_s": 1.4829, "decode_tok_s": 171.96, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-off-r4.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5258, "decode_tok_s": 167.13, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 1, "completion_tokens": 256, "decode_s": 1.5248, "decode_tok_s": 167.23, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 2, "completion_tokens": 256, "decode_s": 1.5776, "decode_tok_s": 161.64, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 3, "completion_tokens": 256, "decode_s": 1.6056, "decode_tok_s": 158.82, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 4, "completion_tokens": 256, "decode_s": 1.6706, "decode_tok_s": 152.64, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 5, "completion_tokens": 256, "decode_s": 1.7005, "decode_tok_s": 149.95, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 6, "completion_tokens": 256, "decode_s": 1.6312, "decode_tok_s": 156.32, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 7, "completion_tokens": 256, "decode_s": 1.5556, "decode_tok_s": 163.92, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-off-r5.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5394, "decode_tok_s": 165.65, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 1, "completion_tokens": 256, "decode_s": 1.4982, "decode_tok_s": 170.2, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 2, "completion_tokens": 256, "decode_s": 1.538, "decode_tok_s": 165.8, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 3, "completion_tokens": 256, "decode_s": 1.6132, "decode_tok_s": 158.07, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 4, "completion_tokens": 256, "decode_s": 1.6159, "decode_tok_s": 157.8, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 5, "completion_tokens": 256, "decode_s": 1.61, "decode_tok_s": 158.39, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 6, "completion_tokens": 256, "decode_s": 1.5651, "decode_tok_s": 162.93, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6426, "decode_tok_s": 155.24, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-on-r1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.6266, "decode_tok_s": 156.77, "draft_n": 112, "draft_n_accepted": 40, "draft_rounds": 26}
{"q": 1, "completion_tokens": 256, "decode_s": 1.5516, "decode_tok_s": 164.34, "draft_n": 113, "draft_n_accepted": 51, "draft_rounds": 32}
{"q": 2, "completion_tokens": 256, "decode_s": 1.5729, "decode_tok_s": 162.13, "draft_n": 26, "draft_n_accepted": 7, "draft_rounds": 8}
{"q": 3, "completion_tokens": 256, "decode_s": 1.5401, "decode_tok_s": 165.57, "draft_n": 98, "draft_n_accepted": 41, "draft_rounds": 20}
{"q": 4, "completion_tokens": 256, "decode_s": 1.6162, "decode_tok_s": 157.78, "draft_n": 39, "draft_n_accepted": 10, "draft_rounds": 12}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6848, "decode_tok_s": 151.35, "draft_n": 87, "draft_n_accepted": 25, "draft_rounds": 26}
{"q": 6, "completion_tokens": 256, "decode_s": 1.7004, "decode_tok_s": 149.97, "draft_n": 48, "draft_n_accepted": 12, "draft_rounds": 19}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6887, "decode_tok_s": 151.0, "draft_n": 30, "draft_n_accepted": 7, "draft_rounds": 10}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-on-r2.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.6084, "decode_tok_s": 158.54, "draft_n": 112, "draft_n_accepted": 40, "draft_rounds": 26}
{"q": 1, "completion_tokens": 256, "decode_s": 1.5398, "decode_tok_s": 165.61, "draft_n": 113, "draft_n_accepted": 51, "draft_rounds": 32}
{"q": 2, "completion_tokens": 256, "decode_s": 1.6317, "decode_tok_s": 156.28, "draft_n": 26, "draft_n_accepted": 7, "draft_rounds": 8}
{"q": 3, "completion_tokens": 256, "decode_s": 1.6023, "decode_tok_s": 159.15, "draft_n": 98, "draft_n_accepted": 41, "draft_rounds": 20}
{"q": 4, "completion_tokens": 256, "decode_s": 1.6535, "decode_tok_s": 154.22, "draft_n": 39, "draft_n_accepted": 10, "draft_rounds": 12}
{"q": 5, "completion_tokens": 256, "decode_s": 1.7123, "decode_tok_s": 148.92, "draft_n": 87, "draft_n_accepted": 25, "draft_rounds": 26}
{"q": 6, "completion_tokens": 256, "decode_s": 1.7558, "decode_tok_s": 145.23, "draft_n": 48, "draft_n_accepted": 12, "draft_rounds": 19}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6875, "decode_tok_s": 151.11, "draft_n": 30, "draft_n_accepted": 7, "draft_rounds": 10}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-on-r3.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.5409, "decode_tok_s": 165.49, "draft_n": 112, "draft_n_accepted": 40, "draft_rounds": 26}
{"q": 1, "completion_tokens": 256, "decode_s": 1.4684, "decode_tok_s": 173.65, "draft_n": 113, "draft_n_accepted": 51, "draft_rounds": 32}
{"q": 2, "completion_tokens": 256, "decode_s": 1.5031, "decode_tok_s": 169.65, "draft_n": 26, "draft_n_accepted": 7, "draft_rounds": 8}
{"q": 3, "completion_tokens": 256, "decode_s": 1.4946, "decode_tok_s": 170.61, "draft_n": 98, "draft_n_accepted": 41, "draft_rounds": 20}
{"q": 4, "completion_tokens": 256, "decode_s": 1.5369, "decode_tok_s": 165.91, "draft_n": 39, "draft_n_accepted": 10, "draft_rounds": 12}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6357, "decode_tok_s": 155.9, "draft_n": 87, "draft_n_accepted": 25, "draft_rounds": 26}
{"q": 6, "completion_tokens": 256, "decode_s": 1.628, "decode_tok_s": 156.64, "draft_n": 48, "draft_n_accepted": 12, "draft_rounds": 19}
{"q": 7, "completion_tokens": 256, "decode_s": 1.5973, "decode_tok_s": 159.64, "draft_n": 30, "draft_n_accepted": 7, "draft_rounds": 10}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-on-r4.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.6633, "decode_tok_s": 153.31, "draft_n": 112, "draft_n_accepted": 40, "draft_rounds": 26}
{"q": 1, "completion_tokens": 256, "decode_s": 1.6397, "decode_tok_s": 155.51, "draft_n": 113, "draft_n_accepted": 51, "draft_rounds": 32}
{"q": 2, "completion_tokens": 256, "decode_s": 1.6473, "decode_tok_s": 154.8, "draft_n": 26, "draft_n_accepted": 7, "draft_rounds": 8}
{"q": 3, "completion_tokens": 256, "decode_s": 1.6147, "decode_tok_s": 157.92, "draft_n": 98, "draft_n_accepted": 41, "draft_rounds": 20}
{"q": 4, "completion_tokens": 256, "decode_s": 1.7009, "decode_tok_s": 149.92, "draft_n": 39, "draft_n_accepted": 10, "draft_rounds": 12}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6781, "decode_tok_s": 151.96, "draft_n": 87, "draft_n_accepted": 25, "draft_rounds": 26}
{"q": 6, "completion_tokens": 256, "decode_s": 1.6752, "decode_tok_s": 152.22, "draft_n": 48, "draft_n_accepted": 12, "draft_rounds": 19}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6684, "decode_tok_s": 152.85, "draft_n": 30, "draft_n_accepted": 7, "draft_rounds": 10}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/a2-on-r5.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"q": 0, "completion_tokens": 256, "decode_s": 1.6518, "decode_tok_s": 154.38, "draft_n": 112, "draft_n_accepted": 40, "draft_rounds": 26}
{"q": 1, "completion_tokens": 256, "decode_s": 1.6409, "decode_tok_s": 155.4, "draft_n": 113, "draft_n_accepted": 51, "draft_rounds": 32}
{"q": 2, "completion_tokens": 256, "decode_s": 1.6338, "decode_tok_s": 156.08, "draft_n": 26, "draft_n_accepted": 7, "draft_rounds": 8}
{"q": 3, "completion_tokens": 256, "decode_s": 1.5802, "decode_tok_s": 161.37, "draft_n": 98, "draft_n_accepted": 41, "draft_rounds": 20}
{"q": 4, "completion_tokens": 256, "decode_s": 1.6015, "decode_tok_s": 159.23, "draft_n": 39, "draft_n_accepted": 10, "draft_rounds": 12}
{"q": 5, "completion_tokens": 256, "decode_s": 1.6639, "decode_tok_s": 153.26, "draft_n": 87, "draft_n_accepted": 25, "draft_rounds": 26}
{"q": 6, "completion_tokens": 256, "decode_s": 1.6599, "decode_tok_s": 153.63, "draft_n": 48, "draft_n_accepted": 12, "draft_rounds": 19}
{"q": 7, "completion_tokens": 256, "decode_s": 1.6469, "decode_tok_s": 154.84, "draft_n": 30, "draft_n_accepted": 7, "draft_rounds": 10}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/ab-off-r1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"style": "chat", "conc": 1, "rep": 0, "wall_s": 1.766, "tokens": 256, "agg_tok_s": 144.93, "decode_tok_s": [145.63], "prompt_n": [27], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 1, "wall_s": 1.551, "tokens": 256, "agg_tok_s": 165.1, "decode_tok_s": [168.32], "prompt_n": [19], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 2, "wall_s": 1.595, "tokens": 256, "agg_tok_s": 160.52, "decode_tok_s": [163.36], "prompt_n": [21], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 3, "wall_s": 1.582, "tokens": 256, "agg_tok_s": 161.86, "decode_tok_s": [165.16], "prompt_n": [20], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 0, "wall_s": 1.739, "tokens": 256, "agg_tok_s": 147.25, "decode_tok_s": [148.6], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 1, "wall_s": 1.721, "tokens": 256, "agg_tok_s": 148.77, "decode_tok_s": [150.0], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 2, "wall_s": 1.789, "tokens": 256, "agg_tok_s": 143.13, "decode_tok_s": [144.31], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 3, "wall_s": 1.778, "tokens": 256, "agg_tok_s": 143.96, "decode_tok_s": [145.13], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/ab-off-r2.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"style": "chat", "conc": 1, "rep": 0, "wall_s": 1.555, "tokens": 256, "agg_tok_s": 164.65, "decode_tok_s": [165.58], "prompt_n": [27], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 1, "wall_s": 1.534, "tokens": 256, "agg_tok_s": 166.93, "decode_tok_s": [170.11], "prompt_n": [19], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 2, "wall_s": 1.658, "tokens": 256, "agg_tok_s": 154.39, "decode_tok_s": [157.12], "prompt_n": [21], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 3, "wall_s": 1.814, "tokens": 256, "agg_tok_s": 141.12, "decode_tok_s": [143.42], "prompt_n": [20], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 0, "wall_s": 1.733, "tokens": 256, "agg_tok_s": 147.71, "decode_tok_s": [149.04], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 1, "wall_s": 1.616, "tokens": 256, "agg_tok_s": 158.41, "decode_tok_s": [159.77], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 2, "wall_s": 1.775, "tokens": 256, "agg_tok_s": 144.24, "decode_tok_s": [145.38], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 3, "wall_s": 1.675, "tokens": 256, "agg_tok_s": 152.81, "decode_tok_s": [154.17], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
8 changes: 8 additions & 0 deletions docs/benchmark_results/data/server-prompt-lookup-gb10-2026-10-11/ab-off-r3.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"style": "chat", "conc": 1, "rep": 0, "wall_s": 1.578, "tokens": 256, "agg_tok_s": 162.24, "decode_tok_s": [163.46], "prompt_n": [27], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 1, "wall_s": 1.567, "tokens": 256, "agg_tok_s": 163.4, "decode_tok_s": [166.56], "prompt_n": [19], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 2, "wall_s": 1.578, "tokens": 256, "agg_tok_s": 162.26, "decode_tok_s": [165.37], "prompt_n": [21], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "chat", "conc": 1, "rep": 3, "wall_s": 1.582, "tokens": 256, "agg_tok_s": 161.85, "decode_tok_s": [165.16], "prompt_n": [20], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 0, "wall_s": 1.699, "tokens": 256, "agg_tok_s": 150.72, "decode_tok_s": [152.15], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 1, "wall_s": 1.721, "tokens": 256, "agg_tok_s": 148.73, "decode_tok_s": [149.91], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 2, "wall_s": 1.672, "tokens": 256, "agg_tok_s": 153.08, "decode_tok_s": [154.36], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
{"style": "copy", "conc": 1, "rep": 3, "wall_s": 1.606, "tokens": 256, "agg_tok_s": 159.44, "decode_tok_s": [160.78], "prompt_n": [155], "draft_kind": null, "draft_n": 0, "draft_n_accepted": 0, "draft_rounds": 0}
Loading
Loading