Skip to content

Commit d6a3072

Browse files
committed
fix(gallery): raise arch-router scoring output cap via parallel:64
Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code]
1 parent 8290c3f commit d6a3072

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

gallery/index.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23985,6 +23985,14 @@
2398523985
# concurrent generation traffic; see model_config.go validation).
2398623986
known_usecases:
2398723987
- score
23988+
# Scoring decodes the whole prompt+candidate in one llama_decode and reads
23989+
# a logit row per candidate token. The llama.cpp server caps the causal
23990+
# output rows at n_parallel, so the default (1) aborts with
23991+
# GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max). Raise it to cover
23992+
# multi-token route labels; kv_unified (the grpc-server default) keeps the
23993+
# full context per sequence, so this does not split the KV cache.
23994+
options:
23995+
- parallel:64
2398823996
parameters:
2398923997
model: Arch-Router-1.5B.Q4_K_M.gguf
2399023998
files:
@@ -23996,6 +24004,9 @@
2399624004
overrides:
2399724005
known_usecases:
2399824006
- score
24007+
# See the q4 entry: lift the scoring output-row cap above the default 1.
24008+
options:
24009+
- parallel:64
2399924010
parameters:
2400024011
model: Arch-Router-1.5B.Q8_0.gguf
2400124012
files:

0 commit comments

Comments
 (0)