Repository navigation
fix(llm): speculative decoding is an operator-facing config surface with no implementation — the API accepts it, nothing consumes it #14214
Description
Summary
Speculative decoding is a fully-plumbed, operator-reachable configuration surface with no
implementation behind it. An operator can enable it, name a draft model, set token count and
ngram mode, have the admin API accept and echo the values back — and nothing changes at
inference time. MEDUSA_HEADS and CUDA_GRAPHS are in the same state.
This needs an owner decision because the two correct fixes are opposite (see below), so it is
filed with both options rather than held back.
Evidence
Built and reachable:
| Layer | Location |
|---|---|
| Config (4 env-backed settings) | autobot_shared/ssot_config.py:1808-1811 — AUTOBOT_SPECULATION_ENABLED, _DRAFT_MODEL, _NUM_TOKENS, _USE_NGRAM |
| Defaults | autobot-backend/config/defaults.py:186-192 |
| Admin API — read | autobot-backend/api/llm_optimization.py:507-510 |
| Admin API — write/echo | autobot-backend/api/llm_optimization.py:603-606; schema autobot-backend/api/schemas_agent.py |
| API docstring claims control | autobot-backend/api/llm_optimization.py:534 — "Local optimizations (speculative decoding, quantization, vLLM)" |
| Optimization category + routing | autobot-backend/llm_shared/optimization/router.py:31,57-60,93,197 — routed as LOCAL_ONLY |
| Delegation comment | autobot-backend/llm_shared/optimization/integration.py:256-259 — "Local optimizations (speculative, FlashInfer, etc.) are handled directly by vLLM/Ollama providers" |
That last comment is false. The providers it names contain no speculation handling:
rg -i "speculat|draft_model|ngram" \
autobot-backend/llm_shared/providers/vllm.py \
autobot-backend/llm_shared/providers/vllm_base.py \
autobot-backend/llm_shared/providers/ollama.py \
autobot-backend/llm_shared/providers/ollama_provider.py
# -> no matchesAnd no consumer of the settings exists outside tests:
rg -n "speculation_enabled|speculation_draft_model|speculation_num_tokens|speculation_use_ngram" \
-g '*.py' | grep -v ssot_config.py | grep -v _test.py | grep -v secret_redaction
# -> only config/defaults.py, api/llm_optimization.py, optimization/router.py — all surface, no sinkSame for the sibling categories: MEDUSA_HEADS (router.py:34) and CUDA_GRAPHS
(router.py:33) have no references anywhere except llm_shared/optimization_test.py.
Impact
An operator tuning inference has a control that reports success and does nothing. There is no
error, no warning, and no metric that would reveal it — the setting round-trips through the API
cleanly. This is a promise-without-delivery surface, not merely a missing feature.
Decision needed — two opposite fixes
Option A — wire it. Pass the config into vLLM engine args (speculative_model,
num_speculative_tokens, ngram variant) in llm_shared/providers/vllm_base.py, and correct the
integration.py:256-259 comment to describe what actually happens. Also decide whether
MEDUSA_HEADS / CUDA_GRAPHS get wired or dropped in the same pass.
Option B — remove the promise. Delete the speculation fields from the admin API
(api/llm_optimization.py:507-510,603-606, api/schemas_agent.py), drop the unused
OptimizationCategory members, and fix the false comment. Keep the config keys only if
something is expected to consume them later, and document that.
Option A is the larger change and only pays off if local vLLM inference is a path we intend to
tune. Option B is small and makes the surface honest immediately. Either is acceptable; shipping
neither is not.
Acceptance criteria
Whichever option is chosen:
autobot-backend/llm_shared/optimization/integration.py:256-259no longer asserts a
delegation that does not happen.- No
OptimizationCategorymember remains whose only references are tests. - A guard test asserts that every operator-settable optimization flag exposed by
api/llm_optimization.pyhas at least one production consumer — so this class of surface
cannot regrow silently. - If Option A: evidence of speculation active at inference (engine args logged, or a
throughput delta measured with and without a draft model). If Option B: the admin API no
longer accepts the removed fields.
Activity
mrveiss commentedon Oct 3, 2026
Recommendation: B — remove the promise, keep the config keys, document why. The issue frames both options and ends with "either is acceptable"; this supplies the missing recommendation, with the one fact that bears on it checked rather than assumed.
The fact I checked, because it is the one that could have argued for A
vLLM is not hypothetical here. It is a registered provider (
llm_shared/provider_registry.py:357,369→VLLMBaseProvider) and it is deployed (autobot-slm-backend/ansible/roles/backend/, plusautobot-infrastructure/shared/config/config.yaml.template). So "local vLLM inference is a path we intend to tune" is plausible, not fanciful, and that is the premise Option A needs.It still does not make A the right move now, for a reason the evidence makes sharp.
Why B, despite vLLM being real
The defect is not a missing feature. It is a false affirmation. An operator sets
AUTOBOT_SPECULATION_ENABLED, the admin API accepts it, echoes it back, and the docstring atllm_optimization.py:534tells them it controls "speculative decoding, quantization, vLLM". Nothing happens at inference time. That is worse than the surface not existing: an absent feature prompts someone to ask for it, while a lying one closes the question. Anyone diagnosing latency will see speculation enabled and rule it out as already-tuned.B removes the lie today at small cost. A is a performance project. Wiring
speculative_model/num_speculative_tokensinto vLLM engine args needs draft-model selection, a measurement harness to show it helps, and a decision onMEDUSA_HEADSandCUDA_GRAPHSin the same pass. That work should be justified on its own terms — "we want faster local inference and here is the measurement" — not undertaken because a stale admin surface needs to become true. Doing A to retire a false comment is the tail wagging the dog.And B does not foreclose A. Keeping the four
ssot_config.py:1808-1811keys, with a comment saying they are retained for a future wiring and currently consumed by nothing, preserves every option while removing the operator-facing claim. The issue's own option B allows exactly this.What I would add to the acceptance criteria
Criterion 3 asks for a guard that every operator-settable optimization flag is consumed. Make it assert the honest invariant rather than the aspirational one: a flag exposed through the admin API must have a non-test consumer, or be listed in an explicit, documented "accepted but inert" table with its reason. Otherwise the guard forces A by construction — the only way to pass it is to wire everything — and a guard that permits only one of two acceptable outcomes is a decision disguised as a check.
That also makes the
MEDUSA_HEADS/CUDA_GRAPHSquestion answerable in the same pass without expanding scope: they go in the same table.Framing and recommendation only; the call is the owner's, and A remains defensible if local-inference tuning is on the near roadmap.