perf(llm): wire response/semantic cache + tiered router + prompt caching into LLMService.chat()

Task 1 — Wire LLM cost machinery into the hot path

Theme: LLM token usage / cost. Highest ROI in the audit — the infrastructure is already built; production LLMService.chat() just bypasses it.

Problem

services/llm_service.py initializes a response/semantic cache (:136) and a TieredModelRouter (:141) but chat() (:250) calls provider.chat_completion() directly — neither cache nor router is consulted. The cache is only used by chat_optimized() (:688), whose only callers are the vLLM path (effectively dead). Anthropic prompt caching (llm_shared/providers/anthropic.py:203) fires only on escalation (complexity_router.py:175), so every direct Claude call forfeits the ~90% input-token discount on large static system prompts. And _TASK_TYPE_DEFAULTS (:12-20) maps max_tokens/temperature per llm_type but never maps a cheaper model.

Subtasks

Verification

Unit tests for cache hit/miss and router tier selection
Manual: identical repeated chat() returns cached (log shows hit); trivial query routes to cheap tier; Anthropic call shows cache_read tokens > 0 on 2nd call

Notes

Do not delete chat_optimized() — converge its logic into chat() (no _v2 suffixes). Reuse existing get_llm_cache() / SemanticLLMCache.

Part of umbrella #10603 (AutoBot Efficiency & Precision).

Deep-dive design — wiring cost machinery into `LLMService.chat()`

Grounded in a read of autobot-backend/services/llm_service.py (chat() at :162, init at :130, optimized path at :688). The infra is all present; chat() simply never calls it.

Current state (verified)

__init__ builds self._response_cache = get_llm_cache() (:136) and self._tier_router = TieredModelRouter(TierConfig.from_config()) (:141).
chat() (:162→:325) builds an LLMRequest (:202), then loops the fallback chain calling provider.chat_completion(request) (:250). It never reads self._response_cache and never calls self._tier_router.
The only cache users are _optimized_check_cache (:973) / _optimized_store_cache (:1012) — wrappers around self._response_cache.get(key) / .set(key, CachedResponse) — invoked only by chat_optimized() (:696/:734), whose only path is vLLM (effectively dead; real callers use chat()).
Cache API confirmed: get_llm_cache() → async get(cache_key) -> CachedResponse | None, async set(cache_key, CachedResponse) (llm_shared/cache.py:203,272).
Router API: strategy route(...) exists per strategy (complexity_router.py:42), plus registry.route (llm_shared/tiered_routing/registry.py:91) and route_with_escalation (complexity_router.py:121).

Design — 4 surgical insertions, all reusing existing code

1.1 Cache lookup/store in chat() — reuse the existing _optimized_check_cache/_optimized_store_cache helpers (generalize them to take a provider name instead of hardcoded "vllm"):

Insert a cache check after _apply_task_defaults (:200) and before the fallback loop (:219), only when not streaming and the caller hasn't disabled caching:

cache_key = None
if self._response_cache and not kwargs.get("no_cache"):
    cached, cache_key = await self._optimized_check_cache(
        messages, current_model or "", provider_name or "auto", request.request_id, start_time)
    if cached is not None:
        self._calculate_cache_hit_rate(cached)
        return cached

On the success branch (after :253, before return response), store:

if cache_key and not response.error and response.content:
    await self._optimized_store_cache(cache_key, response, request.request_id)

Refactor note: _optimized_check_cache (:973) currently bakes in "vllm"; parameterize the provider so both chat() and chat_optimized() share it. No new cache class — converge, don't duplicate.

1.2 Tiered routing — when the caller does not pin model_name/provider_name, ask the router for a tier before the loop:

if self._tier_router and model_name is None and provider_name is None:
    decision = self._tier_router.route(messages, llm_type=resolved_type)  # confirm exact signature
    current_model = decision.model or current_model
    current_provider_name = decision.provider or current_provider_name

Explicit caller overrides must always win (guard on is None).
Emit a llm.tier span attribute + metric for observability.

1.3 Anthropic prompt caching by default — llm_shared/providers/anthropic.py:203 only sets cache_control: ephemeral when metadata["enable_prompt_cache"]. Change to default-on for the system block behind a config flag (LLM_PROMPT_CACHE_DEFAULT=true); verify via usage.cache_read_input_tokens > 0 on the 2nd identical call. Add a second breakpoint on large static tool/context blocks.

1.4 llm_type → model mapping — extend _TASK_TYPE_DEFAULTS (:71) OR add a parallel _TASK_TYPE_MODEL map keyed by LLMType, resolving cheap tasks (CLASSIFICATION/EXTRACTION/ROUTING/SUMMARIZATION) to the cheap models already in autobot_shared/ssot_config.py:71-82 (gemma2:2b / phi3:mini / mistral:7b). Applied in chat() only when model_name is None and routing (1.2) didn't pin one.

Ordering & risk

1.1 and 1.4 are independent and lowest-risk → land first.
1.2 depends on confirming the exact TieredModelRouter.route() return shape (Decision object vs tuple) — read base_strategy.py:29 + registry.py:91 before coding.
1.3 is provider-local and isolated.
Each behind a config flag, default conservative; A/B token counts on a fixed workload as the acceptance gate.

Why this is safe

No new classes, no _v2. We connect chat() to the same _response_cache/_tier_router the constructor already builds and the optimized path already proves. Streaming (stream(), :327) is deliberately left out of caching (correct — can't cache a stream cheaply); routing could still apply to stream() in a follow-up.

perf(llm): wire response cache + default prompt caching into LLMServi…

PR #10619 (merged to Dev_new_gui) references this issue with a close keyword.

perf(llm): wire response cache + default prompt caching into LLMService.chat() (#10597)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

✅ Closed — implementation + integration verified via PR #10619 (squash 40311b3).

Implemented (active):

Response cache wired into LLMService.chat() — chat() ← _is_cacheable() + _optimized_check_cache()/_optimized_store_cache() (gated to safely-reusable: low-temp, no tools/structured/thinking; max_tokens in key).
Anthropic prompt caching default-on via config.llm_prompt_cache_default (llm_shared/providers/anthropic.py).

Wired, gated OFF pending benchmark → #10621:

Tiered routing + llm_type→model map: chat() ← _select_chat_model(), gated by config.chat_tiered_routing.

Verification: 9 unit tests (tests/services/test_llm_service_caching.py); flake8/black clean; code-review (2 HIGH issues fixed). New methods all have callers (no orphan modules).

Part of umbrella #10603.

perf(knowledge): set llm_type on cognifier LLM calls + fix entity pro…

Subtask 1.4's step line is corrected in the body above, with the original wording kept inline rather than deleted.

It named four task types — CLASSIFICATION, EXTRACTION, ROUTING, SUMMARIZATION. LLMType has eight members and neither ROUTING nor SUMMARIZATION is among them (autobot-backend/llm_shared/types.py: ORCHESTRATOR, TASK, CHAT, RAG, ANALYSIS, CLASSIFICATION, EXTRACTION, GENERAL). The merged map at services/llm_service.py:1086-1089 has two entries and is complete against the real type set.

This matters because the criterion, not the code, was wrong: anyone auditing delivery against the original wording finds two of four and reports a 50% shortfall that does not exist. That nearly happened while reviewing the Task 1 tick on #10603 — the verdict there is that the tick is correct (comment 5962686192).

Only the criterion wording changed. No checkbox, state or scope was touched.

Navigation Menu

Repository navigation

perf(llm): wire response/semantic cache + tiered router + prompt caching into LLMService.chat() #10597

Task 1 — Wire LLM cost machinery into the hot path

Problem

Subtasks

Verification

Notes

Deep-dive design — wiring cost machinery into `LLMService.chat()`

Current state (verified)

Design — 4 surgical insertions, all reusing existing code

Ordering & risk

Why this is safe

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

perf(llm): wire response/semantic cache + tiered router + prompt caching into LLMService.chat() #10597

Description

Task 1 — Wire LLM cost machinery into the hot path

Problem

Subtasks

Verification

Notes

Activity

mrveiss commented on Jun 28, 2026

mrveiss commented on Jun 28, 2026

Deep-dive design — wiring cost machinery into LLMService.chat()

Current state (verified)

Design — 4 surgical insertions, all reusing existing code

Ordering & risk

Why this is safe

github-actions commented on Jun 28, 2026

mrveiss commented on Jun 28, 2026

mrveiss commented on Oct 2, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Deep-dive design — wiring cost machinery into `LLMService.chat()`