Repository navigation
Integration: LayerInferenceEngine not registered as LLM provider — no API access #3104
Description
Activity
- added 4 commits that reference this issue
on Apr 1, 2026 Reopening the record rather than the issue: this was closed without the fix landing.
The problem stated here —
LayerInferenceEnginenot registered as an LLM provider, no API access — is still present as ofDev_new_guitoday.LayerInferenceAdapterwas written (llm_shared/adapters/layer_inference_adapter.py:32) and exported (llm_shared/adapters/__init__.py:23), but_register_llm_adapters()ininitialization/lifespan.py:1274-1312still registers only Ollama, OpenAI, Anthropic and Groq.grep -rn "LayerInferenceAdapter"outside the definition and the export returns nothing, so it never reachesGET /api/adapters.Writing the adapter class satisfied step 1 of the fix; wiring it into startup did not happen, and the closure gate did not require evidence that the adapter appears in the API response.
Superseded by #13035, which carries this plus the duplicate meta-eviction implementation, and is sequenced behind the correctness and streaming fixes in umbrella #13030 — registering an engine that cannot currently produce correct output would turn dormant defects into live ones.
Leaving this closed and tracking the work on #13035 to avoid two issues for one fix.
Problem
LayerInferenceEngine (#1946) implements layer-by-layer inference for running 70B+ models on 8GB VRAM. However, it's not registered in the LLM provider registry and has no API endpoint. Users cannot trigger layer-by-layer inference through chat or any API call.
Also, the companion modules (meta_eviction #1952, kv_cache #1964, hf_quantizer #1954, attention_backend #1951) are all standalone utilities — none are wired into an end-to-end generation pipeline yet.
Discovered During
Batch 21 implementation of #1946, #1952, #1951, #1954.
Impact
High — Four inference optimization modules exist but form no usable pipeline. Each works in isolation (tested independently) but the integration that chains them (load config → detect quantization → select attention → layer-by-layer forward → meta evict → generate tokens) is not assembled.
Fix