docs(research): the disk-tiered MoE streaming analysis against llm_shared/optimization (#13030)

[mrveiss]

Refs #13030

Single-issue rationale: a docs-only research write-up for the #13030 umbrella. The findings are filed as their own issues (#16569, #16570), so this PR carries only the analysis and its index row.

Thinking Path

A research-only session compared an external disk-tiered MoE expert-streaming engine with AutoBot's llm_shared/optimization/ stack, with Phase 1 and Phase 2 done under the audit-first gate. It made no commits and left the doc uncommitted in the shared main checkout. That tree must stay clean, so the doc lands here on a branch cut from the latest origin/main.

What Changed

• docs/research/moe-expert-tiered-streaming.md: the analysis, with file:line evidence into llm_shared/optimization/. The external engine stays unnamed, per the third-party naming convention. A closing "Filed issues" table maps each finding to its issue.
• docs/research/_index.md: the index row, citing umbrella(optimization): large-model inference on lean hardware is scaffolded but non-functional — 7 defects #13030 and the filed issues.

Findings:

• Adopt now, as feat(optimization): every lossy optimization reports a sampled fidelity score, not just its savings #16569: report a sampled fidelity score beside every lossy optimization's savings. Prompt compression and HF quantization report savings only.
• Adopt with conditions, as feat(optimization): persist layer access frequency across restarts to seed the warm set (blocked by #13035) #16570, blocked by tech-debt(optimization): LayerInferenceAdapter never registered + duplicate meta-eviction implementation (#13030) #13035: persist layer access frequency across restarts. The layer-streaming engine has no production caller until tech-debt(optimization): LayerInferenceAdapter never registered + duplicate meta-eviction implementation (#13030) #13035 registers LayerInferenceAdapter.
• Rejected on hidden metrics: a disk-persisted KV cache, and dual-SSD weight striping. AutoBot scales this problem horizontally instead.
• Recorded, not filed: layer-granularity streaming can't capture MoE sparsity. That only matters if a local MoE model is targeted.

Verification

• Docs only. It closes nothing: this references umbrella(optimization): large-model inference on lean hardware is scaffolded but non-functional — 7 defects #13030, and feat(optimization): every lossy optimization reports a sampled fidelity score, not just its savings #16569 and feat(optimization): persist layer access frequency across restarts to seed the warm set (blocked by #13035) #16570 are separate.
• The doc was scanned for internal info: no IPs, hostnames, paths or personal names. Its "token" hits are inference tokens.
• The pre-commit and pre-push hooks passed on d03f48d1c.

Model Used

Claude Opus 5 (coordinator session). The research itself came from a separate research-only session.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: f87e4d19-5126-425d-826c-e67fd67299d6

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Notice: 29 open PRs — past the runaway threshold (25)

There is no PR queue limit, and this is not a request to defer this PR. Work proceeds one issue at a time without a cap on open PRs; review capacity is the constraint.

This notice only means the count is high enough to be worth a glance for a runaway — something opening PRs in a loop, or a merge pipeline that has stalled so nothing is draining.

Currently open:

• security(chat): finish #16545's PII/injection scan on /stream-ai-stack and resume (#16561) #16567 security(chat): finish security(chat): scan/redact PII and flag injection risk on every raw chat message (#16529, #16530) #16545's PII/injection scan on /stream-ai-stack and resume (security(chat): the #16545 PII/injection scan leaks a 200 on /stream-ai-stack and can be skipped there #16561)
• fix(knowledge): give _CPUBackend a non-recursive vector-search leaf (#15165) #16564 fix(knowledge): give _CPUBackend a non-recursive vector-search leaf (perf(api): /api/knowledge_base/search stops answering under concurrent load — hidden for the life of the file by a swallowed timeout #15165)
• fix(rag): wire GPU faiss onto an automated install, gated on GPU alone (#15163) #16560 fix(rag): wire GPU faiss onto an automated install, gated on GPU alone (fix(rag): FAISS-GPU vector search path never activates — faiss-gpu is installed by no automated path #15163)
• docs(research): add internal audit of LLM request-pipeline gaps #16559 docs(research): add internal audit of LLM request-pipeline gaps
• fix(infra): clear the debt blocking the last four branch-name swaps, and default install.sh to main (#16540) #16558 fix(infra): clear the debt blocking the last four branch-name swaps, and default install.sh to main (infra(git): fix the debt blocking the last five branch-name swaps (#16482 section 1) #16540)
• fix(config): register AUTOBOT_BROWSER_SERVICE_HOST/PORT, AUTOBOT_VNC_PORT (#15151) #16555 fix(config): register AUTOBOT_BROWSER_SERVICE_HOST/PORT, AUTOBOT_VNC_PORT (bug(config): AUTOBOT_PLAYWRIGHT_HOST and AUTOBOT_PLAYWRIGHT_API_PORT are documented but read by nothing #15151)
• fix(scripts): detect-environment.sh only selects files this repo tracks (#15143) #16553 fix(scripts): detect-environment.sh only selects files this repo tracks (bug(scripts): detect-environment.sh selects three .env files that do not exist #15143)
• fix(security): agent_config, sync and hardware-priority record the real caller, not "admin" (#16547, #16541) #16550 fix(security): agent_config, sync and hardware-priority record the real caller, not "admin" (fix(agents): api/agent_config.py records the audit trail as created_by="admin" for every actor #16547, bug(security): three agent-config writes record created_by="admin" instead of the real caller #16541)
• feat(llm): wire extractive prompt compression and enforce per-provider concurrency cap (#16526, #16527) #16546 feat(llm): wire extractive prompt compression and enforce per-provider concurrency cap (Wire the existing extractive prompt compressor into the live LLM completion path #16526, Enforce the existing per-provider LLM concurrency cap #16527)
• security(browser): validate the URL in capture_screenshot (#13204) #16544 security(browser): validate the URL in capture_screenshot (security(browser): send_to_browser_vm and playwright_service bypass the SSRF/DNS-rebind guard the other browser paths enforce #13204)
• fix(perf): re-derive Multimodal processor startup's budget post-#15297 (#16535) #16537 fix(perf): re-derive Multimodal processor startup's budget post-fix(ml): drop resume_download, removed from from_pretrained in transformers v5 (#15054) #15297 (perf(test): Multimodal processor startup budget still calibrated against #15054's broken (work-skipping) baseline #16535)
• fix(llm-shared): remove dead SemanticLLMCache, keep live semantic cache (#16528) #16536 fix(llm-shared): remove dead SemanticLLMCache, keep live semantic cache (Consolidate duplicate semantic LLM-response cache implementations #16528)
• feat(tooling): check_run_status gains a pending-with-age helper (#16126) #16534 feat(tooling): check_run_status gains a pending-with-age helper (docs(process): two review findings on MEASUREMENT_DISCIPLINE.md — family order and D's missing mechanism #16126)
• fix(files): conversation uploads accept PDF and GIF, not the truncated '.pd' and '.gi' (#16521) #16523 fix(files): conversation uploads accept PDF and GIF, not the truncated '.pd' and '.gi' (bug(files): conversation uploads reject every PDF and GIF because the allowlist says '.pd' and '.gi' #16521)
• fix(infra): consolidate the last 3 private TLS-cert-generation copies (#16020) #16522 fix(infra): consolidate the last 3 private TLS-cert-generation copies (nginx cannot start on a node with no TLS cert: five roles consume the shared keypair, four provision it #16020)
• security(lint): classify test_first_remediation.py as production, fix the 2 real gaps it surfaced (#16514) #16518 security(lint): classify test_first_remediation.py as production, fix the 2 real gaps it surfaced (chore(lint): classify scripts/test_first_remediation.py as production and document the sync_orchestrator ssh gap #16514)
• docs(infra): record redis role tools as binary names, not units (#16060, #16071) #16511 docs(infra): record redis role tools as binary names, not units (Redis is managed by two unit names that do not exist, and a failed restore-stop is silent #16060, redis naming: five more sites name a unit the fleet does not have (#16060 follow-up) #16071)
• refactor(lifespan): split cleanup_services and _init_graph_rag_service into steps under the length limit (#16250) #16509 refactor(lifespan): split cleanup_services and _init_graph_rag_service into steps under the length limit (function-length-check pre-commit hook blocks any commit to lifespan.py: cleanup_services (267 lines) and _init_graph_rag_service (68 lines) exceed the 65-line ceiling #16250)
• security(routers): gate knowledge_search, knowledge_search_aggregator, knowledge_suggestions and voice_stream (#16507) #16508 security(routers): gate knowledge_search, knowledge_search_aggregator, knowledge_suggestions and voice_stream (security(routers): gate knowledge_search, knowledge_search_aggregator, knowledge_suggestions and voice_stream #16507)
• test(user-management): a user in org A gets no results from org B (#16279) #16506 test(user-management): a user in org A gets no results from org B (security(users): /user-management/users/search answers anonymous callers with names from every organisation #16279)
• security(settings): record the real caller, not the literal "admin" (#16501) #16504 security(settings): record the real caller, not the literal "admin" (security(settings): sync_config and update_hardware_priority record the literal "admin" instead of the real caller #16501)
• fix(chat-knowledge): contexts can be deleted, cascade with their chat, and orphans get cleaned (#16490) #16503 fix(chat-knowledge): contexts can be deleted, cascade with their chat, and orphans get cleaned (bug(chat-knowledge): contexts can be created but never deleted, so orphans accumulate #16490)
• fix(slm-frontend): give the service-log fetch one path and its real timeout (#16256) #16500 fix(slm-frontend): give the service-log fetch one path and its real timeout (tech-debt(slm-frontend): the two log viewers fetch the same journal through two paths with different timeouts #16256)
• fix(a2a): cooperative cancellation checkpoints in execute_a2a_task (#16174) #16498 fix(a2a): cooperative cancellation checkpoints in execute_a2a_task (Cancelling an A2A task does not stop its executor, so claims can only lapse rather than release #16174)
• fix(sdk): TypeScript SDK request/response parity + missing /api prefix (#15528, #16495) #16497 fix(sdk): TypeScript SDK request/response parity + missing /api prefix (bug(sdk): the TypeScript SDK still sends the parameters and parses the envelopes the Python SDK stopped sending #15528, bug(sdk): every TypeScript SDK request omits the /api prefix — 404 against a real backend #16495)
• refactor(agents): extract NPUCodeSearchAgent's pattern helpers, add declared_scopes (#16173) #16496 refactor(agents): extract NPUCodeSearchAgent's pattern helpers, add declared_scopes (NPUCodeSearchAgent cannot declare its write scopes: its file is at the size ceiling #16173)
• fix(frontend): converge the permission vocabulary onto the backend's real Permission enum, wire in v-permission (#16243) #16493 fix(frontend): converge the permission vocabulary onto the backend's real Permission enum, wire in v-permission (tech-debt(frontend): usePermissions' permission strings share nothing with the backend's Permission vocabulary #16243)
• security(research): treat fetched repos and pages as untrusted data (#16488) #16492 security(research): treat fetched repos and pages as untrusted data (security(research): treat fetched repos and pages as untrusted data: prompt-injection hardening for research, adopt and web-fetch #16488)
• feat(frontend): wire router meta.minRole into a real guard, ranked against the backend's role order (#16244) #16489 feat(frontend): wire router meta.minRole into a real guard, ranked against the backend's role order (tech-debt(frontend): router meta.minRole is declared and read by no guard #16244)

If the queue is draining normally, ignore this. Otherwise:

1. Check whether CI is dispatching at all — see the ci-dispatch-watchdog status on these PRs
2. Merge the ones whose CI has finished and review has passed: gh pr merge <number> --squash --delete-branch
3. Look for a loop opening near-identical PRs

Warn-only runaway detector — .github/workflows/pr-queue-gate.yml. It never blocks a merge.

[mrveiss]

Content review: approve.

Well-sourced research write-up, docs-only. Verified both "filed separately" findings exist and match the description exactly: #16569 (fidelity-score-on-lossy-toggles, OPEN) and #16570 (persisted access-frequency, blocked by #13035, OPEN). Correctly gates its "What We Can Adopt" verdicts against the existing #13031-13036 audit state (notably: #13035's adapter-never-registered means the underlying engine has no production caller yet, which the doc explicitly uses to discount throughput claims to zero and count only maintenance cost — good discipline). Evidence is concrete file:line citations against the current tree, not just prose claims.

No manual link (closingIssuesReferences=[]), matches body (closes nothing, as stated). CI green. Ready to add to a batch.