Repository navigation
epic(kb): retrieval quality is unmeasurable — real benchmarks, span metrics, chunker comparison, evidence-based tuning #13251
Description
Why
We can change retrieval quality but we cannot measure it. Every chunking and retrieval constant in
the KB path was chosen by hand and has never been scored against a retrieval-quality metric, and the
one benchmark harness we have scores a synthetic corpus using a fake embedding function. The result
is that no one can answer "did that change make retrieval better?" with evidence.
This umbrella closes that loop: real measurement first, then comparison, then the tuning the
measurement unlocks. The ordering is deliberate — the tuning issues are worthless without the
measurement issues, and running them early would produce confident numbers we cannot trust.
Task tree
Phase 1 — make the measurement real (blocking; nothing downstream is meaningful without these)
- fix(kb): RAG benchmark harness scores a synthetic corpus with a fake embedding — wire it to the real KB and embedding model #13243 — benchmark harness runs against the real KB collection and the real embedding model,
not a 20-doc synthetic corpus and a 128-term count vector - feat(kb): add span-level retrieval metrics (recall/precision/precision-ceiling/IoU) — doc-ID precision@k cannot compare chunkers #13244 — span-level metrics (recall / precision / precision-ceiling / IoU) so strategies with
different chunk sizes are comparable at all - feat(kb): generate a domain-specific retrieval eval set from AutoBot's own corpora #13248 — generate a domain-specific eval set from our own corpora, with excerpt verification,
quality filtering, and de-duplication
Phase 2 — compare what we already run
- feat(kb): chunking-strategy comparison runner — five production chunkers, zero quality measurement #13245 — comparison runner across the five production chunking strategies, per content type
- feat(kb): bounded retrieval-config sweep runner with a significance gate and budget ceiling #13249 — bounded sweep runner with a significance gate and an enforced budget ceiling
Phase 3 — tune, on evidence
- feat(kb): sweep and validate chunk size + overlap defaults — 800-1000 token chunks were never measured #13246 — sweep and validate chunk size and overlap defaults
- feat(kb): add global-segmentation semantic chunker — percentile-threshold splitting is the weaker variant #13247 — add a global-segmentation semantic chunker and measure it against the incumbent
What we already do well (audited — no work proposed)
Each of these was checked against the code before anything was filed, and each is stronger than the
comparable technique in the reviewed external work:
- Held-out dev/test discipline, runtime-enforced.
autobot-backend/knowledge/rag_benchmarks.py:933-1419—BenchmarkSplit,
BenchmarkDataset.enforce_dev_only/enforce_test_only, and aheld_out_scoreflag that is only
true when the run touched test IDs and no dev IDs. Cross-split access raises, it is not advisory.
This is the single biggest weakness in the external work and we already have the answer to it.
fix(kb): RAG benchmark harness scores a synthetic corpus with a fake embedding — wire it to the real KB and embedding model #13243 fixes what the discipline is applied to, not the discipline itself. - Two-stage retrieval already exists. Hybrid search plus optional cross-encoder reranking:
autobot-backend/knowledge/search.py:194-291,autobot-backend/knowledge/vector_search_engine.py:276-322,
autobot-backend/utils/hybrid_search.py. - Structure-aware document chunking. Markdown is split on H2/H3 boundaries with section metadata
preserved (autobot_shared/doc_chunking.py), rather than blind sentence or character splitting.
Whether its sizes are right is feat(kb): sweep and validate chunk size + overlap defaults — 800-1000 token chunks were never measured #13246; its approach is sound. - Optimizer wired into production.
autobot-backend/services/rag_service.py:19,104-128— the
RAG optimizer is live in the service path, not a stranded module. - Deliberate chunker separation is already documented.
autobot-backend/utils/text_chunking.py:16-21
records why two chunkers were kept separate during fork convergence. feat(kb): chunking-strategy comparison runner — five production chunkers, zero quality measurement #13245 measures the
strategies; it does not reopen that decision.
Deliberately NOT proposed (do not re-raise without new information)
- An unattended agent loop that edits pipeline source and hard-resets git per discard. This is
the headline idea of the reviewed external work and it is rejected on hidden costs: destructive
history rewriting on every discard, unbounded overnight API spend with no ceiling anywhere in the
design, greedy hill-climbing into local optima, and keep/discard decisions taken on differences
smaller than the measurement noise — which yields a confident written log of improvements that are
not real. The parts worth having (a durable leaderboard, a fixed comparison protocol) are in
feat(kb): bounded retrieval-config sweep runner with a significance gate and budget ceiling #13249 in non-destructive form. - Converging the five chunking strategies onto one implementation. They are legitimately
different and the separation was a considered decision. Measure first (feat(kb): chunking-strategy comparison runner — five production chunkers, zero quality measurement #13245); convergence is only
justified if the data shows two strategies are equivalent. - Replacing the percentile semantic chunker. It has four production call sites. feat(kb): add global-segmentation semantic chunker — percentile-threshold splitting is the weaker variant #13247 adds an
alternative and measures it; the incumbent stays until evidence says otherwise. - Replacing the synthetic benchmark fixture. It runs with no model and no network, which is worth
keeping for CI. fix(kb): RAG benchmark harness scores a synthetic corpus with a fake embedding — wire it to the real KB and embedding model #13243 adds a real path beside it rather than swapping it out.
Closure gate
This umbrella closes when Phase 1 and Phase 2 are merged and at least one Phase 3 sweep has produced
a result — including a null result. "We measured and the current defaults are already best" is a
legitimate outcome and closes #13246; "we never measured" is not.
Activity
mrveiss commentedon Aug 1, 2026
This gap is already blocking shipped work. #10621 (enable chat tiered routing by default) is gated on "benchmark chat quality with tiered routing ON vs OFF via
autobot-backend/knowledge/rag_benchmarks.py" — a validation that cannot currently be performed, because that harness scores a 20-doc synthetic corpus with a 128-term count vector instead of the real embedding model (see #13243). A cost-saving routing change has therefore sat gated off rather than being validated. #13243 unblocks it.mrveiss commentedon Aug 3, 2026
A comparative platform audit (umbrella #13413) filed #13415 as the SQL-path half of this epic:
services/nl_database_service.pyconverts natural language to SQL and executes it, and has no test file and no eval (rg -l 'nl_database'returns the service, the router, two schema files, the router registry and generated FE types — nothing else).Proposed to extend the existing
eval/harness rather than build a second one: golden reference queries graded on execution accuracy (result sets, position-matched columns, names stripped), 4-way verdicts, and a floor test so the corpus cannot be narrowed to raise the score.mrveiss commentedon Aug 4, 2026
Cross-link from a code-intelligence audit (umbrella #13467).
#13472 (cohesion-based community re-split + hub exclusion in
community_clusterer) changes which mesh nodes get promoted to retrieval anchors. It is explicitly blocked on this epic: until retrieval is measurable, landing it would swap one unmeasured heuristic for another with no way to show the swap was an improvement.No overlap in scope — this epic builds the measurement, #13472 is a consumer of it.
mrveiss commentedon Aug 4, 2026
Sequence map: this umbrella is track U#13251 in #13483, which places all 282 open issues into ordered tracks.
Derived order for this track's 7 open children:
How much to trust this: track membership is reliable. The order is derived from stated blockers (only 9 exist backlog-wide), then priority label, then issue number — it is a starting default, not a proven critical path. Where a child carries its own
## Implementation order, that wins.64% of open issues have no priority label, which is why some rows sort last for no good reason — see #13485.
mrveiss commentedon Aug 8, 2026
Cross-link: umbrella #13685 independently reconfirms this blocker from the context-assembly side.
An audit of the tiered L0–L4 context stack found two of five layers structurally unable to render (#13686, #13687) and the whole stack flag-off with no recorded A/B result (#13689). Five of the items under that umbrella want a tuning decision, and none can be evaluated while
knowledge/rag_benchmarks.pyguards a fake embedding over 20 synthetic docs.Concretely, this issue gates the one design idea worth importing from an external layered-memory implementation: event-driven distillation with idle-flush and min/max interval bands instead of fixed crontabs. Adding seven interacting scheduler constants before a recall-quality metric exists would be adding tuning surface we cannot evaluate — the same mistake the external implementation's own docs concede they made.
#13689 is deliberately not blocked on this: it judges what the stack demonstrably puts in the prompt, plus token cost and latency, all observable today. What this issue blocks is tuning the layers, not deciding whether they fire.
feat(memory): enable the tiered L0-L4 context stack (#13689)
feat(memory): run the #5066 A/B, record the result, and enable the ti…
feat(memory): add an idle-flush trigger to skill distillation (#13695)
feat(memory): add an idle-flush trigger to skill distillation (#13695)
feat(memory): add an idle-flush trigger to skill distillation (#13695)
feat(memory): add an idle-flush trigger to skill distillation (#13695) (
17 remaining items
mrveiss commentedon Oct 3, 2026
Correction: the scheduling inversion I reported above was not real. Seven of the eight edges were spurious and have been removed.
My earlier comment reported seven dependents scheduled before their blocker and asked whether to move this epic forward or the dependents back. That framing rested on the edges, and the edges were wrong. Nothing was scheduled against a blocker three milestones out, because seven of those eight issues were never blocked by this one.
Exactly one dependent is genuine: #13472, which states the dependency in its own voice with reasoning — "#13251 builds the measurement, this consumes it." Its edge is untouched. And even it is decision-permissive: "Land after #13251, or at minimum with a recorded before/after anchor-set diff on this issue."
The other seven all trace to one copied line. Six of the eight contain this status-table row verbatim:
That line says #13472 is blocked. It does not say the containing issue is. It appears in #13470, #13473, #13475, #13476, #13477 and #13467 — boilerplate replicated across a sibling set, with a native edge drawn from each occurrence because
13251matched in the body.#13483 is the clearest case: its two mentions are "#13251 (retrieval measurement) gates #13472" and a list entry
#13472 <-blocked by #13251. The sequencing tracker was subject to a gate it was accurately documenting.#13473 settles it beyond argument. Its body reads "Safe to start immediately. Independent of the code-graph track and of #13251" — while carrying a live edge to #13251. The edge contradicted the text it was derived from.
What was actually released
Three in-milestone, not the five a first count suggested — two of the seven had other open blockers the #13251 edge was masking:
Removing a spurious edge does not make an issue startable — it makes the remaining edges visible. That distinction cost two issues off the count.
Mechanism, so it is not repeated
13251appearing in a body was treated as this body depends on 13251, with no check on what the sentence asserted. A prose reference to a sibling's gate became an edge on the containing issue.Each removal carries its evidence on the issue itself. #13472's edge is left in place. The
needs-decisionlabel can come off as far as the reschedule-versus-repoint question goes: neither is needed, because the dependency was real for one issue that already has its own escape hatch.