Repository navigation
feat(kb): chunking-strategy comparison runner — five production chunkers, zero quality measurement #13245
Description
Problem
Five distinct chunking strategies run in production paths, each chosen by whoever wrote the call
site, and none has ever been compared against a retrieval-quality metric. We cannot answer "which
chunker should markdown docs use?" or "is the semantic chunker actually better than paragraph
splitting for code comments?" — the information does not exist.
This is not a convergence request: the strategies are legitimately different and several were
deliberately kept separate. What is missing is a way to measure them so the choice at each call
site is evidence-based.
Evidence
Production chunkers and their call sites (all verified by grep):
| Strategy | Module | Callers |
|---|---|---|
| Markdown H2/H3 structure, ~800-token packing | autobot_shared/doc_chunking.py |
autobot-backend/services/knowledge/doc_indexer.py:27-30, autobot-infrastructure/shared/tools/index_documentation.py:53 |
| Paragraph packing, char-bounded | autobot-backend/utils/text_chunking.py |
autobot-backend/knowledge/connectors/file_server.py:30, autobot-backend/services/knowledge/cognition_seeder.py:30 |
| Sentence-embedding percentile split | autobot-backend/utils/semantic_chunker.py |
autobot-backend/npu_semantic_search.py:516,532, autobot-backend/llm_shared/structured_ops.py:314, autobot-backend/api/codebase_analytics/chromadb_storage.py:215 |
| Token-count segmentation with overlap | autobot-backend/knowledge/pipeline/extractors/semantic_chunker.py |
autobot-backend/knowledge/pipeline/extractors/__init__.py:13 |
| Fixed-size sliding window with overlap | autobot-backend/utils/payload_optimizer.py |
payload trimming path |
autobot-backend/utils/text_chunking.py:16-21 explicitly records that the last two were kept
separate on purpose — that decision stands, and this issue does not revisit it.
No module compares any two of these. autobot-backend/knowledge/rag_benchmarks.py:188-214 has a
test_document_chunking_benchmark, but it measures chunking throughput (ms/op) on a synthetic
string, not retrieval quality.
Proposed approach
- Define one narrow chunker interface:
split_text(text) -> list[str]. Every existing strategy
gets a thin adapter — no rewrites, no behaviour changes to the strategies themselves. - Add a comparison runner that, for a fixed corpus + query set, indexes with each chunker in turn
and reports the span metrics per strategy in one table. - Report per content type (markdown docs, source code, chat transcripts) — the winner is almost
certainly different per type, and a single global winner would be the wrong conclusion. - Output a leaderboard artifact so results are reviewable and re-runnable, not a one-off log line.
Depends on the span-metrics issue in this umbrella; without it the comparison is meaningless
(see that issue for why doc-ID precision@k cannot compare chunkers).
Risks / constraints
- Each strategy × corpus combination re-embeds the corpus. Cache embeddings keyed by
(chunker, chunk params, embedding model) or the run cost is unbounded. - The adapters must not alter existing call-site behaviour; the strategies stay byte-identical.
- Two strategies are async and hardware-dispatched; the adapter must not force them onto a sync
path in production code.
Acceptance criteria
- Adapter for each of the five strategies, with a test asserting adapter output is identical to
the strategy's current output for a fixed input. - Comparison runner produces a per-strategy metric table for at least three content types.
- Results committed as a reviewable artifact, with the run command documented.
- No behavioural change at any existing call site, shown by the existing chunker tests passing.
Activity
mrveiss commentedon Aug 1, 2026
Part of umbrella #13251 (retrieval quality measurement). See that issue for phase ordering, the audited "already do well" list, and the options deliberately rejected.
mrveiss commentedon Oct 4, 2026
A third arm worth adding: contextualized text
Suggestion from a research pass over the reference work, verified against merged code before posting.
Alongside the arms this issue already weighs, add contextualized embedding: embed the heading/clause path together with the body, while displaying only the body.
The retrieval unit gains the context that disambiguates it; the user still sees clean text.
Why it is not already covered
#17209 puts the heading path in metadata only. Metadata is filterable but not embedded, so a chunk whose body says "it grew 12%" is embedded without any trace of what "it" was. Two chunks from different sections with near-identical bodies are near-identical vectors today.
Dependency, stated rather than assumed
This needs extraction to preserve heading structure, which it does not currently do for DOCX —
extract_docxkeeps onlyp.textand drops style (now #17985). So this arm cannot be evaluated fairly until that lands, and a benchmark run before it would measure the absence of headings rather than the value of contextualizing them.