Repository navigation
A question picks its definitions instead of taking the first thirty #574
Description
Chat reads its confirmed definitions with confirmed(kb, 30): the first thirty by concept name. The seeded upper bound on the wide corpus, 27 definitions, scored 17/18 (#520) — three short of the cap. A real base with a hundred definitions leaves seventy of them, the ones later in the alphabet, outside the prompt on every question. “Twenty-seven works” does not extrapolate.
The other wall is already visible at 27: the one miss in the seeded run was the model asking which of three adjacent names (gross profit, gross margin, gross merchandise value) the person meant. More definitions in the prompt means more of that.
Shape
Definitions become retrievable, in the shape the ontology index already has: concept_mappings.embedding / embedded_model / embedded_text (migration 0048), re-embedded when the model or the text changes, refreshed lazily before a lookup so nobody has to remember to.
Two channels, RRF-fused, top k = 8 into the prompt:
- vector — “name: summary (unit)” embedded; the question embedded once; pgvector orders. Absent when no embedding model is configured.
- lexical — every confirmed definition pulled back and scored in Rust by how many of the question’s tokens it contains (ASCII words; CJK bigrams). A base has tens to a few hundred definitions; this is cheaper than Chinese tokenisation in SQL and needs no
pg_trgm.
Degradation: vector + lexical → lexical only → the first k by name when nothing matches (today’s behaviour). No step is “no definitions”.
GET /kbs/{id}/mappings/relevant?q=&k= exposes the same lookup, so the bench can score it and a page can later show which definitions a question used.
Measured by
recall@k, without asking anything. The bench already knows which definition each question needs (by number, #520), so ask.mjs --recall 8 calls the endpoint per question and checks whether a correct row is in the top 8 — seconds per round, no chat model. Misses are annotated with the truth file’s near field: if they cluster on near-name pairs, that is the case for a reranker (a separate cut, with a third model slot and this same degradation); if they scatter, the embedded text is the thing to fix.
A base that exceeds the cap: mappings.mjs --fresh --corpus wide --also tpch mounts both sources into one base, seeding both truths gives 45 definitions across two sources. Then ask.mjs on each corpus checks that 17/18 survives selection.
Activity
WaylandYang commentedon Sep 9, 2026
First measurement, on a base that actually exceeds the old cap: wide and tpch mounted together, both truths seeded, 45 confirmed definitions across two sources (the old
confirmed(kb, 30)would have dropped fifteen of them by alphabet).For reference, the single-source seeded runs were 24/24 (tpch) and 17/18 (wide). So selection kept the tpch upper bound intact with 45 definitions in play, and cost wide two questions.
Where the wide misses come from, and it is not near-names
The five wide misses at recall@8 (
valid_orders,valid_lines,aov,gross_profit,gross_margin) all have the same shape: the question is Chinese, the seeded definition is English (Paid orders (truth): Paid orders — bench truth), and the top 8 fill up with tpch definitions whose English summaries happen to share tokens. Two things stack:--seedwriteslabel — from, which for wide isPaid orders — bench truth. A person or exploration writes a real summary; the bench does not, and the embedded text is mostly that summary.tpch’s one miss (
late_lines→Late deliveries — Q12, receipt after commit) is a paraphrase miss, and chat got the question right anyway.So the misses are not clustered on the truth file’s
nearpairs. That is the answer to the reranker question for now: not yet. The lever is the embedded text — a definition’s summary in the language people ask in — and the bench should stop seeding junk summaries before the next measurement. A reranker gets its own cut when a recall miss shows two adjacent names both in the top 8 and the wrong one first.Also fixed on the way:
ask.mjsnow scores only the definitions on the corpus’s own source (the other source’s rows reference tables the corpus base does not have).WaylandYang commentedon Sep 9, 2026
Same 45-definition base, same questions, one change: the seeded definitions now carry a real one-line summary in the language people ask in (the wide truth gained
summaryfor all 18; tpch uses itsnote/from). Twenty of the 45 rows re-embedded lazily on the first lookup, as designed.Wide went from two questions lost to selection to none — and 18/18 is one better than the single-source seeded run, whose one miss was the model asking which of three adjacent names the person meant; with eight relevant definitions in the prompt instead of twenty-seven, it answered.
So the whole wide gap was the embedded text, and the lever was a sentence per definition. Nothing about the retrieval changed. tpch keeps its one paraphrase miss (
late_lines), which chat answers correctly regardless.Reranker: still no case for it. There is no miss left where two adjacent names both sit in the top 8 with the wrong one first. This measurement stands as the bar a reranker would have to move.
WaylandYang commentedon Sep 9, 2026
Re-measured on merged dev (
0fc44a7e, the binary that carries #561–#575 together, including the bounded refresh and the model-scoped vector channel), same 45-definition two-source base,--recall 8:late_lines, the same paraphrase)Unchanged from the branch numbers above, as expected at 45 rows and one model. The bar for a reranker stands where it did.
Make Stop cancel chat generation and preserve the partial answer (#969)