Repository navigation
bug(kb): summary quality is scored against text[:200] — a 300-word summary judged on a 200-character window #16110
Description
Activity
- added 9 commits that reference this issue
on Sep 9, 2026 #16130 is merged (
c251c2f1fb) and it delivers two of the four criteria. Leaving this open — the other two are not met, and the difference between them is not cosmetic.Met
C1 — the evaluator's view is not a fixed truncation of the head.
recursive_summarizer.py:66-84on base:span = budget // 3 middle = (len(text) - span) // 2 tail = text[len(text) - span :] excerpt = f"{text[:span]}\n[...]\n{text[middle : middle + span]}\n[...]\n{tail}"
Head, middle and tail, with omissions marked rather than silently joined.
C2 — the prompt states what fraction it saw, and the score is not presented as fidelity.
:260:eval_query = _PARTIAL_SOURCE_NOTE.format(percent=coverage_percent(coverage)) + eval_query
with
coverage_percentflooring and holding anything short of whole at 99, so "partial" and "100%" cannot co-occur.Not met
C3 — a test asserts a fact placed past the truncation boundary and dropped from the summary lowers the score. The eleven tests assert the fact is in the excerpt:
assert "TAIL-FACT" in excerpt, "the end of a long document must still reach the evaluator"
That is a claim about the evaluator's input. The criterion is about its output — that a dropped fact actually moves the score. No test on base calls the scoring path at all: grepping the test file for
evaluate|quality_score|ResponseQualityreturns three hits, two of them docstring prose and one a test name.This is exactly the substitution this issue was filed about, one level out. The original defect was a score that could not see what it judged; the test now proves the evaluator can see the fact and never that seeing it changes anything. A widened window that fed into an evaluator ignoring it would pass every test here.
C4 — chunk, section and document levels each covered. No test exercises the three levels; all eleven drive
build_eval_excerptdirectly._EVAL_SOURCE_BUDGETis one constant, but the levels differ in what they feed it, and that is what the criterion asks about.Why this stays open rather than closing as substantially done
Two of four is real progress and the merged work is good — I reviewed it twice and held it once. But C3 is the criterion that would have caught the original bug, and it is the one still outstanding. Closing now would record that a blind score was fixed and verified, when what was verified is that the input is wider.
The remainder is a coherent piece of work: a test that scores a summary missing a tail fact and asserts the score is lower than for one containing it, across the three levels. That belongs to this issue rather than a successor — splitting it would let the fix land as "the excerpt is wider" while the fidelity claim stayed unproven, which is the failure mode
blocked_byon #16111 was recorded to prevent.Status against base
8e7d16918: three criteria met (ticked), one not yet. Leaving this open. The fix merged in #16130 (c251c2f1f).AC Verdict Evidence The evaluator's view is not a fixed head truncation ✅ knowledge/pipeline/cognifiers/recursive_summarizer.py:60build_eval_excerptsamples the head, middle and tail within_EVAL_SOURCE_BUDGET = 6000(:57). Its docstring records that it replacestext[:200], and it's used at:257.A partial source is stated as a fraction ✅ _PARTIAL_SOURCE_NOTE(:47-52) is prefixed onto the evaluation query whencoverage < 1.0(:259-260). It says "approximately {percent}% of the source … [...] marking omissions", and tells the evaluator not to treat an unseen fact as dropped.Chunk, section and document levels are each covered ✅ All three levels reach the same _summarize_with_refinement(:246): the chunk loop at:171, the section loop at:197, and the document at:228. The excerpt and the note are built in that one path, so none of the levels can bypass them.A test asserts a fact placed past the boundary and dropped from the summary ⬜ half recursive_summarizer_eval_coverage_test.py:36puts a fact at character 500 and asserts it reaches the evaluator's excerpt.:43is the contrast: the oldtext[:200]misses it.:52does the same for the tail of an oversized document. No test has a summary that omits the fact, and checks that the evaluator is given both the fact and a summary without it. That is the criterion's second half.To close: one test through
_summarize_with_refinementwith the LLM stubbed. The generated summary leaves out a late fact; the assertion is that the evaluator's input contains the fact and the summary it scores does not.- added a commit that references this issue
on Sep 12, 2026 Closing — fully delivered; never auto-closed because Dev_new_gui wasn't GitHub's configured default branch at merge time.
Verified against
origin/main:autobot-backend/knowledge/pipeline/cognifiers/recursive_summarizer.py:60build_eval_excerpt()does budget-based head/middle/tail sampling instead oftext[:200], called at line 257._PARTIAL_SOURCE_NOTE(:47) is applied at :260 when coverage is partial.recursive_summarizer_scoring_test.py:72test_a_dropped_late_fact_lowers_the_scoreassertsdropped < kept, closing the gap left by an earlier PR that only proved the fact reached the query, not that dropping it lowered the score. Test parametrized at chunk/section/document levels.Note: merged PR #16130 alone left the score-sensitivity criterion incomplete; PR #16345 completed it. Both were needed together.
Parent: #16108
Problem
RecursiveSummarizerscores every summary withResponseQualityEvaluator, asking it to judgewhether the summary "preserv[es] key facts and source attribution". The source it is given to
judge against is the first 200 characters:
autobot-backend/knowledge/pipeline/cognifiers/recursive_summarizer.py:204—source_preview = text[:200]:215-219—evaluator.evaluate(query=_SUMMARY_EVAL_QUERY.format(source_preview=source_preview), response=summary_text, ...):40-42— the eval query templatedocument_max_words = 300(:60)So a document-level summary of up to 300 words is scored for factual preservation against a
200-character window of its input. Anything dropped from past character 200 is invisible to
the check, and the refinement loop (
max_refinement_depth=2,:62) iterates against thatsame blind score.
Why this is worse than having no evaluator
The call returns a confident
quality_scorethat reads as evidence of fidelity while beingstructurally incapable of measuring it. A missing check is visible; a check that always passes
is not. Downstream,
best_score(:206) selects a "best" summary on this basis.Acceptance criteria
the score is not presented as a fidelity measure
lowers the score
Selector
grep -nE "evaluate\(|source_preview" autobot-backend/knowledge/pipeline/cognifiers/recursive_summarizer.py— control: the same selector returns the live call sites at
:215-219, so the empty result forfidelity terms elsewhere is a real absence, not a selector artefact (#16102).