Umbrella: self-correction, compaction-fidelity and measurement gaps from a comparative prompt/skill-library audit #16108

Context

A comparative audit of AutoBot's self-correction, compaction-fidelity and measurement
surfaces against an external prompt-and-skill library. Nothing here proposes importing
external code, data or dependencies — every adoptable item is a design decision restated
in AutoBot's own stack. Evidence (file:line, greps, controls) lives in each child, not
here.

Sibling umbrella: #13413, the same audit genus against a different reference and a
non-overlapping surface set (governance / approval / GUI). Read together, not merged.

What the audit found

The reference's corpus is a dead end for this scope: its curated dataset contains zero
self-correcting entries — every constraint in it is output-format discipline. Its loop
design is behind ours on every axis we checked. So this umbrella is deliberately small:
four narrow adoptions and three defects our own audit surfaced while looking.

We already do this better — do not re-raise

Deliberately NOT proposed

• Any runtime dependency on the external service. It is unversioned, unmetered, has no
  stability contract, and would sit in the request path — rejected against our egress policy.
• Adopting the corpus itself. It is unreviewed, persona-shaped for chat rather than
  tool-calling agents, and unevaluated.
• The reference's contribution model, which buys throughput by removing review. That is the
  trade this umbrella exists to avoid.

Children

Compaction fidelity

• bug(kb): summary quality is scored against text[:200] — a 300-word summary judged on a 200-character window #16110 — Summary quality is scored against text[:200] — a 300-word summary judged on a
  200-character window (highest severity here: the score is evidence-shaped)
• feat(kb): no compaction path verifies the source is recoverable from the summary — add a fidelity check #16111 — No compaction path verifies the source is recoverable from the summary
  (blocked by bug(kb): summary quality is scored against text[:200] — a 300-word summary judged on a 200-character window #16110 — fix the window before building on the signal)

Self-correction

• feat(rlm): reflection is a single scalar — no itemized checks and no re-audit after repair #16112 — Reflection is a single scalar: no itemized checks, no re-audit after repair
• feat(tools): tool constraints are per-tool only — no combination or ordering rules at the live gate #16113 — Tool constraints are per-tool only: no combination or ordering rules at the live gate
• chore(rlm): ReflectionResult records no provenance — no scoring model id, no evidence refs #16114 — ReflectionResult records no provenance (trivial; unblocks debugging the two above)

Measurement

• eval: the trajectory-eval gate is non-blocking with 3 goldens — broaden the corpus so review can safely shorten #16115 — The trajectory-eval gate is non-blocking with 3 goldens — the throughput item

Discovered while auditing

• tech-debt(agent_loop): docstring asserts schema self-correction is unported — it shipped in #4482 #16116 — Docstring asserts schema self-correction is unported; it shipped in feat(tools): Pydantic schema self-correction retry loop for tool argument validation #4482
• tech-debt(prompts): the served and loaded prompt-file sets disagree — .prompt/.yml/.yaml load but never serve #16121 — The served and loaded prompt-file sets disagree; .prompt/.yml/.yaml load but
  never serve, and any guard scoped to either list is narrower than the union
• tech-debt(quality): LLM-judged quality is implemented three times — ReflectionResult, JudgmentResult and ThinkCategory.SELF_REFLECTION #16133 — LLM-judged quality is implemented three times over three vocabularies
  (ReflectionResult / JudgmentResult / ThinkCategory.SELF_REFLECTION) with no shared type
  (take before feat(rlm): reflection is a single scalar — no itemized checks and no re-audit after repair #16112 and chore(rlm): ReflectionResult records no provenance — no scoring model id, no evidence refs #16114, which otherwise improve one plane of three)

Sequencing

#16114 and #16116 are independent and small — take them first. #16110 before #16111 (native
blocked_by recorded). #16115 is the highest-leverage item for delivery throughput and does not
collide with any of the others.

Related

• Umbrella: agent governance, measurement and approval-surface gaps from a comparative platform audit #13413 — sibling umbrella, same audit genus, different reference and non-overlapping surfaces
• security(prompts): four shipped prompt profiles instruct the model to never refuse on safety grounds #16105 — safety-bypass directives in four shipped prompt profiles, found during this audit and
  filed by a peer session after re-deriving it wider than first reported
• docs(process): name the selector — an empty result is not a true negative (#15953) #16102 — the selector/false-negative discipline this audit twice depended on
• bug(chat): a user instruction erodes across repeated compactions — verbatim protection does not compound #14393, umbrella: the tiered L0–L4 context stack was built and never ran — 2 of 5 layers cannot render #13685 — adjacent compaction and tiered-context work

Method note

Two findings in this audit initially rested on greps that returned nothing, and both were
partly false — one from too narrow a word list, one (in a peer session, on the same finding)
from a path that did not exist. Each child states the selector that produced its evidence.
See #16102.