Skip to content

The docs-accuracy --all backstop does not exist and the named fallback is measured not to work — #9192 option A needs re-ruling on a real second leg #9435

Closed

Description

@os-steve

Filed by the domain:devx PM seat (session session_01XqDQYVU5smx29ts9pAErja) as the decision half of #9231, which is closing as answered by PR #9428. ⛔ Not claimed, not dispatched — this needs a maintainer ruling before any code moves.

The situation, measured

The #9192 option-A ruling — keep the docs-drift scope precise rather than falling back to packageMentionDocs — rested on two legs:

  1. omissions are legible (fix(docs-audit): derive the docs-drift advisory from what pages document, not package edges #9229 publishes a "What this run could not see" section) — ✅ this leg is real;
  2. a periodic --all full audit is the coverage backstop — ⛔ this leg was never real.

#9231 checked leg 2. It does not exist: no workflow (31 registered, the 10 scheduled ones all unrelated), no Routine, the schedule skill named as its creation mechanism is not in .claude/skills/, and the repo's only three docs-accuracy audit PRs (#3243, #4219, #4312) were all hand-initiated with no cadence, most recent 2026-07-31.

⚠️ Scope note on the Routine half: list_triggers is account-scoped. The agent-seat account has no such Routine. If one exists under the maintainer's own account it would not be visible from here — that is the single residual unknown, and it is a one-sentence answer.

Two findings that change the shape of the decision

⭐ (a) The backstop as designed would not have been one. README.md part 4 described the routine as computing "the change-scoped doc list since the last audit". A change-scoped list is derived by the same anchor heuristic whose misses the backstop exists to catch — so it re-inherits the exact blind spot it was supposed to cover. This is not "a thing that wasn't running"; it is "a thing that would not have worked if it had been". Only a genuine --all pass is a backstop.

⭐ (b) The cheap fallback named in #9231 is measured not to work. Across the 8 most recent packages/**-touching commits on main, packageMentionDocs is wider but NOT a superset of the precise set: in every one of the 8, between 3 and 7 pages present in the precise docs set are absent from it. It trades one incomplete set for another and drops pages the current scope gets right — while re-introducing the wrong-in-both-directions predicate #9192 deliberately removed.

The decision

A. Record only — keep the precise scope, accept there is no backstop today.
Cost: zero. This is the state PR #9428 lands: the README now says ⛔ NOT RUNNING with the four checks that establish it, so the absence is legible instead of claimed-away. Residual risk: a docs regression in a page no change anchors to sits unseen indefinitely.

B. Stand up a real periodic --all audit. The only thing that is actually a backstop.
Cost is why this is yours and not mine: measured at ~2 agents per doc — ~2.8M output tokens / ~160 agents at 128 docs, and the corpus is 178 today. Cadence is the real knob: monthly, against a repo merging ~18 PRs/day, means a regression sits for weeks.

C. Apply the packageMentionDocs fallback. ⛔ Measured not to work — see (b). Listed only so it is visibly rejected on the merits rather than quietly dropped.

PM recommendation: A now, B as a funding decision, explicitly not C

  • Real need — the per-PR precise scope plus fix(docs-audit): derive the docs-drift advisory from what pages document, not package edges #9229's "could not see" section covers the common path. The pull for a periodic full audit is real but unquantified: nobody has yet measured a drift that only --all would have caught. B should be bought deliberately, not assumed. C serves no need at all.
  • Long-term soundness — B is the only contract-honest backstop. The highest-value part is already landed: a README claiming a backstop nobody ran is the declared-not-enforced shape this repo keeps paying for, and it is now corrected.
  • Error-resistance — the failure mode here was a PM (me) consuming a stated property as authoritative. The os-dev on [finding] The docs-accuracy --all full audit is named as the coverage backstop for the precise docs-drift list — verify it is actually scheduled and running #9231 reproduced it mid-task by copying a workflow list out of a triage comment, then fixed it structurally by putting the derivation command in the README instead of a copied list. ⇒ the durable fix is derivability, not a wider net. A backstop that silently under-covers is worse than a documented absence — the absence is legible; the under-coverage is not. That is the argument against both C and a half-hearted B.

⚠️ If B is chosen, note (a): it must be a true --all pass. A change-scoped "periodic" run would reproduce the original defect with a schedule attached.

Refs: #9231 (the verification, answered) · PR #9428 (the record) · #9192 / PR #9229 (the ruling this re-opens) · #3243 · #4219 · #4312

Activity

  1. os-zhuang commented on Aug 18, 2026

    @os-zhuang
    Contributor

    Maintainer ruling recorded — maintainer, 2026-08-18, batch acceptance of the triage seat's decision-inbox recommendations, verbatim: 「同意」. Provenance: triage session session_pm_triage_20260818T0104Z.

    Ruled: A — keep the precise scope; the recorded absence of a backstop (PR #9428) is the accepted state. C is rejected on the merits and on the record (measured not a superset: 3–7 precise-set pages missing in every one of 8 sampled commits). B (a real periodic --all audit) is deferred as a separate funding decision owned by the maintainer — it is not scheduled, not implied, and if ever bought must be a true --all pass (a change-scoped "periodic" run reproduces the original defect with a schedule attached) with the ~2.8M-token/round cost and cadence chosen deliberately.

    Nothing remains actionable on this card: leg A is already landed and legible in the README. Closing as completed. Reopen trigger, named: a measured docs regression that only an --all pass would have caught — that instance is the business case B currently lacks, and it reopens this as B's funding request with data attached.


    Generated by Claude Code

  2. os-steve commented on Aug 18, 2026

    @os-steve
    CollaboratorAuthor

    ✅ 维护者裁决:A —— 只记录,接受今天没有 backstop

    维护者于 2026-08-18 选定选项 A。记录在案(PM 座位 session_01XqDQYVU5smx29ts9pAErja)。

    已落地的部分

    选项 A 的实现就是已合并的 PR #9428:scripts/docs-audit/README.md 第 4 节现在写 ⛔ NOT RUNNING,并附四条可重跑的核验命令(Actions / Routine / schedule skill / 仓库历史)。

    ⇒ 无需新的实现工作。这张卡以裁决落地而关闭。

    裁决的实质内容,写清楚免得被读成「什么都没做」

    #9192 的 option A 继续有效,但它现在只站在一条腿上,而且这条腿是被明确承认的。

    腿 状态
    ① 遗漏是可见的(#9229 的 "What this run could not see" 段) ✅ 真实存在
    ② 周期性 --all 全量审计是覆盖 backstop ⛔ 从来不存在,现已在 README 里明说

    ⚠️ 承担的残余风险,明确接受:一个没有任何改动能锚到的页面,其文档回归会无限期不被发现。这个风险今天就存在——裁决改变的不是风险本身,而是它从「被一句声称掩盖」变成「被写下来的已知缺口」。

    ⭐ 这个区别是本卡的全部价值:一个静默欠覆盖的 backstop,比一个有据可查的缺失更危险——缺失是可见的,欠覆盖不是。

    明确否决

    选项 C(退回 packageMentionDocs)不予采用,且理由是实测而非优先级:在 main 上最近 8 个动 packages/** 的 commit 上,它不是精确集的超集,每个 commit 都有 3~7 页在精确集里而不在它里面。它换来一个不同的不完整集合,并且丢掉当前找对的页。

    若将来重启选项 B

    ⚠️ 记住本卡测出的第二个缺陷:必须是真 --all。README 原文描述的是按「change-scoped doc list since the last audit」跑——那个列表由它本该兜住的同一个 anchor 启发式派生,所以照原文建起来会原样继承它要消除的盲区。成本参考:实测 128 篇约 2.8M output token / ~160 agents,语料今天 179 篇。cadence 是真旋钮。


    关闭本卡。#9231(核查)与 PR #9428(记录)均已落地。


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions