Skip to content

Commit 4d320a4

Browse files
os-steveclaude
andauthored
docs(docs-audit): record that the periodic full-audit backstop is not running (#9428)
* docs(docs-audit): record that the periodic full-audit backstop is not running Part 4 of the docs-audit README described a cron routine running the docs-accuracy audit on a monthly/per-release cadence. Measured 2026-08-18: no such schedule exists. No GitHub Actions workflow runs the audit (the 11 scheduled workflows are all unrelated), no Routine on the agent-seat account runs it, the `schedule` skill the section named as the creation mechanism is gone, and repo history holds only three hand-initiated audit PRs with no cadence between them. This matters beyond the README: the #9192 option-A ruling (keep the docs-drift scope precise) rests on this backstop as its coverage leg. Recording the measurement rather than standing the routine up — a periodic LLM audit spends budget on a cadence, which is the maintainer's call. Also records a second defect the measurement surfaced: the old text scoped the routine to "the change-scoped doc list since the last audit" while the cost note called part 4 the "periodic full backstop". Only the latter is a backstop — a change-scoped list is derived by the same anchor heuristic whose misses the backstop exists to catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja * docs(docs-audit): correct the scheduled-workflow list to 10, derived not copied The first commit said 11 scheduled workflows and included `cut-rc`. That list was copied from an earlier triage comment rather than derived. On origin/main `cut-rc.yml` has no `schedule:` trigger — its only `schedule:` match is a comment stating it deliberately has none, which an unanchored grep counts. Ten workflows carry a real `schedule:` trigger. The conclusion is unchanged (none of them runs the docs audit), but a list consumed as authoritative without deriving it is the exact failure this card exists to measure, so the section now carries the derivation command instead of the copied list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja * docs(docs-audit): record that packageMentionDocs is not a superset, so it is no backstop The obvious cheap substitute for a missing full-audit backstop is to widen the audit's scope back to the coarse packageMentionDocs set. Measured across the 8 most recent packages/**-touching commits on main, that does not work: the coarse set is wider but not a superset — 3 to 7 pages present in the precise docs set are absent from packageMentionDocs in every one of the 8. That is the #9192 finding restated. The two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package. Swapping to it trades one incomplete set for another and loses pages the precise scope gets right. Only --all is a backstop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 6e211b7 commit 4d320a4

1 file changed

Lines changed: 66 additions & 8 deletions

File tree

‎scripts/docs-audit/README.md‎

Lines changed: 66 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -352,17 +352,75 @@ Always follow a run with the docs build gate:
352352
pnpm --filter @objectstack/docs build # must compile all pages clean
353353
```
354354

355-
## 4. Scheduled routine — periodic backstop
356-
357-
A cron routine (created via the `schedule` skill) runs on a cadence (default monthly /
358-
per-release) to catch drift the CI gate missed. It computes the change-scoped doc list
359-
since the last audit, runs the `docs-accuracy-audit` workflow on it, runs the build, and
360-
opens a PR when there are fixes. See the routine prompt for the exact steps.
355+
## 4. Scheduled routine — periodic backstop ⛔ NOT RUNNING
356+
357+
**Measured 2026-08-18: no live schedule runs this audit.** This section previously
358+
described the backstop in the present tense; it does not exist, so **it cannot be cited
359+
as coverage for what the part-1 scope leaves out.** Recorded rather than fixed on the
360+
spot: standing up a periodic LLM audit spends real budget on a cadence, which is the
361+
maintainer's call.
362+
363+
The intended design, for whoever stands it up: a cron routine on a cadence (default
364+
monthly / per-release) catches drift the CI gate missed — it runs the
365+
`docs-accuracy-audit` workflow, runs the build, and opens a PR when there are fixes.
366+
367+
**Two independent defects in the old text, both worth keeping in view:**
368+
369+
1. **It never ran.** Checked four ways, all negative — repeat these rather than
370+
re-deriving them:
371+
372+
- **GitHub Actions** — no workflow runs this audit at all. Of the 31 workflows
373+
registered on the repo, the 10 carrying a `schedule:` trigger (`codeql`,
374+
`coverage-nightly`, `engine-split-metric`, `prerelease-pin-watch`, `publish-smoke`,
375+
`rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, `validate-deps`)
376+
are all unrelated. Derive that list with `grep -rl '^\s*schedule:' .github/workflows/`
377+
rather than copying it: an unanchored grep for `schedule:` also matches `cut-rc.yml`,
378+
whose only hit is a comment saying it deliberately has no schedule.
379+
⚠️ **Read the registered workflow list and its run history, not the YAML on disk.**
380+
The two disagree in *both* directions: a workflow can be registered in Actions with
381+
no file on `main` (`matrix-aggregate-experiment.yml` is, today), and a scheduled
382+
workflow that GitHub auto-disabled after 60 days of repo inactivity leaves its file
383+
byte-identical. "The file is there" cannot answer "is it running" — the same
384+
read-a-conclusion-off-a-field-that-cannot-carry-it failure this docs-audit subsystem
385+
keeps paying for.
386+
- **Routines** — the Claude Code Remote `list_triggers` tool lists every Routine the
387+
agent-seat account owns, and this is the surface that answers the question (it was
388+
once written off as unreadable by any agent; it is not). The only cron Routine is the
389+
hourly triage seat, `18 * * * *`. There is no docs-audit Routine, enabled *or*
390+
disabled. A cron Routine is never hidden by the `include_completed` filter, so the
391+
absence is real rather than a listing artifact.
392+
- **The creation mechanism this section named is gone** — there is no `schedule` skill
393+
in `.claude/skills/`.
394+
- **No trace of a periodic run.** Repo history holds exactly three docs-accuracy audit
395+
PRs (#3243, #4219, #4312), every one hand-initiated against a named issue family, the
396+
most recent 2026-07-31. Nothing on a cadence.
397+
398+
2. **A change-scoped run is not a backstop.** The old text had the routine compute "the
399+
change-scoped doc list since the last audit", while the cost note below calls part 4 the
400+
"periodic **full** backstop". Those are two different runs and only the second is a
401+
backstop — a change-scoped list is derived by the very anchor heuristic whose misses
402+
the backstop exists to catch, so scoping it that way re-inherits the blind spot it is
403+
meant to cover. A backstop has to run `--all`: all 178 hand-written docs (run
404+
`check-audit-scope.mjs` for today's number rather than trusting this one).
405+
406+
3. **And the obvious cheap substitute is not a backstop either.** When the backstop is
407+
missing, the tempting one-line fix is to widen the audit's scope back to the coarse
408+
`packageMentionDocs` set part 1 still emits. Measured across the 8 most recent
409+
`packages/**`-touching commits on `main`: it is wider (21 pages vs 5 on `a4331227b`)
410+
but it is **not a superset** — in every one of the 8, between 3 and 7 pages present in
411+
the precise `docs` set are absent from `packageMentionDocs`
412+
(`api/error-catalog.mdx`, `automation/approvals.mdx`, `api/plugin-endpoints.mdx`,
413+
`data-modeling/relationships.mdx` among them). That is #9192's finding restated: the
414+
two sets miss in *different* directions, because the coarse one is a dependency-graph
415+
proxy and pages documenting a change through the SDK surface never name the
416+
implementing package. Swapping to it trades one incomplete set for another and loses
417+
pages the precise scope gets right. Only `--all` is a backstop.
361418

362419
---
363420

364421
**Cost note:** a full audit is ~2 agents per doc — measured at ~2.8M output tokens /
365422
~160 agents when the scope was 128 docs, and the hand-written set is 178 today (run
366423
`check-audit-scope.mjs` for the current number; don't trust a count written down here).
367-
Always prefer the change-scoped list (`affected-docs.mjs`) over `--all` except for the
368-
periodic full backstop.
424+
Always prefer the change-scoped list (`affected-docs.mjs`) over `--all`; reach for `--all`
425+
only for a deliberate full audit. (This sentence used to name the periodic full backstop as
426+
the exception — see part 4: that backstop is not running.)

0 commit comments

Comments
 (0)