Skip to content

ci(hook-selftests): discover the self-test matrices instead of enumerating them - #7019

Merged
os-sam merged 1 commit into
mainfrom
claude/issue-6906-hook-selftest-collector
Aug 31, 2026
Merged

os-sam merged 1 commit into
mainfrom
claude/issue-6906-hook-selftest-collector

Conversation

@claude

@claude claude Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #6906

What changed

.github/workflows/hook-selftests.yml names itself "The standing caller for .claude/hooks/*.selftest.sh" and was not one: its steps: were a hand-kept enumeration, so a new matrix shipped uncalled until someone remembered to edit this file. guard-tree-enum.selftest.sh is the receipt — it landed in both repos on the same card, objectstack's collector picked it up with no workflow edit, and here it had to be added by hand (the removed step even carries a comment saying so).

The three run: steps are replaced by one step that discovers its matrices:

mapfile -t selftests < <(find .claude/hooks -type f -name '*.selftest.sh' | sort)

Ported from objectstack's collector in its lint.yml (step "Claude hook guard self-tests"), keeping both load-bearing properties.

file why
.github/workflows/hook-selftests.yml the change
content/docs/guide/ci-cd-pipeline.md paired write — this workflow's own section hand-listed two of the three matrices, and would otherwise describe the change incorrectly
.changeset/6906-hook-selftests-collector.md empty-frontmatter changeset (CI-only; no published package's src/ changed)

.claude/** is not touched — governed surface, and this job is the runner, not the matrices. Verified mechanically against the base sha recorded at branch creation (b458300ca), not against a moving origin/main:

$ git diff --name-only b458300ca...HEAD | grep -c '^\.claude/'
0
$ node scripts/check-governed-queue-guard.mjs --test .github/workflows/hook-selftests.yml content/docs/guide/ci-cd-pipeline.md .changeset/6906-hook-selftests-collector.md
NOT GOVERNED - 3 path(s) checked against 5 governed surface(s); none matched.

The two load-bearing properties, exercised rather than read

Both were exercised against the step's real run: block, extracted from the YAML by a parser (not retyped) and executed as GitHub executes it, under bash -e. Every negative case ran in a scratch directory; nothing under the repo's real .claude/ was deleted, moved or disabled.

1. An empty discovery is RED

scratch dir with an EMPTY .claude/hooks/      -> exit 1
  "Hook self-tests - DISCOVERED NOTHING under .claude/hooks/."
scratch dir with NO .claude/ directory at all -> exit 1  (same message)

A step that verified nothing is not a pass: a renamed or moved directory cannot degrade this gate into a silent green no-op.

2. Tolerate-and-collect

Scratch .claude/hooks/ with three stubs — aaa-deliberately-red (exit 1), mmm-deliberately-red (exit 3), zzz-green (exit 0, prints a marker):

-- .claude/hooks/aaa-deliberately-red.selftest.sh
FAIL  .claude/hooks/aaa-deliberately-red.selftest.sh
-- .claude/hooks/mmm-deliberately-red.selftest.sh
FAIL  .claude/hooks/mmm-deliberately-red.selftest.sh
-- .claude/hooks/zzz-green.selftest.sh
LATER-MATRIX-RAN - 7 passed, 0 failed
PASS  .claude/hooks/zzz-green.selftest.sh

Hook self-tests - the following FAILED:
  .claude/hooks/aaa-deliberately-red.selftest.sh
  .claude/hooks/mmm-deliberately-red.selftest.sh
                                                   -> exit 1

The later matrix still ran, both failures are named, and the step is still red.

Counter-proof that the shape matters. The same three stubs under a naive for f in ...; do "$f"; done:

exit 1, and "LATER-MATRIX-RAN" occurrences in the log: 0

bash -e aborts at the first red matrix and the rest are neither green nor red — which is exactly what this collector must not do. Please do not "simplify" the loop.

find, not a flat glob. A stub dropped at .claude/hooks/nested/deep.selftest.sh in the same scratch tree is discovered; .claude/hooks/*.selftest.sh would miss it.

Coverage is not reduced — measured, not asserted

The hardcoded run: commands were parsed out of the workflow as it exists on the base sha, and set-differenced against what discovery returns on this tree:

hardcoded on b458300ca            discovered on HEAD
  guard-main-checkout-bash          guard-main-checkout-bash
  guard-shared-stash                guard-shared-stash
  guard-tree-enum                   guard-tree-enum

comm -23 hardcoded discovered  ->  0 lines   (nothing dropped)
comm -13 hardcoded discovered  ->  0 lines   (no net additions today)

Every self-test CI executes today is still executed. No self-test file was moved, renamed, skipped or quarantined.

Every discovered self-test, run locally

Exit codes captured individually, before any pipe:

self-test exit tail
.claude/hooks/guard-main-checkout-bash.selftest.sh 0 121 passed, 0 failed
.claude/hooks/guard-shared-stash.selftest.sh 0 41 passed, 0 failed
.claude/hooks/guard-tree-enum.selftest.sh 0 36 passed, 0 failed

And through the new collector, from the repo root: exit 0, discovered 3 hook self-test(s), Hook self-tests - all 3 ran and passed. None was failing on main, so there is no finding to report there.

scripts/dependabot-merge-gate.mjs — read, unchanged

Its OPTIONAL_CONTEXTS entry for Hook Self-Tests is already count-free ("the PreToolUse guard self-test matrices must pass"), and its paths sentence is still accurate after this change. Nothing was edited there. The workflow header now records why it must stay count-free, next to the existing argument for count-free step names.

Checks run (at 91eb09282, the head of this branch)

check result
YAML parse of the workflow (PyYAML) + bash -n on the extracted run: block parses; triggers, concurrency, permissions and timeout-minutes unchanged; syntax OK
pnpm exec vitest run scripts/__tests__/ 90 files, 2572 tests passed (includes ci-cd-pipeline-doc, merge-queue-reporting, dependabot-merge-gate, lint-workflow, the four check-changeset-* suites)
pnpm check:control-bytes OK - 5820 tracked text files
pnpm check:doc-fences / check:doc-types / check:doc-links OK
pnpm check:shell-escape-residue OK - 205 files, 1320 fenced blocks
pnpm check:skills-paths OK
pnpm check:action-forward-parity OK
pnpm check:governed-queue-guard OK - 132 self-test cases pass
pnpm check:entry-guard OK
node scripts/check-changeset-presence.mjs OK - 1 changeset added; no released source changed

Narrowings, declared

  • pnpm check:doc-snippets was NOT run to a verdict. It exits 2, PRECONDITION NOT MET - it needs every package built, and its own output says that is "I could not run", not a failure. It reads TypeScript blocks in docs; this diff adds and removes zero fenced blocks (git diff b458300ca...HEAD -- content/docs/guide/ci-cd-pipeline.md | grep -c '^[+-].*\``'is0`), so there is nothing here it could judge. CI runs it either way.
  • Repo-wide pnpm lint was narrowed away, and the narrowing is measured, not assumed. Population read from eslint.config.js itself: every files: selector in it is .ts/.tsx. This diff's .ts/.tsx count is 0 (3 changed files: one .yml, two .md), and it touches no eslint.config.* and no tsconfig*, so no untouched file's lint verdict can move either. CI runs the farm regardless.

Draft on purpose - the PM lands this. #6451 remains open and is a different mechanism (a missing self-test FILE); this PR is the runner that will call it once it exists.


Generated by Claude Code

…ating them

`.github/workflows/hook-selftests.yml` calls itself "the standing caller for
`.claude/hooks/*.selftest.sh`" and was not one: its `steps:` were a hand-kept
enumeration, so a new matrix shipped uncalled until someone remembered to edit
this file. Measured, not argued — `guard-tree-enum.selftest.sh` landed in both
repos on the same card, objectstack's collector picked it up with no workflow
edit, and here it had to be added by hand.

Ports objectstack's collector (its `lint.yml`, "Claude hook guard self-tests")
into a single step that runs `find .claude/hooks -type f -name '*.selftest.sh'
| sort`, keeping both load-bearing properties:

  - an EMPTY discovery is RED, so a renamed or moved directory cannot degrade
    the gate into a step that verified nothing and reported green;
  - the loop TOLERATES AND COLLECTS rather than sequencing, so one red matrix
    cannot abort the run under `bash -e` and hide the others; the step still
    fails, naming every matrix that failed.

Coverage is unchanged and measured: discovery returns exactly the three
self-tests the removed steps ran (set difference "hardcoded minus discovered"
is empty). Does not touch `.claude/**` — governed surface, and this workflow is
the runner, not the matrices.

Paired write: `content/docs/guide/ci-cd-pipeline.md`, whose section for this
workflow hand-listed two of the three matrices.
`scripts/dependabot-merge-gate.mjs` was read and left unchanged — its
`OPTIONAL_CONTEXTS` description of this check is already count-free.
Merged via the queue into main with commit d42c2be Aug 31, 2026
32 checks passed
yinlianghui pushed a commit that referenced this pull request Sep 2, 2026
`guard-main-checkout.sh` was the only PreToolUse guard in this repo with no
self-test — and it is the one both CLAUDE.md and AGENTS.md §9 name by path as
the mechanism that enforces worktree-first. Its Bash-side follow-on and
guard-shared-stash both ship hermetic matrices that `hook-selftests.yml` runs
on every PR touching `.claude/hooks/**`; nothing exercised the primary guard at
all, in either direction (a fail-open regression stops guarding the shared
checkout in silence; a fail-closed one reads to an agent as a broken
environment).

Ported from objectstack's copy of the same matrix. The port is verbatim: the
two repos' hooks differ by 9 diff lines, all inside one comment block, no
executable line differs, and both settings.json route the same
Edit|Write|NotebookEdit matcher — the sibling file was run here byte-for-byte
unmodified first and returned 87 passed, 0 failed. The only edits are the two
KNOWN HOLE sections, re-pointed at this repo's own cards for the same defects.

87 cases, 0 failed. No wiring edit is owed: `hook-selftests.yml` discovers
matrices with `find` since #7019, and its header states that its recorded case
counts are a dated measurement that must not be refreshed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BGMDbrVa8JjZcCQ7DWYH1b
akarma-synetal pushed a commit to akarma-synetal/objectui that referenced this pull request Sep 9, 2026
….sh (objectstack-ai#7264)

`guard-main-checkout.sh` was the only PreToolUse guard in this repo with no
self-test — and it is the one both CLAUDE.md and AGENTS.md §9 name by path as
the mechanism that enforces worktree-first. Its Bash-side follow-on and
guard-shared-stash both ship hermetic matrices that `hook-selftests.yml` runs
on every PR touching `.claude/hooks/**`; nothing exercised the primary guard at
all, in either direction (a fail-open regression stops guarding the shared
checkout in silence; a fail-closed one reads to an agent as a broken
environment).

Ported from objectstack's copy of the same matrix. The port is verbatim: the
two repos' hooks differ by 9 diff lines, all inside one comment block, no
executable line differs, and both settings.json route the same
Edit|Write|NotebookEdit matcher — the sibling file was run here byte-for-byte
unmodified first and returned 87 passed, 0 failed. The only edits are the two
KNOWN HOLE sections, re-pointed at this repo's own cards for the same defects.

87 cases, 0 failed. No wiring edit is owed: `hook-selftests.yml` discovers
matrices with `find` since objectstack-ai#7019, and its header states that its recorded case
counts are a dated measurement that must not be refreshed.


Claude-Session: https://claude.ai/code/session_01BGMDbrVa8JjZcCQ7DWYH1b

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

finding(ci): hook-selftests.yml hand-enumerates the self-tests it claims to be the standing caller for — a new matrix ships uncalled

1 participant