Skip to content

docs: record how to write a gate that can actually fail - #789

Merged
scttfrdmn merged 1 commit into
mainfrom
docs/gate-convention
Oct 10, 2026
Merged

scttfrdmn merged 1 commit into
mainfrom
docs/gate-convention

Conversation

@scttfrdmn

Copy link
Copy Markdown
Contributor

Writes the convention this session kept rediscovering into CLAUDE.md, as a
## Writing a gate section.

A large share of this repo's CI consists of gates: tests whose job is to fail
when someone removes a property, rather than to check that today's code works.
They are cheap to write and easy to write uselessly — and a gate that cannot
fail is worse than no gate, because it is counted as coverage.

Four rules, each drawn from a gate in this repo that broke it:

rule what broke it
Verify against a revert that compiles three gates "verified" against reverts that failed go vet first — the test never ran
Test the invariant where it lives #787's !Determined guard survived a compiling revert; every test built the struct through DetectIdleness, which cannot produce the offending combination
Match the assignment or invocation, never the bare name four gates matched their own prose — spawn:version in a parameter Description and an echo, put-retention-policy in a comment
Discover the set, don't list it a gate over a hand-written list reports a clean bill of health for everything the list forgot

Plus the corollary for absences: a suspicious negative indicts the invocation
more often than the system. Four false negatives in one session came from the
probe, not the code — strings on a Go binary, awk $2 on tab-separated
gh pr checks output, --max-items reshaping a JMESPath result to null, and
FilterLogEvents --limit returning an empty first page with a continuation
token. The last nearly produced a "fix" to working code.

Written from the gates, not from memory

Two examples were wrong in the first draft and were corrected against the
source rather than shipped:

  • scripts/lambda-deploy-census.sh does not hardcode function names today —
    it enumerates, and its first version is the cautionary case. Rule 4 now cites
    the first version, and names footprintExactNames in pkg/aws/footprint.go as
    the surviving unavoidable list, with the rule that an entry is added only after
    seeing it in a live account.
  • Rule 3 originally quoted a grep form the gates don't use. It now quotes the
    real anchored forms (^\s*spawn:version:\s*!Ref\s+Version,
    --tags.*spawn:version=, ^\s*aws logs put-retention-policy).

Also corrects the make check description, which still said "fmt, vet, lint,
short tests" after #788 added the nested-module loop.

Verification

Docs only; no Go source changed. make check green, exit 0, including
✅ all 12 nested modules build, vet and test clean. Changelog entry is under
[Unreleased] — verified by line number against the first release heading, with
the release-heading count unchanged at 121, because an entry of mine landed in a
released section earlier in this session.

Much of this repo's CI is gates — tests whose job is to fail when someone
removes a property, not to check that today's code works. `make check`
runs dozens. They are cheap to write and easy to write uselessly, and a
gate that cannot fail is worse than no gate because it is counted as
coverage.

Four rules, each from a gate in THIS repo that broke it:

  1. Verify against a revert that COMPILES. Three gates were "verified"
     against reverts that failed go vet first — unused variable, dropped
     import — so the test never ran and the verification proved nothing.

  2. Test the invariant where it lives. #787's !Determined guard survived
     a compiling revert because every test built the struct through
     DetectIdleness, which cannot produce the offending combination.

  3. Match the assignment or invocation, never the bare name. Four gates
     matched their own prose: spawn:version inside a parameter
     Description and an echo line, put-retention-policy inside a comment
     saying which call to make.

  4. Discover the set, don't list it, and fail when discovery finds
     nothing. A gate over a hand-written list reports a clean bill of
     health for everything the list forgot.

Plus the corollary: a suspicious negative indicts the invocation more
often than the system. Four false negatives in one session came from the
probe — `strings` on a Go binary, awk $2 on tab-separated `gh pr checks`
output, --max-items reshaping a JMESPath result to null, and
FilterLogEvents --limit returning an empty FIRST page with a
continuation token. The last nearly produced a fix to working code.

Written from the gates as they stand, not from memory: the quoted match
forms are the ones in lambda-deploy-census.sh, and two examples were
corrected while drafting. The census does not hardcode function names
today — its first version did, which is the point — and the footprint
exact-name map is cited as the surviving unavoidable list, with the rule
that entries come from a live account and never from recollection.

Also fixes the `make check` description, which still said "fmt, vet,
lint, short tests" after #788 added the nested-module loop.

Docs only. make check green, including all 12 nested modules.
@codecov

codecov Bot commented Oct 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@scttfrdmn
scttfrdmn merged commit 6bb9502 into main Oct 10, 2026
7 checks passed
@scttfrdmn
scttfrdmn deleted the docs/gate-convention branch October 10, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant