Repository navigation
docs: record how to write a gate that can actually fail - #789
Merged
Merged
Conversation
Much of this repo's CI is gates — tests whose job is to fail when someone
removes a property, not to check that today's code works. `make check`
runs dozens. They are cheap to write and easy to write uselessly, and a
gate that cannot fail is worse than no gate because it is counted as
coverage.
Four rules, each from a gate in THIS repo that broke it:
1. Verify against a revert that COMPILES. Three gates were "verified"
against reverts that failed go vet first — unused variable, dropped
import — so the test never ran and the verification proved nothing.
2. Test the invariant where it lives. #787's !Determined guard survived
a compiling revert because every test built the struct through
DetectIdleness, which cannot produce the offending combination.
3. Match the assignment or invocation, never the bare name. Four gates
matched their own prose: spawn:version inside a parameter
Description and an echo line, put-retention-policy inside a comment
saying which call to make.
4. Discover the set, don't list it, and fail when discovery finds
nothing. A gate over a hand-written list reports a clean bill of
health for everything the list forgot.
Plus the corollary: a suspicious negative indicts the invocation more
often than the system. Four false negatives in one session came from the
probe — `strings` on a Go binary, awk $2 on tab-separated `gh pr checks`
output, --max-items reshaping a JMESPath result to null, and
FilterLogEvents --limit returning an empty FIRST page with a
continuation token. The last nearly produced a fix to working code.
Written from the gates as they stand, not from memory: the quoted match
forms are the ones in lambda-deploy-census.sh, and two examples were
corrected while drafting. The census does not hardcode function names
today — its first version did, which is the point — and the footprint
exact-name map is cited as the surviving unavoidable list, with the rule
that entries come from a live account and never from recollection.
Also fixes the `make check` description, which still said "fmt, vet,
lint, short tests" after #788 added the nested-module loop.
Docs only. make check green, including all 12 nested modules.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes the convention this session kept rediscovering into
CLAUDE.md, as a## Writing a gatesection.A large share of this repo's CI consists of gates: tests whose job is to fail
when someone removes a property, rather than to check that today's code works.
They are cheap to write and easy to write uselessly — and a gate that cannot
fail is worse than no gate, because it is counted as coverage.
Four rules, each drawn from a gate in this repo that broke it:
go vetfirst — the test never ran!Determinedguard survived a compiling revert; every test built the struct throughDetectIdleness, which cannot produce the offending combinationspawn:versionin a parameterDescriptionand anecho,put-retention-policyin a commentPlus the corollary for absences: a suspicious negative indicts the invocation
more often than the system. Four false negatives in one session came from the
probe, not the code —
stringson a Go binary,awk $2on tab-separatedgh pr checksoutput,--max-itemsreshaping a JMESPath result tonull, andFilterLogEvents --limitreturning an empty first page with a continuationtoken. The last nearly produced a "fix" to working code.
Written from the gates, not from memory
Two examples were wrong in the first draft and were corrected against the
source rather than shipped:
scripts/lambda-deploy-census.shdoes not hardcode function names today —it enumerates, and its first version is the cautionary case. Rule 4 now cites
the first version, and names
footprintExactNamesinpkg/aws/footprint.goasthe surviving unavoidable list, with the rule that an entry is added only after
seeing it in a live account.
real anchored forms (
^\s*spawn:version:\s*!Ref\s+Version,--tags.*spawn:version=,^\s*aws logs put-retention-policy).Also corrects the
make checkdescription, which still said "fmt, vet, lint,short tests" after #788 added the nested-module loop.
Verification
Docs only; no Go source changed.
make checkgreen, exit 0, including✅ all 12 nested modules build, vet and test clean.Changelog entry is under[Unreleased]— verified by line number against the first release heading, withthe release-heading count unchanged at 121, because an entry of mine landed in a
released section earlier in this session.