Repository navigation
docs: make the integration path hard to get wrong; fix what the review found behind it - #14
Merged
Merged
Conversation
…y filter that cannot be met
A `require` entry carrying a `tag` marks the row a later job keys on: the one that
uploads coverage, publishes artifacts, or reports the release build. Dropping it
leaves a matrix that looks complete while that job silently never runs, so
`generateRows` now throws for one it cannot satisfy, whether or not
`failOnUnsatisfiableFilters(true)` is set. pgjdbc guards its coverage row with a
hand-written assertion after the call; this makes the library answer for it.
Phase 1 reserves a row for each open tagged requirement, and untagged ones anchor
out of what is left. Anchoring the tagged ones first would satisfy them just as
reliably and cost them their variety: the anchor is the row that carries the
packing bonus, so tagged-first aims every forced pairing at the row the caller
reasons about. On a four-requirement fixture that leaves the tagged row a varied
partner on 108 of 300 seeds, against 90 for the symmetric packer it replaces and
none for tagged-first. One tagged match outweighs every untagged match combined,
but only while the reserve is short; unconditionally, a packed row chases a tagged
requirement that was in no danger.
Requirements that must share rows are packed greedily, so a budget below one row
per tagged requirement can strand one even where a packing exists. The error names
the budget that makes it unreachable, counting rows pinned before the call.
`Axis.candidateValues` answers which values a filter admits, so a filter pinning
`0`, `false` or `''` is honored rather than tested for truth and dropped -- with
the old test a tagged pin on a falsy value threw on 9 of 40 seeds at a budget with
room to spare. `pickValue` keeps its throw for direct callers, and
`_generateCandidate` returns null instead, so an unsatisfiable filter of any shape
-- a misspelling, a falsy pin, a predicate matching nothing -- reports through
`generateRows`: it warns, or throws with the tagged message, or obeys
`failOnUnsatisfiableFilters`. The coverage fill asks for its rows with warnings
off, so a fill filter admitting no value is checked before that phase rather than
truncating the matrix in silence.
Diagnostics name what they are about: a filter written as predicates rendered as
`{}`, because `JSON.stringify` drops a function, and a require entry carrying a
tag with no filter rendered as nothing at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewing a fresh integration of this generator (uber/NullAway#1781) turned up seven misses, six of which the README could have prevented. They are the ones every integration rediscovers, so document them where they are looked up: - Reproducibility. `RNG_SEED` and `GITHUB_PR_NUMBER` were documented only in a JSDoc comment, so a workflow copied from the README drew a fresh matrix on every push and could not replay a failing row. The example workflow wires both and takes a seed through `workflow_dispatch`. - Choosing axes. What makes an axis worth having, four groups to draw from, the rule that an axis has to reach the code under test rather than the process that launches it, and the rule that the configuration a project ships belongs on axes as much as the environment does. - Ruling out combinations that do not exist. `imply()` states the rule the way you know it and `exclude()` states it inside out; a filter matches the axis value as declared, so an object-valued axis needs `{scram: {value: 'yes'}}`; nothing reports a rule that never fires, and a misshaped `imply()` consequent rejects every row its antecedent admits rather than doing nothing; and a rule kept to two axes also drops those pairs from the pairwise targets. - Job count and cost. The budget bounds the rows `generateRows` creates, and rows pinned through `generateRow()` are returned whether or not they fit. `weight` is the importance of an uncovered pair, so it leans the fill and saturates: the section gives the command that measures it on the shipped example and the counts it prints. - Caching. A randomized matrix changes the cache key of any action that hashes the matrix into it; `gradle/actions` is the worked example. - An integration checklist. `examples/matrix.mjs` gains an `assertions` axis, since `-ea` costs only the jobs that carry it, and weights on the `os` axis, which the cost section calls for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
vlsi
force-pushed
the
docs/integration-guidance
branch
from
August 30, 2026 16:07
b5da7a4 to
7bf23a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Reviewing a fresh integration of this generator (uber/NullAway#1781) turned up seven misses, and six were ones the README could have prevented. They are the same misses every integration rediscovers:
RNG_SEEDandGITHUB_PR_NUMBERwere documented only in a JSDoc comment, and the README's workflow example set neither, so a workflow copied from it draws a fresh matrix on every push. A failure on an exotic row then vanishes on the next commit, and nobody can tell whether the fix worked or the row simply did not come up again.osaxis was left unweighted whileallAxisValues('os')guaranteed all three, so most rows landed on the runners that cost 2x and 10x.gradle/actions/setup-gradlehashes the matrix into its cache key, so a randomized matrix stops hitting the matrix-level restore key and writes a new entry per job per run.JAVA_HOME, while the test tasks kept taking their JVM from a toolchain. Seven job names, one set of bytes.The seventh was a library gap: pgjdbc guards its coverage row with a hand-written
if (!include.some(v => v.collectCoverage)) throw, because itsfailOnUnsatisfiableFiltersis deliberately off. The README points at pgjdbc'smatrix.mjsas the current usage, so the workaround propagates without the reason.Then six rounds of cross review found four defects behind that gap, three in the fix itself. They are in this branch because it made each symptom fatal.
What
A tagged requirement is load-bearing, so the budget reserves a row for it.
generateRowsthrows when a{filter, tag}requirement cannot be satisfied, whether or notfailOnUnsatisfiableFilters(true)is set: the tag marks a row a later job keys on, so dropping it leaves a matrix that looks complete while that job never runs. Phase 1 holds a row for each open tagged requirement rather than anchoring them first — anchoring satisfied them and cost them their variety, because the anchor is the row that carries the packing bonus, and on one fixture the tagged row went from a varied partner on 90 of 300 seeds to none. The bonus outweighs every untagged match with one tagged match, but only while the reserve is short.A filter that pins
0,falseor''is now honored.Axis.pickValuetested its filter for truth, so such a pin was ignored during generation and held only when the draw agreed — 9 of 40 seeds threw at a budget with room to spare.Axis.candidateValuesanswers which values a filter admits;pickValuekeeps its throw for direct callers, and_generateCandidatereturnsnull, so an unsatisfiable filter of any shape reports throughgenerateRows. The coverage fill asks for its rows with warnings off, so a fill filter admitting no value is checked before that phase rather than truncating the matrix in silence. Diagnostics name what they are about: a filter written as predicates used to render as{}.Requirements that share rows are still packed greedily, so a budget below one row per tagged requirement can strand one even where a packing exists. That budget is tested, documented, and named in the error, which counts rows pinned before the call.
The README gains five sections and a checklist: reproducibility and the seed, choosing axes (including the rule that an axis must reach the code under test, and that a project's own configuration belongs on axes), ruling out combinations that do not exist (
implyversusexclude, and why a filter matches the axis value as declared), job count and cost, and caching. Every heading that was there before is unchanged, so existing anchors still resolve.The review corrected four claims in those sections: weights lean the fill rather than setting a rate;
failOnUnsatisfiableFiltersdoes not catch a misshaped constraint filter; a misshaped filter is a no-op inexcludeand in animplyantecedent but a purge in a consequent; andgenerateRowsdoes return rows pinned before the call even past the budget.examples/matrix.mjsgains anassertionsaxis and weights onos.How to verify
npm test63 tests. Eleven are new, and each fails when its own hunk is reverted and only then.
Prints 175 of 300 rows at the example's
weight: 4, which is the figure the cost section quotes. Set the weight to1,40and1000and it prints 97, 180 and 180.Open
The version. This turns warnings into throws and changes the matrix for anyone with a falsy pin, and
git tagshows the project does use major bumps. It is set to 2.5.0 here; 3.0.0 is the safer reading, and it is your call.