Skip to content

docs: make the integration path hard to get wrong; fix what the review found behind it - #14

Merged
vlsi merged 3 commits into
mainfrom
docs/integration-guidance
Aug 30, 2026
Merged

vlsi merged 3 commits into
mainfrom
docs/integration-guidance

Conversation

@vlsi

@vlsi vlsi commented Aug 30, 2026 •

Copy link
Copy Markdown
Owner

Why

Reviewing a fresh integration of this generator (uber/NullAway#1781) turned up seven misses, and six were ones the README could have prevented. They are the same misses every integration rediscovers:

  • The seed was never wired. RNG_SEED and GITHUB_PR_NUMBER were documented only in a JSDoc comment, and the README's workflow example set neither, so a workflow copied from it draws a fresh matrix on every push. A failure on an exotic row then vanishes on the next commit, and nobody can tell whether the fix worked or the row simply did not come up again.
  • The os axis was left unweighted while allAxisValues('os') guaranteed all three, so most rows landed on the runners that cost 2x and 10x.
  • Nothing was said about caching. gradle/actions/setup-gradle hashes the matrix into its cache key, so a randomized matrix stops hitting the matrix-level restore key and writes a new entry per job per run.
  • An axis was added that never reached the tests: the JDK vendor changed JAVA_HOME, while the test tasks kept taking their JVM from a toolchain. Seven job names, one set of bytes.
  • Only the environment went on axes. The project's own options — where pgjdbc gets most of its value — were not considered.

The seventh was a library gap: pgjdbc guards its coverage row with a hand-written if (!include.some(v => v.collectCoverage)) throw, because its failOnUnsatisfiableFilters is deliberately off. The README points at pgjdbc's matrix.mjs as the current usage, so the workaround propagates without the reason.

Then six rounds of cross review found four defects behind that gap, three in the fix itself. They are in this branch because it made each symptom fatal.

What

A tagged requirement is load-bearing, so the budget reserves a row for it. generateRows throws when a {filter, tag} requirement cannot be satisfied, whether or not failOnUnsatisfiableFilters(true) is set: the tag marks a row a later job keys on, so dropping it leaves a matrix that looks complete while that job never runs. Phase 1 holds a row for each open tagged requirement rather than anchoring them first — anchoring satisfied them and cost them their variety, because the anchor is the row that carries the packing bonus, and on one fixture the tagged row went from a varied partner on 90 of 300 seeds to none. The bonus outweighs every untagged match with one tagged match, but only while the reserve is short.

A filter that pins 0, false or '' is now honored. Axis.pickValue tested its filter for truth, so such a pin was ignored during generation and held only when the draw agreed — 9 of 40 seeds threw at a budget with room to spare. Axis.candidateValues answers which values a filter admits; pickValue keeps its throw for direct callers, and _generateCandidate returns null, so an unsatisfiable filter of any shape reports through generateRows. The coverage fill asks for its rows with warnings off, so a fill filter admitting no value is checked before that phase rather than truncating the matrix in silence. Diagnostics name what they are about: a filter written as predicates used to render as {}.

Requirements that share rows are still packed greedily, so a budget below one row per tagged requirement can strand one even where a packing exists. That budget is tested, documented, and named in the error, which counts rows pinned before the call.

The README gains five sections and a checklist: reproducibility and the seed, choosing axes (including the rule that an axis must reach the code under test, and that a project's own configuration belongs on axes), ruling out combinations that do not exist (imply versus exclude, and why a filter matches the axis value as declared), job count and cost, and caching. Every heading that was there before is unchanged, so existing anchors still resolve.

The review corrected four claims in those sections: weights lean the fill rather than setting a rate; failOnUnsatisfiableFilters does not catch a misshaped constraint filter; a misshaped filter is a no-op in exclude and in an imply antecedent but a purge in a consequent; and generateRows does return rows pinned before the call even past the budget.

examples/matrix.mjs gains an assertions axis and weights on os.

How to verify

npm test

63 tests. Eleven are new, and each fails when its own hunk is reverted and only then.

for s in $(seq 1 60); do RNG_SEED=$s MATRIX_JOBS=5 node examples/matrix.mjs; done | grep -c "os: 'ubuntu-latest'"

Prints 175 of 300 rows at the example's weight: 4, which is the figure the cost section quotes. Set the weight to 1, 40 and 1000 and it prints 97, 180 and 180.

Open

The version. This turns warnings into throws and changes the matrix for anyone with a falsy pin, and git tag shows the project does use major bumps. It is set to 2.5.0 here; 3.0.0 is the safer reading, and it is your call.

@vlsi vlsi changed the title docs: make the integration path hard to get wrong, and fail on a dropped tagged requirement docs: make the integration path hard to get wrong; fix what the review found behind it Aug 30, 2026
vlsi and others added 3 commits August 30, 2026 19:06
…y filter that cannot be met

A `require` entry carrying a `tag` marks the row a later job keys on: the one that
uploads coverage, publishes artifacts, or reports the release build. Dropping it
leaves a matrix that looks complete while that job silently never runs, so
`generateRows` now throws for one it cannot satisfy, whether or not
`failOnUnsatisfiableFilters(true)` is set. pgjdbc guards its coverage row with a
hand-written assertion after the call; this makes the library answer for it.

Phase 1 reserves a row for each open tagged requirement, and untagged ones anchor
out of what is left. Anchoring the tagged ones first would satisfy them just as
reliably and cost them their variety: the anchor is the row that carries the
packing bonus, so tagged-first aims every forced pairing at the row the caller
reasons about. On a four-requirement fixture that leaves the tagged row a varied
partner on 108 of 300 seeds, against 90 for the symmetric packer it replaces and
none for tagged-first. One tagged match outweighs every untagged match combined,
but only while the reserve is short; unconditionally, a packed row chases a tagged
requirement that was in no danger.

Requirements that must share rows are packed greedily, so a budget below one row
per tagged requirement can strand one even where a packing exists. The error names
the budget that makes it unreachable, counting rows pinned before the call.

`Axis.candidateValues` answers which values a filter admits, so a filter pinning
`0`, `false` or `''` is honored rather than tested for truth and dropped -- with
the old test a tagged pin on a falsy value threw on 9 of 40 seeds at a budget with
room to spare. `pickValue` keeps its throw for direct callers, and
`_generateCandidate` returns null instead, so an unsatisfiable filter of any shape
-- a misspelling, a falsy pin, a predicate matching nothing -- reports through
`generateRows`: it warns, or throws with the tagged message, or obeys
`failOnUnsatisfiableFilters`. The coverage fill asks for its rows with warnings
off, so a fill filter admitting no value is checked before that phase rather than
truncating the matrix in silence.

Diagnostics name what they are about: a filter written as predicates rendered as
`{}`, because `JSON.stringify` drops a function, and a require entry carrying a
tag with no filter rendered as nothing at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewing a fresh integration of this generator (uber/NullAway#1781) turned up
seven misses, six of which the README could have prevented. They are the ones
every integration rediscovers, so document them where they are looked up:

- Reproducibility. `RNG_SEED` and `GITHUB_PR_NUMBER` were documented only in a
  JSDoc comment, so a workflow copied from the README drew a fresh matrix on every
  push and could not replay a failing row. The example workflow wires both and
  takes a seed through `workflow_dispatch`.
- Choosing axes. What makes an axis worth having, four groups to draw from, the
  rule that an axis has to reach the code under test rather than the process that
  launches it, and the rule that the configuration a project ships belongs on axes
  as much as the environment does.
- Ruling out combinations that do not exist. `imply()` states the rule the way you
  know it and `exclude()` states it inside out; a filter matches the axis value as
  declared, so an object-valued axis needs `{scram: {value: 'yes'}}`; nothing
  reports a rule that never fires, and a misshaped `imply()` consequent rejects
  every row its antecedent admits rather than doing nothing; and a rule kept to two
  axes also drops those pairs from the pairwise targets.
- Job count and cost. The budget bounds the rows `generateRows` creates, and rows
  pinned through `generateRow()` are returned whether or not they fit. `weight` is
  the importance of an uncovered pair, so it leans the fill and saturates: the
  section gives the command that measures it on the shipped example and the counts
  it prints.
- Caching. A randomized matrix changes the cache key of any action that hashes the
  matrix into it; `gradle/actions` is the worked example.
- An integration checklist.

`examples/matrix.mjs` gains an `assertions` axis, since `-ea` costs only the jobs
that carry it, and weights on the `os` axis, which the cost section calls for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vlsi
vlsi force-pushed the docs/integration-guidance branch from b5da7a4 to 7bf23a0 Compare August 30, 2026 16:07
@vlsi
vlsi merged commit 3e2b5d1 into main Aug 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant