fix: filter and capture handling in the query engine, the reporter aborts it exposed, and four parser boundaries - #727
Open
awsmadi wants to merge 361 commits into
Conversation
awsmadi
force-pushed
the
pr/filters-captures-and-reporter-silence
branch
from
August 25, 2026 21:04
0316b30 to
e220608
Compare
awsmadi
added a commit
to awsmadi/cloudformation-guard
that referenced
this pull request
Aug 25, 2026
aws-cloudformation#727 gained thirteen commits while this branch was being reviewed, and the two lines both changed `guard/src/rules/eval_tests.rs` and `guard/src/rules/eval.rs`. Merged rather than rebased: rebasing this branch's commits one at a time through a test file that both lines append to conflicts once per commit, and each resolution moves code that the next patch expects to find in place. Two resolutions were needed and neither is mechanical. `eval_tests.rs` keeps this branch's layout, with aws-cloudformation#727's nine added items re-applied as one contiguous slice. Extracting them item by item truncated an attribute block, because a `#[case::...]` attribute wraps across source lines and its continuation matches no attribute pattern, which dropped the `#[rstest::rstest]` above it and left two `#[case]` parameters with no macro to consume them. The whole-slice form carries the attributes with the items. `eval.rs` had aws-cloudformation#727's two new arms answering `Ok(Status::FAIL)` in functions this branch changed to return `Result<Outcome>`. They now answer `Outcome::Unevaluatable` role-free, which is this branch's vocabulary for the same intent: the answer is a value and the consumer applies the role, rather than the producer splitting on `role.is_strict()` as the pre-`Outcome` version did. Three of aws-cloudformation#727's tests fail on this merge and are addressed in the commit that follows, not here, so that the reconciliation is visible as its own change rather than buried in a merge.
awsmadi
added a commit
to awsmadi/cloudformation-guard
that referenced
this pull request
Aug 25, 2026
…data cannot supply The merge took aws-cloudformation#727's production change together with the pre-aws-cloudformation#727 version of the test that change was written to replace, so three cases of a_function_argument_that_selects_nothing_does_not_panic failed on a tree where each side passed alone. 9c7f5a5 reclassified an unsuppliable scalar function argument from ParseError to IncompatibleError. ParseError is the class the evaluator reserves for a malformed rules file and is_unevaluatable does not recognise it, so a -1 in a template field that some rule feeds to substring aborted the run at 255 rather than failing the clause at 19. That reclassification is present at b2bb012. The test read its expectation off the error out of eval_rules_file and asserted the message named the second or third argument, so with the error gone it read the empty string and the three cases failed with "must name the second argument rather than abort; got \"\"". 9c7f5a5 had already rewritten the test in the same commit to assert Status::FAIL through rule_status_in, and said so in its message. The merge kept the merge base's test body character for character, so that rewrite was the only part of 9c7f5a5 that did not land. The test is the side that was wrong. What it asserted, that eval_rules_file returns an error naming the argument, is the behaviour 9c7f5a5 deliberately removed, so keeping it would mean reverting the fix to satisfy a test written against the defect. aws-cloudformation#727's replacement is also the stronger assertion: FAIL covers both the panic the test was written for and the abort that replaced it, and here it pins the one thing the argument's role decides, because to_status sends Outcome::Unevaluatable to FAIL for an assertion and to SKIP for a gate. The doc comment now says that instead of aws-cloudformation#727's wording about there being no error left to read. The Outcome translation of aws-cloudformation#727's two right-hand-side arms in eval_guard_access_clause is not implicated in these failures and is right as it stands. Those arms come from 7dcfc1b rather than 9c7f5a5, and a let-bound function argument does not reach them. Outcome::Unevaluatable is what the clause's own query arm further down already answers for the same error, role-free, with a comment explaining that the split on role.is_strict() moved to to_status and closes_gate at the consumer. Two comments are corrected rather than left standing: the sentence retained from aws-cloudformation#727 said a gate still keeps the error and named the left-hand side as its precedent, which the merge made false in both halves, and the per-arm repetition of the reasoning is dropped now that the block comment above both arms carries it. Verified: 1525 tests pass, 0 fail, 2 ignored across 16 test binaries, from 1522 passing and 3 failing. cargo build --release, clippy --release --all-targets -D warnings, fmt --all --check and typos are clean. All 193 convention-paired rule and test pairs in aws-guard-rules-registry at 6aca96e exit 0. aws-cloudformation#727's reproductions behave as it claimed: a substring bound the data cannot supply and a join separator likewise both move 255 to 19, a sibling rule in the same file still reports its own FAIL and still names the security group open to 0.0.0.0/0 as the one non-compliant resource, and the controls where the argument is supplied are unchanged, at 19 with a violating sibling and 0 without. Across 4,536 rule-by-data fixture pairs under guard/resources/validate against the pinned r720c binary there are 37 exit-code differences and 54 text differences, none of them this commit's: the change to eval.rs is comments only and eval_tests.rs sits behind cfg(test), so the release binary is byte-for-byte equivalent in behaviour to b2bb012. All 37 exit-code differences are attributable to a aws-cloudformation#727 commit, moving 255 to 19 or 0 to 19 in the direction that commit intended: bad_function_arguments and join_with_message to 9c7f5a5, count_empty_collection and count_unresolved to c9ff4d6, embedded_json_the_parser_rejects to b90613b, numbers_that_do_not_fit to 9cde8e5 and regex_replace_no_match to 145e8ad. Twelve of the 17 text-only differences are c9ff4d6 and 9cde8e5 reporting a count or a conversion as unevaluatable where the pinned binary answered zero or saturated to i64::MAX, at the same exit code either way. The remaining five are the reason suffix that 27a86f5 appends to a memoised rule answer, which means r720c predates 27a86f5 and is not the merge's first parent; it matches 4613118, one commit earlier on the same branch.
awsmadi
added a commit
to awsmadi/cloudformation-guard
that referenced
this pull request
Aug 26, 2026
…erence-to-inapplicable question toward aws-cloudformation#727 Two conflicted paths, and eval.rs was not a textual conflict: the two branches held opposite, separately-argued positions on what a reference to a rule that did not apply means. This branch mapped it onto a violation for assertions -- `(Outcome::NotApplicable, _) if role.is_strict() => Outcome::Violated` -- on the ground that "a rule body asserting `r(...)` claims that `r` holds, and a rule that never ran is not evidence that it does", and flagged the resulting change from main as deliberate. That reasoning is true and the conclusion drawn from it is not: the answer to "no evidence" is no verdict, not a violation. Resolved toward aws-cloudformation#727 for two reasons. It produced a false positive. With two helper rules each gated on their own resource type, a template carrying one clean instance of the first type and none of the second reported FAIL at exit 19, and so did a template carrying neither type -- a violation asserted about a resource type the template does not contain. The row that matters for correctness, a violating instance beside a clean sibling, still FAILs, so this is not a slide into disjunction. It contradicted this PR's own algebra. `Outcome::and` has identity `NotApplicable` and absorbing `Violated`, so one level down an inapplicable clause conjoined with a satisfied one yields satisfied. Mapping that same `NotApplicable` onto the absorbing element at the reference site left the type saying "not applicable is neutral" while this arm said "not applicable is fatal", for one input. Whichever way the question is answered, the reference site and `Outcome::and` have to answer it the same way, and only one answer is expressible without carving an exception out of the type. Both sites are resolved, `eval_guard_named_clause` and `eval_parameterized_rule_call`, and both matches stay exhaustive over the enum rather than ending in `_`: that catch-all had been silently covering both a failing dependent and a negated gate on an inapplicable one. The negated `when` gate is unchanged -- a gate that closes silently disables the rule it guards, and the AWS rule registry's one negated reference is a gate. The comments arguing for the old behaviour are removed rather than left beside code that now does the opposite. eval_tests.rs was additive on both sides; all twelve test functions are kept.
awsmadi
added a commit
to awsmadi/cloudformation-guard
that referenced
this pull request
Aug 26, 2026
aws-cloudformation#727 gained one commit: an empty expansion yields an empty selection only when a filter is still pending at or after the point the collection came up empty, so the two flavours of empty selection stop disagreeing. Merged clean textually. Verified behaviourally rather than on that basis, because the previous merge from this branch was also textually clean and had aws-cloudformation#727's Ok/Status::FAIL arms meeting this branch's Result/Outcome return type, which git cannot see.
…the language has
`docs/FUNCTIONS.md` gives this as the example for `parse_char`:
let converted = parse_char(%security_group.Properties.Char)
%converted == '1'
On a template holding `Char: "1"` it failed, for every input, with
`Value='1'] not equal to value [Value="1"]. Error = [PathAwareValues are not comparable char, String]`
-- the two are the same character, and the message blames the template's type.
There is no `Char` literal in the rules language. `parse_scalar_value` is
`alt((parse_string, parse_float, parse_int_value, parse_bool, parse_regex))` and `parse_char` is
reachable from `range_value` only, so `'1'` and `"1"` both parse to `Value::String` and the single-quote
spelling is not a character literal. A `Char` could therefore only be compared with a `RangeChar` or
with another `parse_char` result. `parse_char` was the only one of the five converters whose output
cannot be checked against a literal: `parse_int`, `parse_float`, `parse_boolean` and `parse_string` all
produce a value that matches one, verified one rule each.
Added to `compare_values`, not to `compare_eq`, so ordering and equality get the same answer. A fix in
`compare_eq` alone would have left `%c == "b"` deciding while `%c < "c"` refused, which is the shape of
asymmetry this file has already had to fix twice.
Ordered as strings and not gated on the string's length. Refusing a multi-character string would make
comparability depend on a value rather than on a pair of types, and `<` needs a total order to be worth
having. `'b'` against `"bc"` is Less, which is what those two values would have compared as before
`parse_char` was applied -- and that equivalence is the point, asserted directly: a conversion must not
change the answer to a comparison the language could already make. Byte-wise `str` ordering agrees with
code-point ordering under UTF-8, so this is consistent with the existing `(Char, Char)` arm.
`parse_char` returning a one-character `String` instead was the other candidate and needs no new arms.
Rejected on a measurement: a `String` does not compare with a `RangeChar` either -- `"b" in r[a,z]`
refuses -- so it would have broken `%c in r[a,z]`, which works today and is the other half of what the
function is for. Whether `Char` should be in the value space at all is a real question and a
`String`-against-`RangeChar` arm is what would settle it; that is wider than this finding and is left
alone.
Fourteen matrix cells move, all of them in the `Char` row and column against `string`, and only for the
comparison operators. `%c in "b"` stays refused: `IN` against a string literal is `string_in`'s
substring test, which requires two strings, and that is a different question from ordering.
Why no test caught it: `converters_tests.rs`'s `test_parse_char` asserts the result
`matches!(.., PathAwareValue::Char(_))` and never compares that `Char` to anything.
…and stop swallowing its errors Two defects in one branch of `EqOperation`, the one where both operands are queries. They are not separable: the comparator decides which pairs report an error, and the direction the diff is taken in decides which errors are seen, so changing either alone produces verdicts neither the old code nor the new code produces. Measured -- an intermediate commit that swapped only the comparator fails `numbers_that_fit_still_convert_and_compare`, turning a rule that passes into exit 19, and `test_operator_not_eq`, and both pass before and after. F11, the swallowed error. The branch decided membership with `Vec::contains`, which is `PartialEq`, which returns `bool` and therefore has to turn a comparison it cannot answer into `false` -- `docs/KNOWN_ISSUES.md` records that suppression. With `Num: 1` and `Str: "x"`, `Num == "x"` refused and named `not comparable int, String`, while `Num == Str` failed with no reason at all and `Num != Str` **passed at exit 0 with nothing in the report**. `<` behaved identically in both spellings, which is what isolates it to the two operators routed through `PartialEq` here rather than to query-versus-query generally. The branch now asks `compare_eq_symmetric` per pair and reports what comes back, so both spellings of both polarities agree. The reason is consulted only when the clause is going to fail. A pair that could not be compared on the way to a match found afterwards decided nothing, and a clause whose operand sets do match must not start refusing because some unrelated pairing inside it had no answer. F10, the wrong operand. `eval.rs` files every element of the diff as `from`, which the reporter renders as the clause's subject: its `PropertyPath`, its `Value`, the resource it groups the finding under, and the source excerpt it prints. The diff was taken from whichever operand set was larger, so for every one-value-against-one-value clause -- the ordinary case -- it came from the right. For `Resources.R1.Properties.A == Resources.S.Properties.B` with `A: 1` and `B: 2`, the finding named `/Resources/S/Properties/B`, printed `Value = 2` and `ComparedWith = [2]`, filed itself under `Resource = S` and quoted S's lines. `A` appeared nowhere, and the reason said `B` "was not present in" a set that visibly contained `B`. Worse than the report of it: the resource grouping and the code excerpt follow the same field, so a reader is sent to another resource's block. The diff is now the left operand's unmatched values, qualified by whether the reporter can place them. A rule literal is built with `Path::root()`, so its path is `""`: no resource to group under and no line to quote. When the left operand contributed only literals and the right holds a document value -- which is what `%expected == %replaced` is, `%expected` being a rule parameter -- the document value is the one a reader can act on, so that side is used and `QueryIn::from_rhs` records it. The reporter then compares against the *opposite* operand, which is the other half of the fix: leaving `to` as `qin.rhs` for a diff taken from the right is what produced "B was not present in [B]". Both directions are computed, because `==` between two queries asks whether the operand sets denote the same values and one direction cannot see an extra value on the other side. Choosing by operand-set size reads as set equality and is not: a left operand selecting `[1, 2]` against a right selecting `[1, 1]` are the same length, so only one direction was checked, it was empty, and the clause passed on two operands that plainly differ. That now fails. The negation wrapper's `rhs.len() >= lhs.len() && Eq` special case is gone. It mirrored the size-based choice, so with the choice replaced it would pick the wrong side; reversing against `qin.lhs` is correct in both cases, including the fallback, where every left value has a match on the right and so all of them survive the filter -- which is the honest answer for `!=`. Two recorded outputs change, four lines each, and both replace a false statement with a true one. `failing_complex_rule.out` printed `Value` and `ComparedWith` as the same string and asserted the Arn was absent from a set holding the Arn; it now shows `ComparedWith = ["random_str"]`, the literal that did not match, and keeps its resource grouping and excerpt. `failing_join_show_summary_all.out` said a value was not present in a set containing it; the clause's left operand is a query for a property named `a,b`, which the template does not have, so the set is empty and `[]` is what it is. `test_operator_eq_vs_in_from_queries` asserted `Fail` for a pairing of `Int(10)` against `List([10, 20, 30])` and now asserts `NotComparable`, named on the reason text. That branch compares whole values and does not decompose a list the way the literal-operand branches do, so the pair has no arm; the old verdict was reached by the suppression this commit removes. Twelve matrix cells move, all in the `Char` column where a document value meets a `parse_char` result: `eq` FAIL to REFUSE and `ne` PASS to REFUSE, which is a clause that could not be decided reported as undecided in both polarities rather than silently passing one of them. All 210 registry rules files are byte-identical in report text and exit code, baseline against fixed.
…tside the range Every comparison against a `Float(NaN)` refused except four, and those four decided. `is_within` is `le`/`lt`/`ge`/`gt`, all false for a NaN, and `float_within_int_range` reads `compare_int_to_float`'s `None` through a `_ => false` arm -- so both reported "not in the range", which is an answer. `!=` and `NOT IN` invert an answer, so with a NaN in the document `A not in r[-1,1]` *passed*: a denylist admitted a value that is not a number. Refusing fails both polarities, which is the property that matters. `libyaml/loader.rs` says why NaN is kept out of the value space in the first place -- the type asserts `Eq` and hashes its own contents, and a NaN is not equal to itself -- and its `is_finite` gate is why `validate` cannot reach this. The `serde_yaml` conversion has no such gate, and that is what `cfn-guard test` and the public library entry point read a document through, so a NaN is reachable and the comparison layer has to hold on its own. Probed through `cfn-guard test`: before, eight operators against a NaN, a number and a range gave three passes; after, the only remaining pass is `NOT IN` between two queries, which is the suppression `docs/KNOWN_ISSUES.md` tracks and `contained_in` deliberately keeps, and it emits its deprecation notice. Only the scalar is tested for NaN. A range's bounds come from `parse_range`, which builds them with `parse_float`, and that rejects a literal that is not finite -- so a `RangeFloat` cannot hold a NaN bound and a guard for one would be unreachable. Said in the comment rather than guarded. `PartialEq` is deliberately not touched, and `Float(NaN) == Float(NaN)` stays false. Making it true would restore reflexivity for `Eq` and would put the two equality entry points back into disagreement -- `compare_eq` and `compare_values` both refuse the pair -- which is the defect shape F1 was. The reflexivity hole closes by keeping NaN out of the value space, which is the conversion's job; this commit makes sure that until it does, nothing answers. No matrix cell moves, and that is the expected result rather than a weak one: all 1,819 cells go through `validate`, whose loader gates the value, so the fix is unreachable from there by construction.
…ocument `Loader::load` returned at the first `DocumentEnd`, so a file holding a `---` stream was answered with its first document and nothing said so. Prefixing a template with a compliant document and a `---` suppressed every finding in the real one at exit 0, with both channels empty -- the shape a reviewer cannot catch, because there is no output to read. Returning there also dropped the parser, so the bytes after the first document were never handed to libyaml: a stream whose later document was not YAML at all passed too. The `documents: Vec<MarkedValue>` field and its push/pop pair were the residue of a multi-document design that arm never implemented; the vector never held more than one element. Both are gone, replaced by a local `Option`. User-visible change: a data file holding more than one YAML document is now refused with a message naming the position of the second document, at exit 255, where it used to be evaluated as its first document alone. Two other files change verdict for the same reason, and both are fixes. A file whose first document is empty (`---` then `---`) used to evaluate nothing and report the empty document's verdict. A file with content after an explicit `...` with no new `---` is not well-formed YAML -- libyaml says "expected <document start>" -- and used to be accepted because the loader returned before reading that far. Evaluating every document was the alternative, and it is a feature rather than a bug fix: `DataFile` carries one `path_value`, and every reporter, the summary and the exit code are written against one document per file, so it needs the reporters to name which document a finding came from and a rule for combining n verdicts into one exit code. Refusing is contained in the loader and cannot report compliance for bytes it did not read, which is the defect. Continuing to evaluate the first silently was not on the table. Blast radius measured: no file in the rules registry snapshot (257 data files) or this repository (122) holds more than one document, counted by parsing each file with a tag-tolerant YAML loader. The 196 registry files that contain a `---` on a line of their own are `###` comment banners followed by a directives-end marker, which opens the first document rather than separating two; `one_document_with_a_ marker_or_a_banner_still_loads` pins that so a later change cannot regress it by counting `---` lines instead of `DocumentStart` events. The message reaches the user through a new `Error::UnsupportedDocument`, which `build_data_file` passes through rather than replacing with the file's first hundred bytes. That substitution exists for libyaml's own failures, which all read "error parsing file" and need the bytes to be useful at all; attaching it to a message that already names the construct and its position only buries it. The FFI error table gets code 23 for the new variant -- it is an exhaustive match, so the compiler required it.
…them does
`system_mark_to_location` passed libyaml's marks through unchanged, and libyaml
counts from zero. So every `Path=[L:n,C:m]` in every data-file finding named the
line above the one a person editing the file sees, and a column one to the left.
The report contradicted itself inside a single block. `validate::cfn`'s
`emit_code` prints a one-based excerpt of the file beside the `L:` value, and
`context_start_line` says so in as many words -- "never below line 1 because line
numbers are 1-based". On `s3-public-read-prohibited-template-non-compliant.yaml`
the `Properties` mapping begins at physical line 14, column 7; the finding read
`L:13,C:6` and printed an excerpt of lines 11 to 16, so the reader was pointed at
a commented-out line rather than at `BucketEncryption:`. Both halves are now
right: the marker says 14,7 and the excerpt runs 12 to 17.
Rules-file locations did not share the convention and did not need changing --
they come from `nom`'s `LocatedSpan`, whose `location_line()` and
`get_utf8_column()` are already one-based. The product was printing one
convention for the rules file it read and another for the data file.
`Location::default()` stays `{0, 0}` and keeps meaning "no position", which is why
the offset is added in the conversion and not in `Display`: it is what a literal
written in the rules file is given, and it is what SARIF's `build_region` reads
`line < 1` to recognise. Literals still print `[L:0,C:0]`, as the fixtures show.
SARIF gains a `startColumn` on two findings that had none. `build_region` drops a
column below 1, so a value at the old column 0 lost its column entirely; at the
real column 1 it is emitted. That is the second half of the same defect: the
field's schema minimum is 1 and the value being fed to it was zero-based.
User-visible change: every data-file line and column in every output format moves
by one, and the excerpt window moves with it.
Expected-output fixtures were regenerated from the binary rather than edited by
hand, because a literal and a document root both printed `[L:0,C:0]` and no
regex can tell them apart. Verified afterwards by masking every digit run in each
fixture and diffing: eleven of twelve files differ only in numbers, and the
twelfth -- structured.sarif -- differs only by the added `startColumn`. Every
numeric change is +1 except the literals, which are unchanged. The remaining
structural differences are the excerpt window sliding one line, which drops the
first line of the old window and adds one at the end.
The 256 `Location` literals in `loader_tests.rs` are all loader-derived
expectations, so all of them move by one; the file contains no rules-file
literal. Two new tests pin the convention itself rather than the numbers:
`a_marker_names_the_one_based_position_of_its_scalar` computes its expectation by
searching the fixture text, and `a_location_with_no_position_stays_zero` pins the
default.
Not updated, deliberately: `guard/ts-lib/__tests__/__snapshots__/validate.spec.ts.snap`
still holds the old markers. `guard_bg.wasm` and `guard.js` are committed
artifacts and the Typescript CI job neither rebuilds them nor runs on a change
outside `guard/ts-lib/**`, so jest checks the snapshot against the committed wasm,
not against this source. Editing the snapshot now would make it disagree with the
artifact actually under test. It needs regenerating when the wasm is next rebuilt.
The null set was `~` and `null` only, so the *empty* scalar -- the value of a key
written with nothing after the colon -- fell through to `String("")`. Both the
YAML 1.1 and 1.2 schemas resolve that to the null node, so the loader could not
tell `k:` from `k: ""`, which YAML says are null and the empty string.
Writing the same document with the null spelled out inverted all three of `== ""`,
`is_string` and `is_null`, which is what makes this a defect rather than a house
choice: two spellings of one value disagreed. The wrong-pass direction is an
`is_string` or `!= null` clause on a property that is actually absent-valued.
`serde_yaml`, which is the loader `guard test` and `run_checks` reach on the same
bytes, already answered null here, so this also removes one of the divergences
between the two.
Deciding on emptiness alone is safe because every quoted scalar is taken by the
`style != Plain` arm before this one: `k: ""` and `k: ''` are still the empty
string. That is the control in
`an_empty_plain_scalar_is_null_and_a_quoted_one_is_not`.
User-visible change: a key with nothing after the colon is null. `exists` still
answers true for it -- the key is present -- and `is_null` now answers true where
it used to answer false.
Blast radius: two expected-output fixtures, three lines between them, and no
verdict, exit code, path or marker changes. `failing_template_with_slash_in_key.yaml`
and `s3-server-side-encryption-template-non-compliant-2.yaml` both have a
`BucketEncryption:` with nothing after the colon, and their diagnostics now read
`Type = null, Value = Null(...)` where they read `Type = String, Value = ""`. Both
fixtures encoded the old reading in their text; the new text says what the document
actually holds.
A plain scalar was resolved against YAML 1.1's set of 22 spellings, which adds
`y`, `Y`, `n`, `N`, `yes`, `no`, `on`, `off` and their casings to the six the 1.2
core schema defines. Three other readers of the same document disagreed with that,
and each disagreement is a way for one file to mean two things:
- `rules::parser::parse_bool`, which reads a boolean literal in cfn-guard's own
grammar, accepts exactly `true|True|TRUE|false|False|FALSE`. So `Enabled ==
true` in a rule and `Enabled: yes` in a document were compared across two
vocabularies.
- `serde_yaml`, which `guard test` and the public `run_checks` reach on the same
bytes, is 1.2 core. A rule proved correct under `guard test` could fail under
`validate`.
- libyaml, whose parser this file wraps, does not resolve the single letters;
nor does PyYAML, the other widely used 1.1 implementation. The 1.1 type page
lists them, the implementations do not.
So the set is now 1.2 core, which makes all four agree.
Two concrete harms go away. `AttributeType: N` is what the DynamoDB documentation
shows for a numeric attribute, unquoted; it resolved to `false`, so
`AttributeType IN ["S","N","B"]` reported a false FAIL, and the same value inside
a filter selected nothing and skipped the rule at exit 0 with only a stderr note.
Separately, every GitHub Actions workflow in this repository was unreadable:
`on:` resolved to a boolean, a boolean is not a valid mapping key, and the file
was refused at exit 255. Both now load and evaluate correctly.
User-visible change, and it cuts both ways: a document writing `Encrypted: yes`
meaning true now yields the string "yes", and a clause comparing it to `true`
reports the type mismatch instead of passing. That is the trade. It is the right
way round, because the tool now says it cannot decide where it used to decide by a
rule none of its other readers shared.
Blast radius: no `.guard` file in the rules registry snapshot or this repository
compares against a bare 1.1-only spelling, and no data fixture in either writes one
as a value -- checked by scanning every `.guard` for a comparison operator followed
by one of the 16 spellings, and every `.yaml`/`.yml`/`.json`/`.template` for one as
an unquoted value. Both counts are zero.
Three tests asserted the old set and are changed. `test_handle_bool_happy_path`
and `an_explicitly_tagged_bool_reads_the_same_set_as_a_plain_one` keep the six
spellings; the 1.1-only ones move to
`a_scalar_outside_the_boolean_set_is_still_a_string`, which asserts they are
strings. `test_a_gate_on_a_boolean_fires_for_every_spelling_yaml_makes_boolean`
keeps the six, and its 1.1-only cases move to
`test_a_non_boolean_string_is_still_not_comparable_to_a_boolean`. They could not
stay in the gate test with the expectation flipped to SUCCESS: `no` and `off`
exited 0 before because the gate evaluated *false* and exit 0 after because the
comparison is *undecidable*, so a case asserting only the exit code would pass
without naming its reason. The companion test asserts the reason.
…s text
Integer resolution was `str::parse::<i64>`, which accepts an optional sign and
decimal digits and nothing else. Two consequences:
- No radix prefix resolved as a number. `0x1F` and `0o17` were strings, so a
rule comparing a netmask or a permission bitmask to a number could not match,
and the short and long spellings of the same value disagreed.
- `0755` was read as decimal 755. A file mode or a netmask written that way
almost certainly means octal 493, so the loader produced a number the author
did not write.
Resolution is now the YAML 1.2 core schema's three forms -- `[-+]?[0-9]+`,
`0o[0-7]+`, `0x[0-9a-fA-F]+` -- with two departures, both stated in the doc
comment on `resolve_int`:
A leading sign is accepted on the radix forms, so `-0x10` is -16. The 1.2 regexes
carry no sign, but no YAML version reads that text as a different number and
`serde_yaml` resolves it, so accepting it removes a divergence and cannot
introduce a wrong value.
A decimal integer with a redundant leading zero stays a string. This is the one
spelling where the two versions assign different *values* to the same characters:
1.1 reads `0755` as octal 493, 1.2 core's decimal regex reads it as 755. Keeping
the literal is the only answer that cannot be quietly wrong, and it is what
`serde_yaml` already does. The string arm also has to short-circuit the float
resolver, because `"0755".parse::<f64>()` is 755.0 and falling through would swap
one wrong number for another.
Prefixes are lowercase only, which is 1.2 core's regex exactly and what
`serde_yaml` does, so `0X1F` and `0O17` remain strings. One divergence from
`serde_yaml` is left standing deliberately: `0b101` is a string here and an integer
there. YAML 1.2 core has no binary form -- it is 1.1's -- and following the
extension would mean re-adding a 1.1-ism of the kind the boolean set just dropped.
User-visible change: `0x…` and `0o…` become integers; `0755`, `00` and `+0755`
become the strings they are written as, where `0755` used to be the integer 755.
Blast radius: no data fixture in the rules registry snapshot or this repository
writes an unquoted value matching `0[0-9]+`, `0x…` or `0o…`, so nothing in either
corpus is affected. Whole suite green with no fixture regenerated.
…loat `i64` first, then `f64` if the value was finite, and no arm in between. So an integer literal past `i64::MAX` landed in `MarkedValue::Float`, and two integers the document spells differently became one value: `9223372036854775809` and `9223372036854775810` both loaded as `9.223372036854776e18` and compared *equal*, so a clause asserting they differ was reported non-compliant. A wrong answer about identity, at the ordinary exit code, with nothing on either channel to notice it by. The literal is kept instead of being converted. `MarkedValue::Int` is an `i64`, so there is no arm here that can hold the value; widening it reaches `PathAwareValue::Int` and from there every comparison operator, the serializers and SARIF, which is a change of a different size and shape from this one. `serde_yaml` holds `i64::MAX + 1` as an integer and refuses anything past `u64`, so keeping the text agrees with neither of its answers. That divergence is accepted deliberately: of the options available inside an `i64` value model, this is the only one that never reports two distinct integers as the same value. User-visible change: an integer outside `i64` is the string it was written as, so `is_int` and `is_float` both answer false and a comparison against a number reports the type mismatch. `i64::MAX` and `i64::MIN` themselves are unaffected, which `an_integer_wider_than_i64_keeps_its_text` pins at both boundaries -- an off-by-one in the range check would either leave the defect standing or turn `i64::MAX` into text. Blast radius: no data fixture in the rules registry snapshot or this repository contains an integer of 19 or more digits. Whole suite green with no fixture regenerated.
…gs produce
CloudFormation documents `!GetAtt logicalNameOfResource.attributeName` as the short
form of `{ "Fn::GetAtt": [ "logicalNameOfResource", "attributeName" ] }`, and the
dotted spelling is YAML-only -- JSON has the list and nothing else. Nothing split on
the dot, so `!GetAtt SecretResource.Arn` became a *string* while
`!GetAtt [SecretResource, Arn]` and the long form both became a *list*. One
reference had two incompatible shapes depending on how the template happened to be
written.
The consequence was a silent wrong SKIP. A filter reaching `"Fn::GetAtt"[0]`
matched the list forms and selected nothing on the dotted one, so the rule was
skipped and the run exited 0 -- and unlike the boolean case, with an empty stderr,
because the filter fails by path traversal rather than by an undecidable comparison.
`!GetAtt Resource.Attr` is the ordinary way to write a GetAtt in YAML, so a rule
authored against JSON templates silently declined to check YAML ones.
The split is on the first dot only, because the attribute name may contain dots.
AWS's own example is `!GetAtt myELB.SourceSecurityGroup.OwnerAlias`, which it gives
as `["myELB", "SourceSecurityGroup.OwnerAlias"]`. Source:
https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-getatt.html
A payload with no dot keeps its string. It is not a valid `Fn::GetAtt` -- the
function takes a resource and an attribute -- and a one-element list is a shape
neither the long form nor JSON can produce.
Both halves carry the location of the scalar they were written as, which is where
they are. Deriving a column for the second half would mean counting from a start
mark whose relationship to the tag is not established here, and an invented column
is the defect the marker change just removed.
User-visible change: `Resources.X.Properties.P."Fn::GetAtt"` is a two-element list
for the dotted form where it used to be a dotted string, so a rule comparing it to
`"Resource.Attr"` no longer matches and one indexing `[0]`/`[1]` now does.
Blast radius: the registry snapshot contains exactly one `Fn::` reference in 210
rules files, `Fn::GetAtt` in
rules/aws/cloudfront/cloudfront_origin_access_identity_enabled.guard, and it does
not index into the payload. Whole suite green with no fixture regenerated.
… than fatal `PathAwareValue::try_from_marked` recurses once per level of nesting and had no bound, so a deep enough document killed the process. Measured on this repository's release build: depth 5281 converts, depth 5375 aborts with SIGABRT and "thread 'main' has overflowed its stack" on stderr -- no diagnostic from cfn-guard, and an exit code outside the set the tool documents. A 40 KB file of nothing but brackets was enough. Depth was also expensive long before it was fatal, because every node rebuilds its full path string from its parent's, so the bytes allocated grow with the square of the depth. Depth 800 took 4.9 seconds, depth 1600 took 39, depth 2000 took 76, and 14 KB of 600-deep mappings with 20-character keys took 43. Bounding the depth bounds that too, which is why this is one change and not two. Where the check goes was decided by measurement, not by reading. A throwaway test showed `Loader::load` returns successfully at depth 20000, and that dropping the `MarkedValue` it returns also survives -- the loop is iterative and the recursive `Drop` does not die. Only the conversion overflowed. So the bound belongs where the deep value is *built*: refusing in `load` means no deep `MarkedValue` is ever constructed for anything downstream to recurse over, which a check inside the conversion would not give. 128 is the recursion limit serde already enforces on the other loader in this product, and at this value the two agree level for level. On files of `a: ` followed by n brackets, `validate` and `rulegen` both accept n = 127 and both refuse n = 128, the second with "recursion limit exceeded". So no document this loader now refuses could have reached `run_checks` or `guard test` either. It is also far above anything real: the deepest data file in the rules registry snapshot is 15 levels and the deepest in this repository is 24. A non-recursive conversion was the alternative. It removes the crash but not the quadratic path-string cost, which is inherent to storing a full path at every node and reaches into `Path`, `PathAwareValue` and every reporter that prints one. A bound fixes both, inside the loader, and leaves that refactor free to happen on its own terms. User-visible change: a document nested more than 128 containers deep is refused with a message naming the level and the position, at exit 255, where it used to be evaluated slowly or abort the process. Measured after: depth 8000 and depth 20000 both return in 0.01 seconds instead of crashing after four; depth 127 costs 0.03 seconds. The boundary cases are pinned at 126, 127 and 128 levels, because an off-by-one there would either refuse a document inside the limit or admit one past it, and neither shows in the deep cases. A 2000-element flat sequence is the control that says the bound is on nesting and not on size.
The refusal carried `val.location().to_string()` and nothing else. In a run over a
directory of templates that reports a position in an unnamed file: `L:2,C:4`
identifies neither which key nor which of N templates to open, and a reader cannot
search for it either. The message now names the key's type and value, says to quote
it, and `build_data_file` prepends the file name, which is the only place that knows
it.
Before:
non string type detected for key in a map at L:2,C:4, cfn-guard only supports
keys that are string types
After:
in data file .../dirprobe/bbb-intkey.yaml: non string type detected for key in a
map at L:2,C:3, where the key is the integer 80. Quote it to make it a string,
cfn-guard only supports keys that are string types
The file name reaches the multi-document message from the previous commit by the same
route, which it also needed.
Two parts of this finding are deliberately not fixed, both for reasons that are not
about effort:
**Non-string keys are still refused rather than stringified.** CloudFormation converts
a template to JSON and stringifies mapping keys, so `80: http` is a template it accepts
and cfn-guard does not -- and the same content in JSON, `{"80": "http"}`, already loads
here. Accepting it looks like a small change and is not: `handle_mapping_end` receives
values the loader has already resolved, and `MarkedValue` does not carry the text the
document wrote. Rendering the value back gives "31" for a key written `0x1F`, "1" for
`1.0`, and "true" for `True`, so the key name would differ from the one in the file.
That is a silent wrong answer of exactly the kind the rest of this branch removes, and
it is worse than a loud refusal. Doing it correctly means carrying each scalar's
original text through the value model, which is a change to the value model rather than
to the loader.
**One unreadable file still ends the run.** `read_data_file` is called with `?` inside
the directory walk, so a single bad file discards the results of every good one. That is
`validate.rs` control flow, not loader behaviour, and changing it needs a decision this
commit should not make on its own: what a run reports and what it exits with when some
of its files were readable and others were not.
`<<` is YAML's merge key (https://yaml.org/type/merge.html): its value is a mapping, or a sequence of mappings, whose keys belong to the mapping that carries it. Nothing resolved it, so `<<` became an ordinary key of that name and every key underneath was hidden. libyaml is a parser rather than a composer, so nothing upstream resolved it either. On the shape essentially every real rule uses, that was a silent wrong SKIP. A template whose `Type` arrives through a merge was invisible to `Resources[ Type == "AWS::S3::Bucket" ]`: the filter selected nothing, the rule was skipped, and a bucket with no `PublicAccessBlockConfiguration` exited 0 unchecked. Writing the same `Type` inline made the identical file exit 19 and FAIL. The two now agree. Precedence follows the spec. A key the mapping writes for itself always wins over a merged one, which is why the fold runs after every explicit key is in -- and it wins whether it is written before or after the `<<`, which both directions are pinned for. Within a sequence of mappings an earlier entry wins over a later one. Two `<<` keys in one mapping are not something the spec defines; earlier wins, by the same rule. A `<<` given anything else is refused with a message naming the position and quoting the requirement, rather than folded into nothing. Only the inline spellings are reachable, and that is worth being clear about: `<<: *base` is how a merge key is usually written, it contains an alias, and `Loader::load` refuses every alias before the fold runs. So this closes the spelling that was quietly misread and leaves the common one loudly refused. That refusal now says so. The alias arm returned a `ParseError`, which `build_data_file` replaced with the file's first hundred bytes, so the one message that tells a user what to change was discarded and an aliased file reported only "Error encountered while parsing data file". It is an `UnsupportedDocument` now, names the position, and mentions that `<<: *anchor` is an alias while `<<: { ... }` is supported -- without which a user hitting the merge fix by the usual spelling gets a diagnostic that does not connect to what they wrote. `yaml_loader_with_alias` asserted only `is_err`, which any failure satisfied including the discarded one; it now asserts the variant and the wording. Blast radius: `git grep '<<:'` finds no data file in the rules registry snapshot or this repository, so no template in either changes. Whole suite green with no fixture regenerated.
… on a mapping
A `!Foo` tag was checked against two hand-written name sets and, on a miss, thrown
away -- the payload survived and the function did not. So the short form of any
intrinsic the sets did not list became something else entirely: `!Transform { ... }`
was indistinguishable from a plain mapping, and a rule forbidding the macro passed at
exit 0 on the very template whose long `Fn::Transform` spelling failed. That is the
opposite of how `!!`-tags behave, where `!!int abc` becomes a `BadValue` and is
reported, so a bad type tag was loud while an unknown function tag was silent.
The sets had gone stale, which is why enumeration was the wrong mechanism rather than
merely an incomplete list. Cidr, ForEach, GetStackOutput, Length, ToJsonString,
Transform and ValueOfAll are all in the CloudFormation Template Reference and were in
neither set. They also created a position trap: `GetAtt` was in both, `GetAZs` in the
scalar set only and `Select` in the sequence set only, so a name the loader did know
lost its tag when used in the other position.
`long_form_of` replaces the gate with CloudFormation's own rule -- `!Foo` is the short
form of `Fn::Foo`, with `Ref` and `Condition` the two documented names that carry no
prefix -- so every `!Foo` is preserved and a name AWS publishes after this commit needs
no change here. It replaces `short_form_to_long`, whose miss arm was `unreachable!()`
and was only unreachable because every caller consulted a set first; removing the gates
without removing that arm would have turned a stale list into a panic. Sources:
https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-rules.html
A tagged *mapping* is a third position no set could have covered: `handle_mapping_start`
never looked at the tag at all, so `!ToJsonString { a: 1 }` and `!Transform { Name: ... }`
lost theirs even though one of those names was in a set. Mappings now take the same
wrapper sequences already did, and `close_tagged_container` is the shared close.
A bare `!` is skipped. It is YAML's non-specific tag rather than a function name, and
the `Fn::` fallback would otherwise name the key "Fn::".
The serde-backed loader in `values.rs` moves with it, because leaving it behind is how
the divergence this branch has been closing gets recreated: `guard test` and the public
`run_checks` read through `handle_tagged_value`, so a rule proved correct under
`guard test` would behave differently under `validate` on the same bytes. That function
also gains the dotted-`GetAtt` split the libyaml loader received earlier, for the same
reason.
User-visible change: `!Foo payload` is `{"Fn::Foo": payload}` for every `Foo`, where an
unlisted name used to yield the payload alone, and a tagged mapping used to yield the
mapping alone.
`test_in_comparison_operator_for_list_of_lists` asserted the old dotted `Fn::GetAtt`
shape in its rule literal -- `{"Fn::GetAtt": "Master.PrivateIp"}` against a template
writing `!GetAtt 'Master.PrivateIp'`. CloudFormation gives that as the two-element list
and JSON has no other shape for it, so the literal was encoding the defect; it is a list
now. Quoting the payload does not change the semantics, which is why the single-quoted
spelling still splits.
Blast radius: the registry snapshot has exactly one `Fn::` reference across 210 rules
files, `Fn::GetAtt` in cloudfront_origin_access_identity_enabled.guard, so no registry
rule keys on any of the newly preserved names. Structural edge cases re-checked for
panics: a tagged sequence, scalar and mapping at the document root, an empty tagged
sequence, a tagged sequence of tagged sequences, and the nested
`!Join`/`!Split`/`!Sub`/`!If` battery all load without a panic or a 255.
cfn-guard has two loaders. `values::read_from`, the libyaml one, is reached only by `validate`'s `build_data_file`; everything else goes through serde -- `commands::helper::validate_and_return_json`, which is the public `run_checks`; the `test` command's spec `input:` blocks; and `rulegen`. Nothing held them to the same answer, and they gave fifteen of twenty-eight probed scalar spellings different ones. That made every other typing question path-dependent. `rulegen` emitted a rule that `validate` then rejected on the template it was generated from, because one read `AttributeType: N` as the string "N" and the other as `false`. A rule whose `guard test` suite was green failed under `validate` on byte-identical input -- so the harness a rule author uses to prove a rule correct did not exercise the loader the rule would run against. And a regression in either loader was invisible to the other's tests. The preceding commits closed the divergences one at a time: the YAML 1.1-only booleans, hex and `0o` integers, the leading-zero decimal, the empty scalar, the dotted `!GetAtt`, and the discarded `!Foo` tag. This is the test that keeps them closed, over 29 spellings in one document, compared through `serde_json` because that is the only form both loaders reach -- the libyaml value carries source locations and the serde one has none. Two divergences are deliberately outside the document, each with its reason recorded where the decision was made. `0b101` is an integer to `serde_yaml` and a string here, because YAML 1.2 core has no binary form and following the extension would re-add a 1.1-ism the boolean set just dropped. An integer wider than `i64` is a string here and an integer there, because `MarkedValue::Int` cannot hold it and widening it reaches every comparison operator. Checked for sensitivity rather than assumed: adding `0b101` to the document fails the test with a diff naming exactly that key, so it is not passing by comparing nothing.
`f64::from_str` signals overflow by returning an infinity and underflow by
returning zero without saying so. The `is_finite` gate caught the first and not
the second, so `1e400` fell through to a string while `1e-400` became
`Float(0.0)`: a value the document writes as a small positive number answered
`== 0` with a PASS and `> 0` with a FAIL, at exit 0, with nothing on either
channel. One rule disagreed with itself at the two ends of the same exponent
range.
It also made a comment in `rules::parser` false. That comment explains why the
rules parser rejects both saturating and underflowing float literals while the
document side does not, on the grounds that "there, `loader.rs` retypes the scalar
to a string, which leaves every comparison against it incomparable and therefore
failing closed". That was true of the overflow half only. Both halves now do it.
The discriminator is the mantissa, not the whole literal. A zero result is genuine
when every significant digit is zero and an underflow when one of them is not, and
the exponent has to be excluded from the test or `0e400` -- a real zero -- would be
read as an underflow on account of the `4`. Eight zero spellings are pinned as
controls for that, including `0e400` and `0.0e-400`.
A small float that is still representable is unaffected, which
`a_small_float_that_is_still_representable_stays_a_float` pins with a subnormal:
without it the change is also satisfied by refusing every negative exponent.
User-visible change: a float literal too small to represent is the string it was
written as, so `is_float` answers false and a comparison against a number reports
the type mismatch instead of answering as though the author had written zero.
Not fixed on the serde-backed conversion, and the reason is not effort.
`serde_yaml::Number` is `enum N { PosInt(u64), NegInt(i64), Float(f64) }` and
retains no source text -- verified against the pinned crate source, 0.9.34 -- so an
underflowed `1e-400` arrives there as `Float(0.0)`, indistinguishable from a
literal `0`. The divergence is recorded in
`both_loaders_resolve_the_same_document_to_the_same_value` with that reason.
Blast radius: no data fixture in the rules registry snapshot or this repository
contains a float literal outside f64's range. Whole suite green with no fixture
regenerated.
…, in the serde conversion
Two defects in one `serde_yaml::Value::Number` arm, both reached from
`PathAwareValue::try_from(serde_yaml::Value)` -- which is what the `test`
subcommand and the public `run_checks` read documents with.
The sign inversion first, because it is the worse of the two. The code read
`num.as_u64().unwrap() as i64` under a comment saying "Yes we are losing precision
here. TODO fix this". It was not losing precision: `as i64` reinterprets the bit
pattern, so the sign flipped. `u64::MAX` read as exactly -1 and `i64::MAX + 1` as
exactly `i64::MIN`. Every numeric guard in the language inverts for such a value --
measured, not inferred: `A < 0` PASSed, `A == -1` PASSed, and `MaxSize <= 1000`
PASSed for an input of 18446744073709551615, all at exit 0 with nothing on either
channel. `compare_values` was blameless; it was handed the wrong integer.
The digits are kept instead. Three alternatives, and what each costs:
- **Error on a u64 above `i64::MAX`.** Smallest, and it refuses rather than lying,
but it refuses the whole *document*: one oversized number and every other rule
in the file stops being checked. Failing closed on the one value is less than
that.
- **Widen `Value` with an unsigned variant.** Correct in principle, and much
larger than it looks. `Value` is matched exhaustively in the rules parser, both
conversions, `Display` and `Hash`; and for the exactness to reach a comparison
at all, `PathAwareValue::Int` needs the same widening, which lands in every
`compare_*` function -- another change's territory. It would put an exact value
in a type nothing yet compares against.
- **Store it as `f64`.** Trades a sign error for a precision error rather than
removing one: `u64::MAX` becomes 18446744073709552000, so `== 18446744073709551615`
fails and `> 18446744073709551000` passes. Still a wrong answer, only quieter.
Keeping the digits is exact -- `u64::to_string` invents nothing -- it makes a
comparison against a number refuse rather than answer from a number the input does
not contain, and it is the answer the libyaml loader already gives an integer this
wide. That last point is why this and the loader's out-of-i64 handling were decided
together: they are the same defect on two loaders, and they now agree.
The float half is an `Eq` violation that was real rather than theoretical.
`PathAwareValue` asserts `Eq` and `Float(NaN)` is not equal to itself, so
`A == A` reported FAIL through this conversion on a document that PASSed through the
libyaml one. `.inf` was worse than inert: `A > 9223372036854775807` PASSed under
`test` where the same document refused under `validate`. The libyaml loader has
carried the finiteness gate all along, with a comment stating the invariant; this
conversion never had it.
Non-finite values become YAML's own spelling -- `.nan`, `.inf`, `-.inf` -- rather
than `f64::to_string`'s "NaN" and "inf", which no YAML document writes. `serde_yaml`
resolves a float only from those three spellings, bare `NaN` and `inf` staying
strings in both loaders, so the canonical spelling makes the two agree exactly on
every input that can reach the function rather than approximately.
User-visible change, on `guard test` and `run_checks` only: an integer above
`i64::MAX` and a non-finite float are strings, so `is_int`/`is_float` answer false
and comparisons against numbers report the type mismatch.
Both divergences were on the documented exclusion list of
`both_loaders_resolve_the_same_document_to_the_same_value`; they are now in its
document, which covers 34 spellings. `0b101` and the underflowing float remain
excluded, each with its reason.
…e it
Any key that was not a `String` was refused, which refused templates
CloudFormation accepts. A template is converted to JSON before it is deployed and
JSON has no key but a string, so
Mappings:
AccountToEnv:
123456789012:
Env: prod
-- an account id written the way a person writes one -- aborted the run at exit
255, while the same content written in JSON, `{"123456789012": {...}}`, already
loaded. `path_value::list_index_of`'s doc comment presents this as fixed and names
account ids, ports and status codes as the cases; the retrieval half was fixed, and
the template half was unreachable, because a document writing the key the natural
way never got as far as retrieval. It does now.
**This reverses the reasoning I recorded two commits ago**, when the diagnostic for
this refusal was improved. That reasoning was that rendering a resolved value
"invents a name the document does not contain", since an `Int` from `0x1F` renders
as "31". The premise was wrong. CloudFormation resolves `0x1F` as 31 by the same
YAML 1.2 core schema this loader implements and then stringifies it for JSON, so
"31" is exactly the key CloudFormation sees. Rendering the resolved value models the
deployment; rendering the source text would not.
Checked against a YAML-to-JSON round trip rather than asserted -- `yaml.safe_load`
then `json.dumps` gives `{"80":…, "1.0":…, "1.5":…, "31":…}` for the same keys, and
the loader now produces the same five names. A whole float is formatted with its
fractional part, so `1.0` is "1.0" and not Rust's "1", because "1.0" is what that
round trip produces.
`Null` is deliberately not in the set. There is no text the two conventions agree on
-- Python's round trip gives "null", JSON has no such key -- and a document writing
`~:` is far more likely to have lost a key than to want one named after nothing, so
the refusal is the more useful answer. Container and `BadValue` keys stay refused for
the stronger reason that they have no scalar text at all.
Most of this finding's blast radius had already gone: of the ten bool-shaped keys
that aborted the run, `y n Y N on off yes no` were fixed by moving to the YAML 1.2
core boolean set, leaving `True` and `Null`. `True` is fixed here.
Two tests asserted the old behaviour and are changed.
`a_non_string_key_is_refused_with_the_key_named` keeps the keys that have no text and
loses the integer, float and boolean cases to a new
`a_scalar_key_becomes_the_text_cloudformation_would_give_it`, which pins the text each
one produces. `test_graceful_handling_when_yaml_file_has_non_string_type_key` loses
`1: foo` and `1.0: foo` for the same reason, keeping the null key, the sequence key,
and the document that does not parse.
One case in the first test was mine and wrong: `": x"` is not valid YAML -- libyaml
refuses it while parsing a block mapping, so it never reaches the key check. The only
empty-key spelling YAML accepts is the explicit `?` indicator, and it resolves to a
null key, which is the case already covered.
The refusal said "more than one", which leaves the reader to find out whether they have two documents or twenty before they can act on it. It now says the number. Counting means draining the rest of the stream, and the rest of the stream may not parse -- a file whose *later* document is not YAML is one of the shapes this refusal exists for. So the count degrades to a lower bound, "at least 2", when libyaml stops early. That is better than either abandoning the count or letting the parse failure propagate, which would replace a message about document structure with a syntax error from further down a file the user is about to split anyway.
…lure A data file cfn-guard could not read reached `main`'s catch-all, which prints "Error occurred" and exits -1 -- the code `guard/tests/utils.rs` names `INTERNAL_FAILURE`. So a template with an unquoted `~` key, an empty data file, a file that is not YAML, and a multi-document stream all told the user cfn-guard had broken, when each is their own input. `ERROR_STATUS_CODE` is 5, which this repository already gives a ruleset it cannot use. Two commits before this branch moved rules-file mistakes to it for exactly this reason, and their wording is the argument here too: 5 is not 19, so a CI gate still tells a broken input from a violation, and unlike -1 it does not additionally claim the tool is at fault. `guard-ffi`'s error table had already reached the same judgement independently, mapping `ParseError` to its code 5. `ParseError` is the discriminator, and it is safe to classify on because every one of its producers is about the user's input: the rules parser and its let-cycle checks, the libyaml loader, the test-spec readers, the payload deserializer, the data-file reader. Enumerated with `git grep`, which is the only search tool immune to the NUL bytes in `parser_tests.rs`. None of the twenty-odd sites is an internal failure, so there is no arm this over-catches. A *missing* file keeps -1, deliberately. `File::open` and `validate_path` return `IoError` and `FileNotFoundError`, not `ParseError`, so they are untouched -- the same line the parse-tree commit drew when it moved that command's parse error and left its missing-file error alone, on the grounds that validate, test and parse-tree already agree on -1 for a path that does not exist. Both `dne` cases in the status-code table still assert it. The classification is a thin wrapper around the existing body rather than a restructuring of it: `execute` now matches on the result of `evaluate`, which is the former `execute` unchanged. Reported by the command rather than left to `main`, because once this function has decided the failure is not internal, "Error occurred" no longer describes it -- so stderr loses that prefix, which `test_a_data_file_with_no_document_is_reported_as_empty` asserts. User-visible change: `validate` exits 5 where it exited 255 for a data file whose content cannot be read, and the stderr line loses its "Error occurred" prefix. Verified unchanged: a compliant file still exits 0, a violation still exits 19, and a path that does not exist still exits 255 with the same message. Twelve test cases asserted the old code and are changed, with the reasoning at each.
…it reverted `failing_complex_rule.out` and `failing_join_show_summary_all.out` were both changed by "Report the operand a query-versus-query equality actually failed on" and then again by "Count data-file lines and columns from one". The second regenerated twelve fixtures from the binary, and that binary did not contain the first -- the two were written on the same base and neither could see the other. So the committed text carries the 1-based line and column numbers and, underneath them, the operand reporting the earlier commit had already removed. Regenerating from a binary is the right way to update a captured output and the wrong way to update one a sibling branch has also touched: it reinstates whatever that sibling removed, silently, and the result still looks internally consistent. Both files now come from the combined binary. Each differs from the committed version in exactly the two lines the operand commit had changed, and in no others, so the line numbers and excerpt windows the numbering commit established are untouched. What the restored text says, and why it is the true statement: - `failing_complex_rule.out`. `%expected == %replaced` with `%expected` the literal "random_str" and `%replaced` a regex_replace of the Arn. `ComparedWith` is `["random_str"]` and the reason names `Path=[L:0,C:0]`, because a rule literal is built with `Path::root()` and has no position in any file. The committed text printed the Arn as both `Value` and `ComparedWith` -- a failed equality whose two operands it reported as the same string, asserting the Arn was absent from a set holding the Arn. - `failing_join_show_summary_all.out`. `"a,b" == join(%collection, ",")`. The left operand in this position is a query for a property named `a,b`, which is why the check renders unquoted as `a,b EQUALS join(...)`; the template has no such property, so the operand set is empty and the diff of its unmatched values is `[]`. `Value` and `PropertyPath` still name the join result, because `QueryIn::from_rhs` places the finding on the side a reader can act on. The committed text again printed one value as both sides. Markers checked against the template by counting rather than by trusting the binary. In `functions/data/template.yaml`, line 10 is ` Arn: arn:aws:...` and its value begins at column 12; line 15 is ` - a` and the `a` is at column 9. The fixtures say `L:10,C:12` and `L:15,C:9`. The excerpt beside each marker runs n-2 to n+3 and each numbered line holds the template's text for that line. No product code changes. validate: 211 passed 4 failed, to 213 passed 2 failed, with no other test in the target moving.
…erts its message
`a_payload_that_will_not_parse_says_where_it_came_from` asserted `INTERNAL_FAILURE`.
It was written when that was the only outcome available: `deserialize_payload`'s
`Error::ParseError` escaped `validate` and reached `main`'s catch-all, which prints
"Error occurred" and exits -1. A later commit on a sibling branch classified every
`ParseError` out of `validate` as the caller's input and gave it `ERROR_STATUS_CODE`,
naming the payload deserializer as one of the producers it meant to cover. Neither
commit could see the other, so the code moved and the assertion did not.
The classification is the side that is right, and this case is the one it argues for
most directly. Empty stdin under `--payload` is the CI shape the message was rewritten
for; -1 is the code this repository reserves for cfn-guard breaking, so the old pairing
told an author who forgot a pipe that the tool was at fault. 5 is not 19, so a gate
still tells a broken input from a violation.
The message assertion needed nothing. `deserialize_payload` still names the flag, the
stream and the expected shape, and the reclassification only drops `main`'s prefix:
Parser Error when parsing `Unable to parse the --payload JSON read from stdin: EOF
while parsing a value at line 1 column 0. Expected {"rules":[...], "data":[...]}`
which is the same form `test_a_data_file_with_no_document_is_reported_as_empty` pins
for an unreadable data file.
Added an assertion that stderr does not carry "Error occurred". The code and the
message are set in two different places -- the code by `execute`'s classification, the
prefix by whether `main` ever sees the error -- and this pairing is what the two
branches disagreed about, so it is worth holding rather than inferring.
validate: 213 passed 2 failed, to 215 passed 0 failed.
rulegen emitted `""` for a null. That was right against a loader which resolved an
empty YAML node to the empty string, and a sibling commit made an empty node resolve to
null -- deliberately, so that `k:` and `k: ""` stop being indistinguishable. The two
crossed over: `"NULL"` against `""` is `not comparable null, String`, and every rule
generated from a template with an empty `BucketEncryption:` failed the template it was
generated from. Three fixtures under guard/resources do this, and the round-trip test
the rulegen commit added is what named it.
Fixing the null arm alone would have been wrong, because the null arm is not the only
place the two readers disagree. Measured over 36 one-property templates, one per YAML
spelling, before this change:
template writes loader reads rulegen emitted round-trip
P: / P: null / P: ~ null "" fails
{Inner: null} / [a, null] null, nested "", nested fails
P: .nan / P: .inf the text ".nan" "" fails
P: 1e-400 the text "1e-400" 0.0 fails
P: 0b101 the text "0b101" 5 fails
P: 9223372036854775809 the text a 19-digit int does not parse
<<: {A: 1} the merge resolved a key named "<<" fails
!Ref / !GetAtt / any !Foo {Fn::Foo: ...} -- abort, exit 1
Ten emitted a clause the source template does not satisfy. One emitted a file that does
not parse, which the re-parse check then discarded. Five aborted the process at exit 1 --
that is every CloudFormation template using a short-form intrinsic, because serde_yaml
will not deserialise a tagged node into serde_json::Value at all.
**Reconciling the two readers by hand is not possible**, and that is why this reads once
rather than twice. serde_yaml maps `.nan` and an empty node to the same `Value::Null`,
and the loader reads the first as the string ".nan" and the second as null. No rendering
rule over serde_json::Value can be right for both, because the distinction is gone
before rendering starts. `!Ref` is the same shape one level up: one reader has no value
at all where the other has a mapping.
So `generate_rule_map` now loads through `rules::values::read_from` and
`PathAwareValue::try_from` -- the two calls `validate`'s `build_data_file` makes -- and
`value_to_guard` renders a `PathAwareValue`. The value rendered is the value compared,
so a future resolution rule in the loader cannot desynchronise them. `PathAwareValue`
and not the loader's `MarkedValue`, because the conversion between them is where a
mapping's duplicate keys collapse last-write-wins, and a template writing `P` twice must
generate a clause about the value the evaluator will see.
Each rendering was measured by generating the clause and validating it against its own
template. Two need saying:
- a null is `null`, at any depth.
- a whole float keeps its fractional part. `f64::to_string` drops it, and `1e20` then
comes out as 21 digits with no point, which the rules parser rejects as an integer too
wide for i64. With the point it round-trips, checked up to 1e300 and down to 1e-300.
Everything the loader keeps as text -- the too-wide integer, `.nan`, `.inf`, `1e-400`,
`0b101`, `0755` -- now renders through the string arm and needs no arm of its own. So do
the intrinsics: `!GetAtt Other.Arn` becomes `{"Fn::GetAtt":["Other","Arn"]}` and matches.
All 36 spellings round-trip now, 35 as a passing clause and one refused with its reason.
Two other changes fall out of this and are not separable from it:
`process::exit(1)` on an unreadable template becomes `ERROR_STATUS_CODE`. Three call
sites exited 1 from inside a library function: a code that collides with
`TEST_ERROR_STATUS_CODE`, that rulegen does not document, and that an in-process caller
cannot survive. `generate_rule_map` returns `Result` and `execute` reports it, which is
the shape `validate` uses for the same class. A template that does not *exist* keeps -1
through the `?` on `read_to_string`, which is the line validate draws too.
`parse_template_and_call_gen` returns `Result<RuleMap>`; it is public but
`#[allow(dead_code)]` and only the unit tests call it.
The round-trip test's candidate filter widens. It read `HashMap<String,
serde_json::Value>` to match what rulegen parsed with, justified as keeping the walk away
from the `process::exit(1)`; both halves of that are now obsolete, and what is left is
that it was the narrower reader -- it could never admit a template using `!Ref`. It now
deserialises only a `Resources` field, ignoring the rest of the root, so an
`AWSTemplateFormatVersion` string does not disqualify a file. The logical ids go through
a HashMap and not a serde_yaml::Value, because a Value refuses a duplicate mapping key
and `duplicate-logical-id-template.yaml` declares one on purpose -- reading it as a Value
silently dropped that template, taking the candidate count from 35 to 34. The final
candidate set is byte-identical to the old one, 35 files, so the widening costs nothing
today and stops the walk from skipping the first intrinsic-using template someone adds.
Two unit tests asserted the old rendering and now assert `null`. The reason each gives is
the one above: the old expectation was correct against the old loader.
Suite: 1806 passed 5 failed, to 1811 passed 0 failed across all 16 targets. Round-trip
over guard/resources: 35 candidates, 28 round-trip, 7 refused at PARSING_ERROR -- four
carry no properties, two carry them only on a resource with no Type, and one is the
boolean-observed-both-ways refusal. clippy --all-targets -D warnings clean, fmt clean.
…ot compile Twenty `let`-bound tables in eval_tests.rs carried their length, so deleting a cell from one was `error[E0308]` rather than a silent shrink. Twelve tables were written as array literals directly in a `for` header, with no binding to annotate, and were exactly as unprotected as before that work: libtest counts test functions, a loop cell is not one, and `cargo fmt`, clippy, typos, shellcheck and the suite are all blind to a missing cell by construction. Each of the twelve is lifted to a `let` above its loop and annotated. The conversion is mechanical, and the shape rule in `every_loop_table_carries_its_length` is why it needed no widening: an inline table already opens its literal at the end of the `for` line, so `let name: [_; N] = [` leaves that opening exactly where the rule looks. The guard's examined population goes 20 -> 32 and the twelve are covered, not merely converted. Writing the first cell on the `let` line instead would have converted them and left them outside the rule, which is a different state. The counts came from the compiler, by the route the guard's own failure message recommends: annotate `[_; 0]`, build, read the true size out of each `error[E0308]`. Twelve diagnostics, twelve sizes, 43 cells. That is not ceremony. `accepted_spellings` holds thirteen cells written several to a line, so reading its length off the line span gives 2 -- and the other eleven tables would have agreed with a line-span count, so nothing about the batch would have looked wrong. A green build is now the proof that all twelve numbers are right, because a wrong one cannot compile. Forced both failures on a table this commit creates rather than trusting the mechanism from the earlier one. Deleting a cell from `condition_shapes` gives `error[E0308]` at its line, "expected an array with a fixed size of 3 elements, found one with 2 elements". Stripping the annotation from `pair_spellings` gives the guard's own failure, "1 of 32 tables carry no length", naming the binding -- which is what establishes the population claim from the test rather than from a reimplementation of its rule. Adds no test function and no cell, so the suite count is unchanged at 3149 passed / 0 failed / 0 ignored across 16 suites on both profiles. What it adds is twelve compile-time constraints, which no test count can show. The guard's comment is corrected where it recorded the twelve as an open limit and the population as twenty, and it now records the one inline shape left uncovered: twelve single-line `for` literals like `for negated in [false, true]`, left deliberately because each enumerates a small domain rather than tabulating cases, so a missing cell breaks the surrounding assertion rather than quietly narrowing coverage.
…t all
`Resources[ keys == /Z9/ ]` exited 0 with `Status = SKIP` and zero reason lines, even
under `--show-summary all`. The block spelling and the clause spelling were both silent.
Two explanations were offered for that and they wanted opposite fixes: nothing was
recorded, or something was recorded in a shape `own_skip_reason` has no arm for. Neither
is what happens. Walking the record tree, the block form files
Rule > BlockGuardCheck(SKIP) > ClauseValueCheck::Comparison(FAIL)
and the clause form files `Rule > GuardClauseBlockCheck(SKIP) > ...`, and `own_skip_reason`
already has an arm for each of those container shapes. Both arms answered `None` because
both records carried `message: None`, set unconditionally at the two sites that had just
branched on the emptiness. Nothing was missing but the message. Confirmed by probe: filling
each site in turn produced a reason line for that spelling and left the other silent, which
is also why both are changed here rather than one.
A distinct sentence rather than the refusal wording reused. `find_skip_reason` exists to
surface a comparison that could not be decided, and an empty selection is not one -- the
query ran and answered. `empty_lhs_message` is the neighbouring helper and is deliberately
not reused: it is about the left-hand *variable* of a comparison resolving to no values, so
it tells the reader to look at what binds the variable and says the clause fails. Both
halves are wrong here. These are ordinary queries with no variable to bind, and the outcome
is a SKIP. Pointing a reader at a `let` they never wrote is the same class of mistake as
naming a condition a rule does not contain.
No cause is named, and that is a correction rather than caution. The first draft said an
empty selection is what "a path the data does not have, an empty collection, and a filter
that excluded every value all produce alike", and two of those three are false. Measured:
Resources[ keys == /Z9/ ] { ... } exit 0 SKIP reaches this branch
Resources.*[ Type == "nope" ] { ... } exit 0 SKIP reaches this branch
Resources.Absent.Type == "x" exit 19 FAIL does not
Resources.One.Properties.Tags[*] { } exit 19 FAIL does not, over `Tags: []`
Resources.*.Type == "x" exit 19 FAIL does not, over `Resources: {}`
A missing path and an empty collection fail closed elsewhere instead of arriving as an empty
selection, so a filter is the only producer measured -- and that is still not put in the
sentence, because one sampled producer is not proof of the only producer. The query is
printed and the author reads their own query.
Attached to the SKIP only, and not because the FAIL has no consumer. The sentence opens "the
rule did not apply", which is false of the `not_empty` FAIL: there the rule did apply and the
clause failed for want of a value. That case wants its own wording and is left alone.
One existing cell moved. `a_type_block_skip_names_the_cause_it_can_support`'s body-filter cell
expected the type block's roll-up, "no AWS::EC2::Volume in the input was checked: no clause in
the type block applied to any of them". The walk searches children before a record's own
message because the deeper message is the more specific one, and here that premise holds in
substance: naming the query that matched nothing says which of the block's clauses did not
apply, which the roll-up cannot. Its forbidden fragment is unchanged, so the mistake it was
written to catch -- a block with no `when` mentioning one -- is still caught, and the roll-up
is still expected by the inner-`when` cell beside it.
Known limitation, not fixed here: the query renders through `SliceDisplay`, which prints the
parser's own name for a filter, so the line reads `Resources. (map-key-filter-clauses)` rather
than `Resources[ keys == /Z9/ ]`. Every other query-naming message in this file renders the
same way; changing it moves all of them.
Verified at d4286e6: 3159 passed / 0 failed / 0 ignored, 16 suites, both profiles, with
`--test-threads=1`. That is +10 on the 3149 baseline, all accounted: this commit's one
function in `eval_tests.rs` counts twice because both `src/lib.rs` and `src/main.rs` build it,
and the parser commit's four `rstest` cases count eight the same way. `cargo clippy
--workspace --all-targets -- -D warnings` clean over 156 units in a deleted-then-confirmed-
absent target dir, run with the working directory inside the tree so the pinned 1.77.2
toolchain is the one that lints. `cargo fmt --all -- --check`, `typos` and `shellcheck
install-guard.sh .github/scripts/check-registry-corpus.sh` all clean. The registry corpus
script's one-argument control exits 2; the corpus assertion itself fails on this host
identically with the unmodified d4286e6 binary, the normalized diff between the two runs
being only the temp directory names, so it is environmental and not moved by this change.
`block` shadowed `input` at each step -- the ordinary shape in this file -- so the span its
refusals reported with was the one left after `fold_many1` consumed the body. That lands on
the closing brace of the enclosing block. Measured before and after:
duplicate `let v` at line 3 column 5 of a rule body line 5 column 1 -> line 1 column 8
the same inside a `when` at line 4 column 9 line 6 column 5 -> line 2 column 27
a `let` cycle at line 2 column 5 of a rule body line 5 column 1 -> line 1 column 8
the same inside a `when` at line 3 column 9 line 6 column 5 -> line 2 column 27
The old positions are the `}` in each case, two lines past the construct, and the error's
fragment was `}` -- which is the giveaway. The last round deferred this site as "imprecise
rather than wrong" while fixing `parse_map`, whose defect was the same shadowing one function
away. A span that names a different construct is not an imprecise column, so that premise does
not hold and the deferral goes with it.
Both refusals in `block` are fixed, not just the duplicate-variable one. They sit one
expression apart and shared the single shadowed span, so fixing either alone would leave a
known-identical defect two lines below it -- which is how this one survived a round. The
duplicate-variable case is the one that was reported; the `let` cycle was found beside it and
measured to the same two positions.
The opening brace rather than the offending statement, and that is a deliberate stop short.
`LetExpr` carries no location, so naming the declaration itself means teaching the fold to
accumulate spans, which is a wider change than a locator repair and belongs on its own. The
brace is the scope the duplicate message is actually about -- "in the same scope" -- it is a
construct the author wrote, and it is the same choice `parse_map` made for the same reason.
Fixed by giving the advanced spans their own names, also as `parse_map` does.
`let_cycle_message`'s own doc comment asserted that "the block-level site carries a span that
says where". That was false when written and is true now; the sentence is kept and corrected
rather than deleted, with the file-level site's total absence of a position named as the
separate gap it is.
Only the position moves. Both refusals keep their exit code and their message text verbatim,
asserted per case alongside the position. The accept/reject boundary is unchanged and was
re-measured: a duplicate in one scope is refused, the same name in two sibling rules is
accepted, a file-level name shadowed by a block-level one is accepted, and a duplicate in a
nested `when` block is refused. The unrelated locators are untouched -- the duplicate-parameter
refusal still reports line 1 column 11 and the repeated-map-key refusal line 1 column 9.
The new test asserts the exact column and asserts the old wrong position absent, because a
test that only checked for "some position" passed before the fix. It re-derives the brace
column from the named fixture line rather than trusting the case parameter, and searches within
that line rather than for a bare `{` over the whole text: the nested cases hold a rule brace
and a `when` brace, and a whole-text search finds the outer one. That is the mistake the
sibling map-key test records catching when it was written the short way.
Verified at d4286e6: 3159 passed / 0 failed / 0 ignored, 16 suites, both profiles, with
`--test-threads=1`. The four `rstest` cases here count eight because both `src/lib.rs` and
`src/main.rs` build `parser_tests.rs`; the cell count is pinned by `#[case]` attributes rather
than by a runtime length, so a deleted case is a deleted test rather than a quieter run. All
four redden at d4286e6, each naming the old position in its failure output. `cargo clippy
--workspace --all-targets -- -D warnings` clean over 156 units in a deleted-then-confirmed-
absent target dir, run with the working directory inside the tree. `cargo fmt --all --
--check`, `typos` and `shellcheck install-guard.sh .github/scripts/check-registry-corpus.sh`
all clean.
…y narrowable The previous commit left twelve `for` headers carrying their array on one line, and justified it: each was "a two-to-five element enumeration of a domain rather than a table of cases", so a missing cell would break the surrounding assertion rather than quietly narrow coverage. That was an argument, not a measurement, and it is false for ten of the twelve. Measured by deleting one cell from each and running the whole suite single-threaded. Ten stay green at 3149 passed, 0 failed: the bucket and let-value expectation pairs, the five rejected number spellings, the boolean flag values, the EMPTY comparators, the size labels, the KeyList preambles, the when spellings, the quantifiers and the line-ending spellings. Deleting `"false"` from `for value in ["true", "false"]` halves that test's coverage and nothing says so. Only `for negated in [false, true]` and `for gate in [false, true]` redden, and not for the reason the argument gave. Their assertions do not differ by polarity; they redden because `every_operator_and_operand_shape_agrees_with_a_stated_oracle` counts cells into a total and asserts on it. Real protection, but incidental and non-local -- delete that counter assertion and those two join the other ten. Both are bound here too, so each table's protection is local to it. All twelve now carry a length, counts from the compiler as before: annotate `[_; 0]`, build, read each `error[E0308]`. Twelve diagnostics, 27 cells. `every_loop_table_carries_its_length` accepts the single-line shape, so the twelve are covered and not merely annotated. Two changes were needed. The shape rule now takes a complete single-line statement as well as a multi-line opening. `];` is what makes that safe rather than a guess: the four lines in this file beginning with `let` and an `[` that are not Rust -- a CloudFormation `Fn::Join` fragment and three Guard-DSL `let` statements inside Rust string literals -- all fail it, and before the twelve were bound the widened rule matched zero lines. And the `=` split moved from the last occurrence to the first. Equivalent for a multi-line table, whose `let` line carries no cell text, and wrong for a single-line one whose cell may hold an `=`. `key_list_preambles` is that case: its two cells are Guard-DSL strings reading `let k = Cfg.KeyList`, so the last `=` sat inside a string literal, the initializer came out as ` Cfg.KeyList[*]"];`, and the binding was annotated, compiling, protected against deletion, and invisible to the test whose whole job is to find unprotected tables. Found by reconciling the population against the twelve rather than by reading the rule. Population 32 -> 44. Forced both failures on tables this commit creates: deleting a cell from `rejected_spellings` gives "expected an array with a fixed size of 5 elements, found one with 4 elements"; stripping the annotation from `flag_values` gives "1 of 44 tables carry no length", naming it. Adds no test function and no cell, so the suite is unchanged at 3149 passed / 0 failed / 0 ignored across 16 suites on both profiles. It adds twelve compile-time constraints.
`incomparable_membership` walks the same left-right cross product
`InOperation::compare`'s `(None, None)` arm walks, to count the refusals a
passing `NOT IN` clause passed on. That arm stops pairing a left-hand value at
the first right-hand value that matches it -- `continue 'each_lhs` at the
string-containment and membership arms, and at the empty-left skip above them --
and the predicate had no such stop, so it kept pairing the value against every
LATER right-hand value and counted refusals from pairings the arm never built.
THE DIVERGENCE, measured rather than argued. Over a 676-clause sweep of the
mixed-left shapes, instrumented at both sites so the two pair sets could be
diffed rather than eyeballed: 54 clauses where the predicate refuses on a pairing
the arm short-circuited past. `MatchStr[*] NOT IN HayInt[*]` over
`{"MatchStr": ["ab"], "HayInt": ["xxabxx", 5]}` is the smallest -- `"ab"` is
contained in `"xxabxx"`, so the arm stops, and `("ab", 5)` is the predicate's
pairing alone. It reproduces through the membership short-circuit too, with
`{"LA": [[1], [9]], "RA": [[1], "s"]}`.
LATENT, AND FIXED ANYWAY. All 54 are held back by the verdict gate in
`binary_operation`: a short-circuit IS a match, a matched value fails `NOT IN`,
and every multi-value shape that could pass instead builds its own refusal on the
sibling that reaches the refusing right-hand value. Measured, no input reaches a
false notice. But the suppression lives in a different function from the
divergence, so changing what either short-circuit fires on -- or adding a third
beside them -- makes it live with nothing in the tree to flag it. `ee60bc5f` is
this same shape one level up, and its argument was sound about the population it
had examined and silent about the one it had not.
BY CALLING THE ARM'S OWN FUNCTIONS. `membership_stops_after` asks
`found_in_string` and `contained_in`, the calls the arm makes, and reads `All`
and `Success` off them. Restating their conditions is how the divergence arose in
the first place, so a copy of the rule must not exist here. It sits directly
above `impl Comparator for InOperation` so that an editor of either short-circuit
meets it.
NOT THE BUILT SET ITSELF, and the reason is the call order rather than the size
of the change. `binary_operation` calls the predicate BEFORE `cmp.compare`,
deliberately: the note there records that the incomparability is not recoverable
from the result, because the not-flag has already turned "no element matched"
into a success by then. Reading the set the arm built needs either the comparison
to run first -- the order the notice cannot use -- or `Comparator::compare` to
carry it out for every operator, which reaches 99 non-test
`QueryResult::Resolved` sites.
THE STOPPING PAIRING IS KEPT, which is the narrow choice on purpose. The arm
reaches `contained_in` for it and only skips the element loop, so what this
removes is exactly the later pairings and nothing else. Dropping its accounting
as well would change what the predicate answers for a list-against-list
`Success` -- `["x", 1]` inside `["x", 1]` has element pairs that refuse while the
subset holds -- and that is a separate question with its own measurement owed,
not a free rider on this one.
VERIFIED. Suite 3151 passed, 0 failed, 0 ignored across 16 suites, up from
d8c2582's 3149 by the one test function added here and nothing else, counting
twice because the tests compile into the lib and the bin target alike. Both
figures measured in freshly deleted target directories with `--test-threads=1`,
the base in an unmodified checkout of d8c2582. Nothing observable moves: 676
clauses are byte-identical before and after on exit code, membership notice,
vacuous notice and report body. The pinned aws-guard-rules-registry corpus at
7f7340c is unchanged -- 5 DEPRECATION lines, all five the incomparable-membership
one, 0 failed rules, and a 576794-byte stdout report identical to the byte. The
`== 'False' OR == false` idiom answers identically with stderr identical to the
byte. `cargo clippy --workspace --all-targets -- -D warnings` checks 156 units
clean from a deleted directory under the pinned 1.77.2, and
`cargo fmt --all -- --check` is empty.
`the_membership_notice_stops_pairing_where_the_operator_stops` asserts the PAIR
SET rather than a verdict, because the verdict cannot discriminate: the gate
suppresses the notice on every clause that reaches the divergence, so a cell
watching an exit code passes with the defect in. It reddens with the stop
disabled and passes with it restored. It carries a miss control, so a fix that
truncates unconditionally fails rather than passing, and a literal-arm control,
because the short-circuit belongs to the two-query arm alone -- the
literal-right-hand arm walks every right-hand value and truncating there would
drop pairings it does build.
…path's
Three comments cite "143 rules of the pinned aws-guard-rules-registry corpus"
beside a decision about the membership refusal -- `eval.rs`'s
`undecided_gate` call, `Unanswerable`'s own doc in `operators.rs`, and
`the_notice_asks_about_every_granularity_the_operator_decides_at` in
`eval_tests.rs`. Each is correct about what it describes and none says which
comparison path it describes, so a reader takes it as the price of promoting
the membership refusal. It is not, and the direction of the error is the
expensive one: it makes a registry-free change look registry-breaking while
nothing records the 19 verdicts that are its actual price.
WHAT THE 143 IS. The cost of reading `Unanswerable::IncomparableKinds` as an
undecidable GATE, which is why every one of the three cites
`ScanOnPush == 'False' OR ScanOnPush == false` -- an `==` clause, answered by
`EqOperation`. Nothing about that figure is stale and no number in it changes
here.
WHAT IT IS NOT. `is_one_of` is the only place a kind mismatch is discarded on
the `IN` path, at its `Err(_) => {}`, and it is reached from exactly two sites:
`contained_in`'s membership loop and the `(None, None)` arm's element loop.
`EqOperation` reaches neither, so the idiom the 143 is measured on cannot
traverse the arm a membership fix would change. Measured at `1ba4648d` by
promoting that one arm and nothing else: the corpus comes back byte-identical,
576794 bytes of stdout, all five DEPRECATION lines present, 0 failed rules.
WHAT IT COSTS INSTEAD, which no comment recorded. 19 clauses of a 676-clause
sweep of the membership shapes move from exit 0 to exit 19 -- `Pair NOT IN
Ubool`, `Ports NOT IN Umap`, `Nest NOT IN D13[*]`, `AbList[*] NOT IN Uint`,
`Deep NOT IN Umap`, `some AbList[*] NOT IN D13[*]` and thirteen more. Each is a
clause that passed on a refusal and now fails closed, which is what
`docs/CLAUSES.md` says a comparison across kinds that are not both numeric
owes, so the 19 move toward correctness. They are verdict changes even so, and
they are what the change has to be argued against.
AND THE IMPOSSIBILITY, recorded because the tree deferred this with a cost that
was wrong in both directions. `incomparable_membership` exists because the
notice has to know a pairing refused while the clause still passed, and the
operator's result cannot say that: instrumented immediately after
`cmp.compare`, every notice-emitting clause carries zero `NotComparable`
results, literal-right-hand shapes included, which is what the registry's five
are. Promoting the refusal moves those 19 verdicts and is the
`docs/KNOWN_ISSUES.md` change, gated on the five registry rules. Carrying it
without moving a verdict needs a state meaning "no match, something refused, and
do not fail closed" -- a fourth `Membership` variant threaded through `Compare`,
`ComparisonResult`, `ValueEvalResult` and `QueryIn`/`ListIn`, 167 non-test sites
at this commit -- and that state is the "refused but passing" distinction the
enum was split to keep out of the result type. So the separate predicate is not
an accident a tidier design removes; it is how the result type stays free of
that distinction, and `membership_stops_after` calling the arm's own functions
is the mitigation for the drift risk that choice carries rather than a
workaround for a missing refactor.
Comments only: 48 insertions, no deletions, no non-comment line. Verified
rather than predicted. Suite 3161 passed, 0 failed, 0 ignored across 16 suites,
unchanged from `1ba4648d` and measured in a freshly deleted target directory
with `--test-threads=1`. The 676 clauses are byte-identical to `1ba4648d` on
exit code, both notice channels and report body.
`every_loop_table_carries_its_length` passes and its population is unchanged at
45 examined with the examined set identical, since a `///` line cannot open a
table. `cargo clippy --workspace --all-targets -- -D warnings` checks 156 units
clean under the pinned 1.77.2 and `cargo fmt --all -- --check` is empty.
`e2cacdc4` states its suite as 3159 and `1ba4648d` states 3151. Measured, the totals at those commits are 3151 and 3161. Neither figure is wrong by its author's own delta, which is the shape the ledger at the head of `eval_tests.rs` already records four times: both are a NEIGHBOUR's total, quoted rather than measured. 3159 is `a9ad44e8`'s, the commit immediately after `e2cacdc4`. 3151 is `e2cacdc4`'s, three commits before `1ba4648d`. A figure that turns up in a neighbouring message in both directions was not computed from a stale base; it was read off a tree that happened to be open. `e2cacdc4` and `a9ad44e8` also both attribute the tree they measured to `d4286e68`, which measures 3149, so neither measured 3159 there. `a9ad44e8`'s 3159 is correct for `a9ad44e8` itself and is now recorded as correct rather than left to look like the two beside it, so a later reader can tell which figures were checked from which merely survived. `1ba4648d`'s DELTA is wrong too, and no earlier row in that ledger does that. It reads "up from `d8c2582d`'s 3149 by the one test function added here". 3149 is genuinely `d8c2582d`'s and one test function is genuinely what it adds, so each half survives inspection alone. The adjacency fails: `d8c2582d` is four commits back rather than the parent, and `e2cacdc4` and `a9ad44e8` land +10 between them. That retires the ledger's standing rule that the deltas in these messages are trustworthy even where the totals are not; past `b1ff3845` a delta needs its baseline's SHA checked against the parent first. The true chain, added as SHA-tied rows: 3149, 3149, 3151, 3159, 3159, 3161 for `d4286e68`, `d8c2582d`, `e2cacdc4`, `a9ad44e8`, `b1ff3845`, `1ba4648d`. Four unmeasured commits sit between the ledger's previous last row and `d4286e68`, and the +5 across them is not apportioned; the gap is written as a row rather than closed up so 3144 and 3149 are not read as consecutive. `b1ff3845`'s "Population 32 -> 44" is the same defect in the same commit's other figure. Applying `every_loop_table_carries_its_length`'s own shape rule to `eval_tests.rs` per commit gives 32, 33, 33, 45, 45, so the transition is 33 -> 45. The +12 is right and both endpoints are `d8c2582d`'s population, because `e2cacdc4` added a 33rd table -- already annotated, so no coverage was ever missing -- before the twelve were bound. 44 is 32 + 12, and it is nobody's measured figure. That is recorded rather than only corrected, because a number reached by arithmetic fails differently from one reached by measurement. A reader looking for 44 measured the population twice, through the guard and through an independent script, got 45 both times, could not reproduce 44 under any shape rule, and declined to invent a cause for it. That was the right call: there is no rule under which this file's population is 44 at any commit on this branch, so the search could not have ended anywhere else. A figure no rule reproduces may not be a measurement, and the cheapest test is whether it equals another figure plus a remembered delta. One in-tree instance, which is the only place a stale figure was reachable without reading a commit message. `every_loop_table_carries_its_length`'s doc comment reported the ten silently narrowable tables staying green "at 3149 passed", which is `d8c2582d`'s total, four commits before the deletions were run. It now reads 3159 and says that it read 3149 until it was corrected, rather than presenting the new figure as though it had always been there. The measurement itself is untouched: what makes a narrowing invisible is that the total does not move, not what the total is. `_typos.toml` gains one identifier, following `34d965ba`'s line directly above it. `typos` reads any digit-flanked `ba` as `by`/`be`, and `1ba4648d` puts it between `1` and `4`, so the six citations the ledger needs all flag. Verified as the cause rather than assumed: with the line removed `typos` exits 2 with exactly 6 hits, and with it restored exits 0 with no output. Avoiding the SHA instead was rejected because naming a SHA rather than stating a bare total is the whole convention the ledger runs on. Verified on this tree: 3161 passed / 0 failed / 0 ignored across 16 suites with `--test-threads=1`, unchanged from `1ba4648d`, which is what a comment-and-config change owes. `cargo clippy --workspace --all-targets -- -D warnings` clean over 156 units in a deleted target directory with the working directory inside the tree, so the pinned 1.77.2 is the toolchain that lints. `cargo fmt --all -- --check` empty, `typos` clean, `shellcheck install-guard.sh .github/scripts/check-registry-corpus.sh` clean. No Rust line changed that is not a comment, checked by counting non-comment added and removed lines in the diff, which is 0. Provenance of the six totals, since the point of this commit is that a figure should say where it came from. `d4286e68` and `1ba4648d` were measured directly here, each from its own `git archive` tree and `CARGO_TARGET_DIR`. The four between them are derived from an inventory of test-function and `#[case]` attributes per commit -- +0, +2, +8, +0, +2, doubling the crate-internal ones because `src/lib.rs` and `src/main.rs` each build `eval_tests.rs` and `parser_tests.rs` -- which sums to +12 and reconciles 3149 to 3161 exactly, and the release gate separately measured 3151, 3159, 3159 and 3161 at the four commits after the base. So every row rests on a direct measurement or on two derivations that agree. Reconciling test FUNCTIONS against the diff is what makes a wrong total visible; comparing one total against another does not, because a wrong total can be internally consistent, which is how both of these survived.
`empty_selection_message` ended "Nothing was refused -- the query ran and matched nothing". The
second half is local and true. The first half is a claim about the whole rule, made by a function
that is handed one query and cannot see the rule's other clauses. When one of them was refused,
the sentence said otherwise:
Resources: {A: {Type: AWS::S3::Bucket, Properties: {KmsKeyId: {Ref: MyKey}, MustBeTrue: false}}}
rule r {
Resources[ keys == /Z9/ ] { Type == "AWS::S3::Bucket" }
or Resources.*[ Properties.KmsKeyId == "alias/aws/s3" ].Type == "nope"
}
The second disjunct's filter compares a map against a string and is refused. Measured, exit 0 and
`Status = SKIP` both ways:
as written the rule did not apply because the query Resources. (map-key-filter-clauses)
selected no values from this input. Nothing was refused -- the query ran and
matched nothing.
disjuncts the rule did not apply; a comparison in one of its query filters reported:
swapped PathAwareValues are not comparable map, String
So the false half was also suppressing the only actionable fact in the report, and which of the two
a reader saw depended on the order they happened to write their disjuncts in.
This is the mechanism the rest of this branch keeps repairing, in a sentence I wrote two commits
after repairing it in `descend`: a fact true of one scope asserted as a fact about a wider one.
`incomparable_membership` re-deriving the operator's pairing decisions is the same shape, and so is
a skip reason reading a referenced rule's gate as the referring rule's own.
The claim is dropped rather than narrowed, and narrowing was the tempting repair. "Nothing about
*this query* was undecidable" is unsupportable at these sites too: a refusal inside the query's own
filter also arrives here with an empty selection. Probed at both computed-status `BlockGuardCheck`
sites in `eval_guard_block_clause` -- `Resources.*[ Properties.Size > 10 ]` over `Size: "50"` fires
the empty-values branch, sets this very message, and is merely shadowed by the deeper refusal
`find_skip_reason` reaches first. Measured rather than reasoned, because the four construction
sites make the status alone ambiguous between two of them.
What is left is what the branch condition gives: the query ran, and it selected nothing. "Ran" is
kept and is local -- the `Err` arm above returns before this point, so reaching here means the
query resolved rather than failed. The empty-selection half is unchanged, which is why the commit
that added it was worth landing, and both existing cells that assert it still pass untouched.
Not fixed here, and named so it is not read as covered: which of two sibling reasons surfaces still
depends on clause order, because the walk takes the first child carrying a message. That is a
ranking question in `find_skip_reason` rather than a false claim in a sentence, and the two want
separate changes.
Why the existing cell did not catch it: `an_empty_selection_skip_says_which_query_matched_nothing`
uses single-clause rules with no refusal anywhere in them, so the false half was true of every
input it runs. It still passes at `1ba4648d`, which is the measurement that shows the gap was in the
fixture rather than in the assertion.
The new cell asserts the whole sentence rather than the absence of a string. A bare
`!contains("Nothing was refused")` is a negative assertion over a literal: it passes for any
rewording of the same false claim, because the rewording no longer holds the string being looked
for. Both orderings run in one test, and the swapped one is the non-vacuity control -- without it,
"the reason must not deny a refusal" would pass on a fixture where nothing was refused, which is
exactly the shape that let this through.
Correcting a figure in `e2cacdc4`'s own message while it is in reach: it says "Verified at
d4286e6: 3159". `d4286e68` is 3149, and 3159 is `a9ad44e8`'s total two commits later. The
per-commit chain is 3149, 3151, 3159, 3159, 3161.
Verified at 1ba4648, which is 3161: 3163 passed / 0 failed / 0 ignored, 16 suites, both profiles,
with `--test-threads=1`. That is +2, one test function in `eval_tests.rs` counting twice because
both `src/lib.rs` and `src/main.rs` build it, confirmed as exactly two harness entries rather than
inferred from the total. Its two cells sit in a `for` loop and are invisible to the harness, so
their count is pinned by `let cases: [_; 2]` and a deleted cell is a compile error. The new cell
reddens at 1ba4648 on the denial assertion, printing the false sentence in full. `cargo clippy
--workspace --all-targets -- -D warnings` clean over 156 units in a deleted-then-confirmed-absent
target dir, run with the working directory inside the tree so the pinned 1.77.2 toolchain is the
one that lints. `cargo fmt --all -- --check`, `typos` and `shellcheck install-guard.sh
.github/scripts/check-registry-corpus.sh` all clean. The registry corpus script's one-argument
control exits 2; the corpus assertion itself fails on this host with the same two manifest
byte-count mismatches recorded last round, where it was shown to fail identically with an
unmodified binary, so it is environmental and unmoved by this change.
…tched
`rhs_values_paired_with` truncated the right-hand walk at the pairing where
`InOperation::compare`'s `(None, None)` arm stops, but kept that pairing's own
accounting: `&rhs_values[..=at]`. It is now `[..at]`.
A value that stops has MATCHED, and all three stops are matches. The empty-left skip
reads an empty list as vacuously present, which is the convention the arm states where
it takes the skip; `found_in_string` answering `All` is a full string containment;
`contained_in` answering `Success` is a membership. A matched value FAILS `NOT IN`, so
it cannot be the value a passing `NOT IN` clause passed on, and counting its refusals
credited a passing clause with a refusal belonging to a value that failed.
Measured at predicate level on the case the previous revision deferred: one left value
`["x", 1]` and one right value `["x", 1]`, both `Resolved`, asserting `refused ==
false`. Inclusive answers true -- the subset holds so `contained_in` succeeds, while
`compare_eq("x", 1)` refuses in the element loop -- and exclusive answers false.
WHICH HALF THIS CLOSES, because the prefix does not reach the other one. For an empty
left-hand list the `(List, List)` arm's whole-value loop is a second instance of the same
class: the element loops iterate zero times, so `compare_eq([], entry)` is the arm's whole
contribution and `NotComparable` against an int entry sets `refused` on a pairing the
operator never built. On the QUERIED path this change removes it, because
`membership_stops_after(empty, X)` is true for every non-String X, so any list element is
a stop and an exclusive prefix drops it -- leaving only Strings inside `[0..at)`, and a
String is not a List, so the arm is unreachable. On the LITERAL path it does not, because
`rhs_values_paired_with` returns at `eval.rs:732-734` before `position()` or the endpoint
is consulted, so `[..at]` and `[..=at]` are indistinguishable there. That half needs a
guard in the arm itself and is the following commit.
The `vacuous_match` exclusion in the `(List, right)` arm is NOT subsumed and stays.
Against a STRING the arm does a plain `continue`, which skips the pairing and keeps the
value, so `membership_stops_after` reports no stop and that pairing survives the
prefix. Measured rather than assumed: with the exclusive prefix in place, dropping
`!vacuous_match` reddens three cases of
`a_skipped_pairing_does_not_earn_a_sibling_a_membership_notice`.
`the_membership_notice_stops_pairing_where_the_operator_stops` was the one cell in the
tree pinning the endpoint as inclusive, and it goes red on this change -- measured as
1413 passed, 1 failed, that test alone. It is updated deliberately and the reason is
recorded in its own doc, not adjusted quietly to make the suite green. Two cells are
added to it, at no cost to the suite total, for the empty-left skip in both
directions: against a string the pairing survives the prefix, which is what keeps
`vacuous_match` load-bearing, and against a denylist holding no string the prefix is
empty.
No shape loses a notice, which is the direction of risk for a strict reduction. Swept
196 clauses -- 7 left by 7 right queried operands by 4 spellings -- comparing exit
code, membership-notice count and full report body between a binary at `1b81431c` and
one at this tree, both exported with `git archive` to keep a shared checkout out of it:
0 differing rows, 0 parser-rejected rows, and 38 rows emitting the notice in the base
tree as the positive control, matched by the same 38 here. A sweep whose control is
zero would prove only that it never reached the notice path.
Registry unmoved. Measured against a fixed `git archive` snapshot of
aws-guard-rules-registry at `7f7340c2`, the SHA read out of `.github/workflows/pr.yml`
at the commit under test rather than from a shared working tree: 0 failed rules, 0
unreadable rules files, 0 unrunnable test cases, five deprecation notices, 19 stderr
lines.
`incomparable_membership`'s `(List, List)` whole-value loop mirrored one of the two gates the operator puts on the same loop. `operators.rs:1031` is `rhsl.iter().any(|elem| elem.is_list())`, which the predicate copied. `operators.rs:1054` is `if !flat_subset`, where `flat_subset` is `elements_not_matched(lhsl, rhsl)`'s diff being empty -- and an empty `lhsl` has nothing to leave unmatched, so its diff is empty, `flat_subset` holds, and the operator skips the loop entirely. `lhsl.is_empty()` implies `flat_subset`, so this restores the missing half of a two-part gate rather than carving out a special case. Stated that way because it is checkable. `operators.rs:1029-1030` records the mechanism already, and it is observable without reaching into the module: `Empty IN [[9], 5]` exits 0, which is `contained_in` answering `Success` on a value it compared nothing against. `Empty NOT IN [[9], 5]` exits 19 reporting that `[]` "did match", which is the same fact from the other polarity. What the missing gate cost. For an empty left-hand list the element loops above iterate zero times, so `compare_eq([], entry)` was the arm's entire contribution, and `NotComparable` against an int entry set `refused` on a pairing nothing built. UNGATED, and the first attempt at this fix wrote `both_queried && lhsl.is_empty()`, which is the inverse of the coverage required. Measured, with the exclusive endpoint from the previous commit in place and this guard removed: the queried cell already answers false, because every non-String element is a stop for an empty left-hand list and the exclusive prefix drops it, leaving only Strings in `[0..at)` and a String is not a list. The literal cell answers true, because `rhs_values_paired_with` returns the whole slice at `eval.rs:732-734` before the endpoint is consulted, so no change to the endpoint can reach it. `both_queried` would have skipped the loop precisely where the prefix already covers it and left it running on the only path that needs it. The two changes are therefore disjoint rather than redundant, which is worth stating because redundant guards mean neither is tested. The endpoint closes the queried path; this closes the literal path; removing either one reddens a different cell of the same test. The third cell asserts false, and the first version of it asserted true. That reading came from `operators.rs:1340`, where the literal arm hands `contained_in` the whole left-hand value with no skip above it, and concluded the pairing must exist. It stops at the call without asking what `contained_in` does with an empty `lhsl`. A cell asserting true there would have pinned this defect as intended behavior and blocked the repair, which is worse than no cell. A RESIDUAL of the same class is left open and named in the comment rather than closed. `flat_subset` is also true when `lhsl` is non-empty and every element matched: the operator skips the loop there too and the predicate still walks it. `lhsl.is_empty()` does not cover it. Closing it needs the operator's `diff` at the predicate, which `Comparator::compare` does not carry out -- the same shape as the `own_skip_reason` gap. It is latent by the argument this arm rests on: every element matching means `contained_in` returns `Success`, which is a match, which fails `NOT IN`, so the verdict gate shuts. Re-deriving `elements_not_matched` in the predicate is deliberately not done, because re-deriving what the operator already computed is how this divergence arose. Latency of the defect being fixed, proved rather than searched for. To open this loop `rhsl` must hold a list. Then every other left value in the clause contributes its own refusal -- a non-List value refuses against that list element in the `(left, List)` arm, and a non-empty List value refuses in its own element loop, since no single value is comparable to both a list and a scalar. And the empty-left value by itself fails the clause, because `contained_in` returns `Success`, which is a match, which `NOT IN` denies. So `refused` can never be solely this over-count in a clause that passed, and the notice gate stays shut.
…serve
`the_membership_notice_stops_pairing_where_the_operator_stops` hands
`rhs_values_paired_with` its third argument as a literal `true` or `false`, so it pins what
the helper does with each answer and cannot see which answer the caller supplies. Mutation
testing at the call site, aimed at the single occurrence of
`rhs_values_paired_with(value, &rhs_values, both_queried)`:
both_queried -> true lib suite green, nothing reports it
call deleted, walk &rhs_values lib suite green under `cargo test`; `dead_code` reports it,
but only in a non-test compilation
[..=at] -> [..at] one test red, the endpoint cell
The second row is worth stating precisely, because where the report comes from is the
point. Under `cargo test` the helper is called directly by the test module, so it is not
dead and the run is silent. Under `cargo build`, and under the gate's
`cargo clippy --all-targets -- -D warnings`, the plain lib target compiles without
`cfg(test)`, both this helper and `membership_stops_after` are unused, and the warnings
become errors. Caught by compilation rather than by any assertion, which is a different
kind of protection and not one that survives someone calling the helper from elsewhere.
The `both_queried` row is the losing direction, which is why it earns a cell with no live
consequence behind it. The helper's own doc gives the reason: the literal-right-hand arm
walks every right-hand value with no short-circuit, so truncating there drops pairings it
does build, and a dropped pairing is a dropped refusal -- a notice that was owed and does
not go out. The other direction announces itself as a false warning; this one is silence,
and silence is what nothing was watching.
One value triple, two cells differing only in `Literal` versus `Resolved` on the left, which
is what isolates the argument. The needle `"ab"` is contained in `"xxabxx"`, so the operator
stops there, and the `5` after it is what refuses. Queried, the stop applies and the prefix
excludes it: false. Literal, the walk is not truncated, the `5` is reached: true. Measured
here as green unmutated, red on `both_queried` to `true`, and red on the deletion -- it
catches both, where a cell built on a single right-hand value would catch neither, since a
one-element slice truncated at its only stop and one walked whole differ in nothing a
refusal can be read out of.
It survives the exclusive endpoint rather than depending on it: under `[..=at]` the queried
cell's prefix was `["xxabxx"]` and `compare_eq("ab", "xxabxx")` is `Ok(false)`, so nothing
refused; under `[..at]` the prefix is empty and nothing is walked. False either way, for two
different reasons.
The shape is a peer's, arrived at independently while auditing the same function. Used as
given rather than reinvented, and it is better than the one it replaces: an earlier draft
here used an empty left-hand list against a literal denylist, which catches only the first
mutation and, worse, asserted a refusal that measurement showed is not owed at all.
The single-line narrowing note credited its measurement to `b1ff3845`: "the total at
`b1ff3845`, where the deletions were run". That is impossible, not merely stale.
`b1ff3845` is the commit that BOUND this table. At it the spelling is
`let flag_values: [_; 2] = ["true", "false"];`, so deleting an element gives
`error[E0308]: expected an array with a fixed size of 2 elements, found one with 1
element` -- verified against rustc -- and a deletion that cannot compile cannot have
been run, nor have left a suite green to report. At the parent `a9ad44e8` the table is
the unbound `for value in ["true", "false"] {`, where the deletion compiles. The
attribution moves to `a9ad44e8`.
No figure changes. The ledger at the head of this file gives `a9ad44e8` 3159 and
`b1ff3845` 3159, so 3159 is correct for both and only the name was wrong. The
alternative phrasing "at `b1ff3845` with the annotations reverted" is deliberately not
used: it describes a tree that never existed as a commit, which is a worse claim than
the one being repaired.
The tell was inside the sentence. The spelling it quotes,
`for value in ["true", "false"]`, is the parent's form -- the very thing `b1ff3845`
replaced. A reader checking the quoted code against the named commit would have found
they disagree without needing to compile anything.
Worth recording because it is the reason this class recurs. `b1ff3845`'s own ledger row
names the type annotation as its entire change, and the type annotation is exactly what
makes the deletion fail to compile, so the commit carried the evidence falsifying its
own adjacent sentence one screen apart. `81eed7de` is the commit written to retire
impossible attributions and it introduced this one; the rule it added -- check a delta's
baseline SHA against the parent before the delta means anything -- is what catches it,
and is already recorded at the head of this file.
Comment-only. No test, no code and no count moves.
… of them
`operators.rs` called `is_one_of` "the only place a kind mismatch is discarded on the
`IN` path". Three code-level `Err(_) => {}` arms sit on that path, each directly below an
`Err(err @ Error::RegexError(_))` arm and therefore receiving `NotComparable` and nothing
else: `is_one_of`'s at `:810`, `contained_in`'s whole-list loop at `:1071`, and
`contained_in`'s scalar arm at `:1182`. Four suppressions counting the `(None, None)`
arm's dropped result, which is not an `Err(_) => {}`.
The tree already said so in three places, which is why this ranks above a wrong commit
message. `eval.rs:327-329` and `eval_tests.rs:12199-12200` both enumerate the four.
`operators.rs:761-765`, one screen below the false claim in the same file, says "The
sibling scalar arm below swallows the same error through its own `Err(_) => {}`" and then
"Two sites, one reading." A reader had no way to tell which statement to believe.
Why it happened, because the shape recurs. The claim was written to establish something
true -- that the 143-rule cost belongs to the gate path and not the membership path -- and
reached for a stronger premise than the conclusion needed. "The only place" is not
required for "`EqOperation` reaches neither"; uniqueness got assumed where reachability
was the point. The conclusion is independently confirmed and is now stated as
reachability: `EqOperation`'s impl references none of `is_one_of`, `contained_in`,
`elements_not_matched` or `substring_or_contained_in`.
THE 19 IS RESCOPED, which is the part that changes a decision. It was measured by
promoting `is_one_of`'s arm alone, and the prose read as though it priced the whole
membership fail-closed change. It prices one arm of three. This paragraph is what the
branch's sequencing plan rests on, so a lower bound presented as a total misprices the
deferral. The per-arm figures are now recorded: 27 distinct lib-target tests move for
`is_one_of`, 8 for the whole-list loop, 3 for the scalar loop, 33 for all three.
The union is exact, and that is the load-bearing fact rather than the totals. The union of
the three single-arm sets equals the all-three set with empty symmetric difference, and
inclusion-exclusion closes at 27 + 8 + 3 - 5 = 33 with pairwise overlaps of 5, 0 and 0. No
test needs two arms promoted together and no promotion masks another, so the change can
land one arm at a time with each step's cost known beforehand. The six tests the other two
arms reach and `is_one_of` does not are named in the comment.
Three limits are stated rather than left to be discovered. The denominator is distinct
lib-target tests, not clauses of the 676-clause sweep, so 27 does not convert to 19 and no
converted figure is written down -- the ratio would suggest roughly 23 of 676, which is
extrapolation across populations. `cargo test` stops after the first failing target, so
each promotion run reported one target rather than sixteen. And 33 excludes the fourth
suppression, which needs its own measurement.
Two figures gain the caveat the tree already uses elsewhere. The 19 and the 54 both rest
on a 676-clause sweep that is not committed, and `:1987-1989` already handles exactly this
for a 140-clause grid: treat the number as a note on what was run, and take the argument
beside it as the reproducible half. `80ccdbbc` stated the 19 as "what this change has to
be argued against" with no such caveat, which reads as reproducible.
The 576794 byte count gains its invocation, without which it is a trap rather than a
fingerprint. It reproduces exactly for `test -d rules --output-format json` from the corpus
root, which is `check-registry-corpus.sh`'s own shape; `test -d .` from inside `rules`
gives 576030, the CI path shape gives 581569, and plain text gives 291528. A reader running
it slightly differently would conclude the corpus had moved. Same class as a suite total
with no baseline SHA.
Two smaller repairs in the same block. The registry-free claim is confirmed rather than
assumed -- with `is_one_of`'s arm promoted and the release binary rebuilt, stdout is
byte-identical at 576794, stderr at 4597, five DEPRECATION lines, corpus rc=0 -- and the
comment now says that confirmation is at `1b81431c` while `80ccdbbc` attributed the
measurement to `1ba4648d`, so a reader does not assume one run. And the five DEPRECATION
lines are recorded as five sites carrying the same message, named individually, because
the count alone reads as five distinct deprecations.
Also restores the `///` separator between `:278` and `:279`. `80ccdbbc` consumed the blank
one that divided `IncomparableKinds` from `EngineGaveUp`, so the two rendered as a single
paragraph while reaching opposite conclusions, in the enum doc a reader opens precisely to
tell them apart. `1ba4648d:operators.rs:244` is where the separator sits.
Comment-only. No code, no test and no count moves.
CORRECTION carried in this same commit, because the first draft of it got the cost
structure wrong in the same way the premise above was wrong. That draft recorded the sites
as ADDITIVE and told the reader the change could be landed one site at a time with each
step's price known from the table. That holds for the three `Err(_) => {}` arms and breaks
as soon as the fourth suppression is included. Measured: the four single-site sets union to
49, the all-four run moves 111, and `49 + 65 - 3 = 111` closes on an interaction of 65 with
3 masked.
The mechanism is that `:1182` and `:1691` are in series rather than parallel, and it is
checkable against the source rather than relayed. `contained_in`'s scalar arm ends at
`:1186-1203` with `match (found, unanswerable)`, whose `(false, None)` gives
`Fail(Compare::ValueIn)` and whose `(false, Some(reason))` gives `not_comparable_because`.
Promoting `:1182` sets `unanswerable` and so converts the first into the second; the
`(None, None)` arm's `match contained_in(..)` lumps `Fail(ValueIn)` and `NotComparable`
together in `_ => {}`, and promoting `:1691` records the second. So `:1182` manufactures the
inputs `:1691` acts on, which is the whole of the +46 on that pair.
The comment now says outright that these prices may not be summed, and states the rule
rather than only the numbers: disjoint individual effects do not imply independence. The
`dropped result` and `scalar` sets have an empty intersection and their combination still
moves 46 more tests than their sum. Additivity held across the three `Err(_)` sites because
all three feed the same consumer, and failed the moment a site was added that consumes what
another produces. A comment recording "additive" without recording why is what invites a
reader to extend it to the site where it fails.
Also noted: the fourth site is `_ => {}`, not an `Err(_) => {}`, so it is not reachable by
the promotion form the other three take -- it goes through the `Membership::Unanswerable`
channel the arm already uses at `:1861-1866`. And three tests move under a single promotion
while being green under all four, named in the comment, which is the signature of a verdict
moved one way by one promotion and back by another.
…sult set `operators.rs` stated that "instrumented immediately after `cmp.compare`, every notice-emitting clause carries zero `NotComparable` results, literal-right-hand shapes included, which are what the registry's five are". Measured over the whole suite at `1b81431c`: 291 notice-emitting clause evaluations, 287 with zero `NotComparable` and FOUR with one or more. Both exceptions are literal-right-hand shapes, which is the case the sentence singled out as covered. The four are two clauses appearing once per build target, both `some`-quantified: `some Multi.*.V NOT IN [/(?!x)((a+)+)b/]` at `eval_tests.rs:12831` and `some WithNonString[*] NOT IN "abc"` at `eval_tests.rs:11463`. The second is the most direct instance of a pattern this round has retired four times: that case reads `#[case::a_refusing_element_under_a_literal_operand(.., true)]`, and the trailing `true` is the notice-expectation flag, so a committed test asserts the notice fires for a shape the comment said could not happen. Nobody connected the two. `some` is the entire mechanism, which is why it is exactly these two. The gate is read under the query's own `match_all` -- `clause_passed(&outcome, match_all)` -- because "did this clause pass" means one value for `some` and every value otherwise. Under `some`, one passing value carries the clause while a sibling's comparison sits in `NotComparable`, so the predicate fires, the clause passes, the notice goes out, and the result set still holds the refusal. Under `all` the refusing value sinks the clause and nothing is emitted. That is the 287. REPAIRED TO THE STRONGER FORM RATHER THAN DELETED, which is the point of the commit. "The incomparability is not recoverable from the result" holds for 287 of 291. A result-reading implementation would work for precisely the two `some` shapes and be blind for the other 287 -- a partial-coverage trap rather than a clean impossibility, and worse to ship than either, because it appears to work on whichever shapes someone happens to test. The downstream note on `membership_stops_after` depends on this and is better supported now than it was by a universal that does not hold. The registry half stands: the pinned corpus is clean, 25 notice-emitting evaluations across its 5 clause sites, all with zero `NotComparable`. Only the generalization beyond the registry was wrong. Provenance recorded rather than smoothed over, because the first pass was believable and wrong. It reported 58 notice-emitting evaluations and 2 candidates with `NotComparable` > 0, both undercounts: libtest writes `test <name> ... ` to stdout with no trailing newline, so under `2>&1` the first `eprintln!` of each test is glued onto that partial line and a `^INSTR_` line anchor misses it. The tell was arithmetic rather than semantic -- two probe sites with an identical guard on the identical `usize` reported 0 and 52, which cannot both be true. Unanchored, both read 52. The committed figures are from the unanchored recount. The positive control is what makes "287 carry zero" a measurement rather than a dead counter: a variant printing for every clause found 712 of 11084 evaluations carrying `NotComparable` > 0, firing on the expected kind mismatches. That 712 is from the anchored run, so it is recorded as a floor rather than an exact count -- writing it down without that caveat would reintroduce the error this commit fixes. One citation is by name rather than by line number, deliberately. A `file:line` in a comment is invalidated by the next insertion above it, which has already happened twice on this branch. Comment-only. No code, no test and no count moves.
`1b81431c`'s message says the corpus assertion "fails on this host with the same two manifest byte-count mismatches recorded last round ... so it is environmental and unmoved by this change". That is wrong in the way that matters: the failure is a deliberate signal that the corpus checkout has moved ahead of the pinned SHA, and `pr.yml` says so two lines from the pin the claim was written beside -- "Fixing the corpus turns this red on purpose: when aws-cloudformation/aws-guard-rules-registry#288 merges, the pin and those lists move together, and the lists become empty." Measured, from an isolated `git archive` of the pinned `7f7340c2`. At the pin the assertion PASSES: rc=0, no `file sizes:` line anywhere in the log, "30 unchecked expectations across 11 rule names, all of them expected", "0 failed rules, 0 unreadable rules files, 0 unrunnable test cases", "3 orphaned test files, all of them expected". At `b9fc1eb7`, four commits past the pin, rc=1 with exactly two `file sizes:` lines, both reporting an actual of 0. Exactly two, which is what makes it a fingerprint rather than an anecdote: `byte_evidence` has exactly two call sites, one per checked-in list, and both lists go empty when the corpus lands the fix. So the count is structural, not incidental. The note goes in the script's own header rather than in an evaluator comment, because the script is what a person runs when this goes red, and the first question they will ask is whether their environment is broken. Why the wrong diagnosis was plausible, which is the reason the note is worth its space: the vocabulary is genuine. The script really does print expected-versus-actual byte counts, and that reads exactly like a truncated download or a bad checkout. Nothing about the output says "your corpus is newer than the pin" -- that has to be inferred from the pin, which is in a different file. This is the third time this round that the text falsifying a claim sat adjacent to it in the same tree, and the count is the point rather than the anecdote: a claim written beside the evidence against it is the failure mode, not bad luck.
…laim Five repairs in the ledger and its neighbours, none of which moves a count. The single-line chain's third term was forty-four and is forty-five. Measured per commit, the annotated-table population runs 20, 20, 32, 33, 33, 45, 45, 45, 45 and 46 at `1b81431c`, with zero unannotated throughout, so the chain's first two terms hold and only the third did not: thirty-three plus the twelve is forty-five. Forty-four is the exact figure `81eed7de`'s own ledger calls "nobody's measured figure", 128 lines from where that commit corrected it and left standing here. `81eed7de`'s "One in-tree instance" undercounts as well -- two sat in a single doc comment. The hardcoded population in the corroboration note is removed rather than updated. It read "1 of 45 tables carry no length", which was already 46 by `1b81431c` and moves again whenever a test adds a table -- this commit's own branch added one. A figure the next commit invalidates is a maintenance trap wearing the costume of precision, and quoting it in the sentence that tells the reader to ask the test instead was the instruction contradicting its own example. The mechanism is cited now and the number is not. `1ba4648d`'s wrong-delta row claimed a shape "no earlier row in this table does". False. `ec2b9a5c` states 3087 against a real 3103, having added its own delta to `69628df7`'s 3085 while skipping `e26817a6`'s +16 in between -- the same assumed adjacency one row up, already annotated as stale in the same table. Narrowed to the claim that survives: `1ba4648d` is the first row whose wrong figure is a delta rather than a total. The novelty is deliberately not argued from author dates, which survive rebases and are non-monotonic across `ec2b9a5c` and `f2f87b82` anyway. Two of `1b81431c`'s own claims are recorded here because a pushed message cannot be amended. "Both existing cells" is two grep hits, not two cells: the assertion sits at two sites but as five loop cells across two tests, `an_empty_selection_skip_says_which_query_matched_nothing` with a `[_; 4]` table and `a_type_block_skip_names_the_cause_it_can_support` at cell 2 of 4. "Both existing tests" is right and "all five existing cells" is right; "both existing cells" is neither. Written down as a class rather than an incident because it is the third grep-hits-as-cells slip on this branch, and the mechanism is always the same -- `grep -c` counts lines while a table-driven test's coverage is counted in cells. And "3159 is `a9ad44e8`'s total two commits later" is off by one under either reading. The chain is `d4286e68` -> `d8c2582d` -> `e2cacdc4` -> `a9ad44e8`: three after the SHA named, one after the commit being corrected. The true "two" relation is the one the same message states correctly elsewhere, `d4286e68` being two commits before `e2cacdc4`, so a number was carried from a neighbouring true claim onto a different pair. Left alone deliberately: "The per-commit chain is 3149, 3151, 3159, 3159, 3161" is exactly right read as starting at `d8c2582d`. Imprecise about its own starting point, derivable, and not a defect -- rewriting correct-but-terse prose while retiring wrong prose only makes the diff harder to check. Comment-only. No test, no code and no count moves.
…t by its slice
The exclusive endpoint in `rhs_values_paired_with` had a cell for its mechanism and none
for its consequence. `the_membership_notice_stops_pairing_where_the_operator_stops` asserts
which right-hand values are walked; nothing asserted what `incomparable_membership` answers
for the pairing at the endpoint. A behavior change whose most direct expression has no cell
is what lets a later change revert it with the suite still green.
One left value `["x", 1]` against one right value `["x", 1]`, both `Resolved`, asserting
`refused == false`. `contained_in` answers `Success` because the subset holds, so the value
MATCHED and fails `NOT IN` -- it cannot be the value a passing `NOT IN` clause passed on.
Its element pairs refuse anyway, `compare_eq("x", 1)` being `NotComparable`, and under the
inclusive `[..=at]` the predicate counted that refusal for a clause it did not belong to.
Proved against the mutation rather than asserted: with `[..=at]` this cell fails, with
`[..at]` it passes. That is the endpoint change expressed as the predicate's own answer
instead of as a slice of paths.
It is also the exact case an earlier revision of that function's doc named and deferred, on
the reading that dropping the stopping pairing's accounting was "a separate question with
its own measurement owed". This is the measurement, and the deferral is closed.
Suite 3167 to 3169: one test function, which cargo builds into both the lib and main
unit-test targets.
… that it was latent `incomparable_membership`'s `(List, List)` arm mirrored one of the two gates `contained_in` puts on its whole-list loop, and the previous commit closed only the empty-left half of the second one. The residual it named and left open is live. `operators::whole_value_pairing_built` is that second gate, asked of the operator's own `elements_not_matched` rather than re-derived. THE LATENCY ARGUMENT WAS FALSE, and it is why this survived review. It ran: every element matching means `contained_in` returns `Success`, which is a match, which fails `NOT IN`, so the verdict gate shuts. That is per-value and the gate is not. `clause_passed` reads the query's own `match_all`, and under `some` it is `any(status == PASS)`, so the matched value's own FAIL does not shut it. A passing sibling holds it open while the matched value's refusal sets `refused`. Measured on release binaries, `Iso` of `[[[1]], [[7], [8]]]`: some Iso[*] NOT IN [[1], [9]] exit 0, notice -> the defect some IsoNoRes[*] NOT IN [[1], [9]] exit 0, silent -> attribution some IsoOnlyRes[*] NOT IN [[1], [9]] exit 19, silent -> sole value Resources.*[ some Props.V[*] NOT IN [[1], [9]] ] exit 0, notice -> filter form The notice's subject is `[[7], [8]]`, every element pair of which is int against int and decided false, so the stated reason is false about the value that actually passed. The third row is the population the latency argument was checked against: with only the matched value the clause fails and the gate really does shut. The fourth is the spelling the aws-guard-rules-registry rules use, which is what makes this user-visible. After the fix all four exit codes are unchanged and rows one and four are silent. Its own precedent is one arm down. The `(List, right)` arm records the same failure mode about its own repair -- an argument true of a left operand whose ONLY value is the empty list, and false as soon as that value has a sibling, because `refused` ORs over the whole cross product. Substituting "every element matched" for "the empty list" gives this finding verbatim. `!lhsl.is_empty()` BECOMES REDUNDANT RATHER THAN REMOVED. An empty left-hand list leaves an empty diff, so the general condition already answers false for it. That is what the two-part gate argument predicted, and it is the cleaner statement of it. Keeping both would leave neither tested. IT CALLS `elements_not_matched` INSTEAD OF RESTATING IT, which is what the prohibition on `membership_stops_after` requires. It is also precisely the accessor the arm's own comment called unavailable -- that comment said closing the residual "needs the operator's `diff` at the predicate, which `Comparator::compare` does not carry out". The diff was six lines and one `pub(super)` away in the same module. The unavailability claim was wrong, not merely pessimistic, and it is what made the residual look unclosable. COST, stated rather than hidden: `elements_not_matched` now runs a second time on the notice path, allocating its diff again. The predicate already accepts extra comparisons for `NOT IN` alone; this doubles that work within the path. The cheap `any(is_list)` is tested first so the second call is skipped on every flat denylist, which is the opposite operand order from the condition being restored but the same answer. THE ARM STILL OVER-COUNTS. A second divergence of the same class is open and this does not close it: the element loops walk the full cross product, while `is_one_of` returns on the first match, so later pairings for a matched element are never built. With `ShortIso` of `[[[1]], [[7, 7], [8, 8]]]`, `some ShortIso[*] NOT IN [[1], ["a"]]` still exits 0 with the notice after this commit, while `some ShortIso[*] NOT IN [[1], [9]]` -- the same left operand -- goes silent. The discriminator is whether the skipped pairing would have compared incomparable kinds: `compare_eq(1, 9)` is `Ok(false)` and refuses nothing, `compare_eq(1, "a")` is `NotComparable` and refuses. One fixture answering oppositely across two denylists is why the clause shape cannot be used to tell the two sources apart, and why the surviving divergence is named in the comment rather than left for the next reader to rediscover. It is sequenced second because closing it alone moves nothing observable -- every clause where it matters also has the residual firing, and the residual holds the notice up -- and because it needs a different shape: an accessor for the pairing set `is_one_of` actually built, not a copy of its walk. Verdict-neutral over 23,496 clauses: notices 12,774 to 12,772, the two removals being `some Iso[*] NOT IN [[1], [9]]` and `some ShortIso[*] NOT IN [[1], [9]]`, 0 gained, 0 exit-code movement, 0 stdout movement. Registry corpus byte-identical on both streams with the same five DEPRECATION lines. Suite 3169 to 3177, four cells per build target, debug equal to release. Mutation testing on the lib target, nothing failing unmutated: removing the second gate entirely reddens 2, reverting it to `lhsl.is_empty()` reddens exactly the new cell 1 and nothing else, inverting the helper reddens 6 including the new cell 4, forcing it true reddens 2. The second row is what says the residual had no coverage before this commit; the first is what says the empty-left coverage survived being subsumed.
…comment carried Every `operators.rs` line number in the `IN`-path discard comment was stale, all in the same direction: the file grew by insertions above each cited construct. `:810` names an arm now at `:973`, `:1071` one at `:1234`, `:1182` one at `:1345`, and `:1186-1203`, `:1691`, `:761-765`, `:768-769`, `:1861-1866` and `:1987-1989` had all moved too. The two `eval_tests.rs` citations were stale by exactly +28. All of them are replaced by construct names rather than by corrected numbers. A corrected number in a file that later commits insert into is a citation with a fuse on it, and this comment is proof: every one of the ten was right when written. The `operators.rs:1241` to `:1354` case already in this tree is the model. Four figures beside them: corpus rc=0 was three figures from the guard invocation plus one from the script, listed as though one exit status covered all four. `cfn-guard test -d` over the pinned corpus exits 1, which is the pinned known state -- 30 expectations name rules their `.guard` file never declares, an unevaluated expectation makes the error code sticky, and the same run reports 0 failed rules. `check-registry-corpus.sh` asserts that whole state and answers 0. A reader who checks "rc=0" against the binary gets 1 and concludes the corpus moved. baseline 1414 passed is not the lib count at the commit the prices were taken at, nor at this one. Replaced with "no test fails unmutated", which is what the prices actually rest on and what does not re-break when a test is added anywhere in the crate. The combination table's predicted column is the UNION of the single-site sets and never said so. Two of its five rows are numerically identical to sums -- 26 is both the union and 18 + 8, and 21 is both the union and 18 + 3 -- so a reader checking the column by adding finds those two agree and reads the other three as arithmetic errors rather than as overlaps. Summing instead of unioning is the mistake the surrounding paragraph exists to warn against. The positive control caveated its numerator as a floor from the anchored run and left its denominator reading exact. The anchor that dropped notice-emitting lines dropped them from the denominator's tally too, so both are floors: "712 of at least 11084". Comment-only. Lib 1421, unchanged.
… and fold five asserts into one Three corrections to how the `rhs_values_paired_with` tests describe themselves. None of them changes what is asserted. THE FIRST CELL OF THE EMPTY-LEFT TEST IS OVER-DETERMINED. Its comment said "the first cell holds by the prefix and the third by the guard in the arm", which reads as one test covering both paths. Measured by reverting the endpoint to the inclusive `[..=at]`: `an_empty_left_hand_list_earns_no_whole_value_refusal_the_operator_never_built` stays GREEN. With the pairing restored the gate in the arm answers false for an empty `lhsl` anyway, because its diff is empty either way, so cell 1 flips only when both changes are reverted and isolates neither. What reddens under `[..=at]` is `a_subset_that_holds_earns_no_refusal_from_its_own_ element_pairs` and `the_membership_notice_stops_pairing_where_the_operator_stops`, neither of which is in that test. Both changes are bound; the binding is across tests rather than within one, and the comment now says so. THE CALL-SITE ROW WAS HISTORICAL AND READ AS CURRENT. It said replacing `both_queried` with `true` leaves the whole lib suite green, that deleting the call does too, and that only the endpoint mutation was caught by anything. All three describe the tree before `only_the_two_query_arm_truncates_the_denylist_walk` existed -- the tree its own commit changed -- so the sentence was falsified by the commit that shipped it. Re-measured here, and with the invocation named on every row, because "the suite" is not one thing: both_queried -> true --lib 1 red: that test call deleted, walk &rhs_values --lib 2 red: that test and a_subset_that_holds_.. call deleted, walk &rhs_values --workspace same 2 red, plus 3 dead-code diagnostics call deleted, walk &rhs_values clippy exit 101, no test runs Under `--lib` the helper is called by the test module, so it is not dead and the two reds are assertions. Under `--workspace` the plain lib target also builds as a dependency of `guard-ffi` and `guard-lambda` without `cfg(test)`, where it is genuinely unused. Under the gate's clippy invocation those diagnostics are errors. The deletion is caught three ways and only one is an assertion. FIVE SEQUENTIAL ASSERTS BECOME ONE TUPLE COMPARISON, which also settles a convention split: the test carrying the note "answered into one comparison so a run reports all cells instead of stopping at the first" was not the one that needed it. With the sequential form a cell 1 failure hid cells 4 and 5, so whether cell 5 was masked by the form or subsumed by cell 4 could not be answered by running the test. It is subsumed: its conditions are a strict subset of cell 4's, both walking an empty left-hand list with `both_queried` true, and cell 4's denylist holds the Int that cell 5's holds alone. No mutation isolates it, so it is labeled as illustrating the argument rather than binding it, and it stays because the argument's endpoint is worth asserting rather than inferring. Stale `operators.rs` and `eval.rs` line citations in these comments are replaced by construct names. Comment and assertion form only. Lib 1421, unchanged.
…n was missing Two figures in `every_loop_table_carries_its_length`'s comment. "Forty-four tables carry a length today. Nothing made the forty-fifth carry one." Forty-four is the figure the single-line section 22 lines below retires to forty-five, so the paragraph disagreed with its own page. Measured at this commit by the rule the test uses: 47 examined, 0 unannotated. The per-commit chain in the same comment records 0 unannotated throughout, which is what the assertion at the foot of the function has required all along, so there has never been an unannotated forty-fifth to make carry one. The framing undercounts the change's own scope as well. Its commit is subject-lined "make the loop-table length guard extend itself", and a guard that extends itself covers every table there will ever be; describing the gap as one missing forty-fifth sizes it at one. Replaced with the durable form the same comment already recommends two paragraphs down -- ask the test, whose failure message states the count it walked. The 20, 32, 45 chain is missing 33. Two groups of twelve cannot produce a step of thirteen, and the term is supplied six lines further on in the per-commit run where it is easy to read past, so a reader checking the arithmetic finds one group the wrong size and no cause. Added inline, and deliberately not attributed to a SHA: the ledger at the head of this file already maps the population per commit, and a fourth coordinate pointing at one fact is the citation class these corrections keep retiring. Comment-only. Lib 1421, unchanged.
…ly built
The second divergence of the class the previous commits have been closing, one granularity down.
`incomparable_membership`'s `(List, List)` arm walked the full `lhsl` x `rhsl` cross product.
The operator does not: `elements_not_matched` asks `is_one_of` per left-hand element, and
`is_one_of` returns `Matched` on the first right-hand element that matches, so the pairings
after a match are never built. The predicate counted them.
Pre-existing at `1b81431c`, so not a regression of this range.
`operators::membership_pairing_refused` replaces the nested loops, and the shape is the whole
point: it runs `is_one_of`'s OWN loop and reports what the pairings that loop built said.
`is_one_of` becomes a thin wrapper over `is_one_of_observed`, which is the same body with the
kind refusals it discards also written to a caller-supplied flag. One copy of the walk, not two.
The alternative was to copy the stop condition into the predicate --
`if right == left || compare_eq(left, right) == Ok(true) { break }` -- which fixes the symptom
by reproducing the cause. That is what the prohibition on `membership_stops_after` forbids and
what produced both divergences. Rejected on that ground, not on size; it is the smaller diff.
The classification cannot drift from the `pair_refused` it replaces. The flag is set on
`is_one_of`'s `Err(_)` arm, which receives everything except the `RegexError` taken by the arm
above it, and that is exactly the partition `pair_refused` makes. The returned `Membership` is
untouched, so a kind mismatch still does not become `Unanswerable` -- promoting it is the
fail-closed change gated on the registry rules, priced at 19 verdicts, and this is not it.
WHY IT IS THE SECOND COMMIT AND NOT THE FIRST. Alone it removes 0 notices of 12,774: every
clause where it matters also had the whole-value residual firing, and the residual held the
notice up. It has no before-and-after of its own until that gate lands. With the gate landed it
removes one more.
Notice level, the lens's 23,496-clause grid, head at `f71e1157`:
residual gate only 12,774 -> 12,772 Iso[*] ..[[1],[9]], ShortIso[*] ..[[1],[9]]
both 12,774 -> 12,771 and ShortIso[*] NOT IN [[1], ["a"]]
0 gained, 0 exit-code movement, 0 stdout movement in both runs.
The two sources are separable because each alone is sufficient to set `refused`, and one fixture
shows it: with `ShortIso` of `[[[1]], [[7, 7], [8, 8]]]`,
`some ShortIso[*] NOT IN [[1], [9]]` needed only the whole-value gate while
`some ShortIso[*] NOT IN [[1], ["a"]]` needed both. The discriminator is whether the skipped
pairing would have compared incomparable kinds: `compare_eq(1, 9)` is `Ok(false)` and refuses
nothing, `compare_eq(1, "a")` is `NotComparable` and refuses. The clause shape does not show
which source is live, which is why the fixture carries both denylists.
PREDICATE LEVEL, because the notice count understates the reach and nothing measured it before.
From one instrumented binary computing the predicate under both policies and reporting both, so
a change the verdict gate hides is still visible. Every clause reached the predicate -- 0 with
no probe line -- so the denominator is complete:
population clauses head=true the fix moves
sweep3 grid 14,700 11,685 60
the wider grid with ShortIso 23,286 19,323 117
So 117 clauses answer differently at the predicate and one at the notice. That gap is the
verdict gate, and it is the reason this could not be priced from notice movement alone. The
11,685 is measured with the whole-value gate already landed; the same grid read 11,694 before
it, and that 9 is that gate's own predicate-level reach.
Mutation testing on the lib target, nothing failing unmutated: reverting this to the nested
loops reddens exactly the new `a_skipped_pairing_across_kinds` cell and nothing else, which is
what says the suite could not see this before. Making the observer flag dead -- `Err(_) => {}`
again -- reddens 11 tests, which is what says the helper carries the refusals the nested loops
carried rather than quietly reporting nothing.
The new cell 2 is the discriminator and was already green: same left operand, comparable skipped
pairing. It is here so a reader cannot conclude the fix is about the fixture rather than about
the kinds.
Registry corpus byte-identical to the pre-fix binary on stdout and stderr, same five DEPRECATION
lines, script 0 and guard run 1 as pinned. Suite 3177 to 3187, five cells per build target,
debug equal to release. Clippy clean at 156 units in a fresh target dir.
…dicate Three arms of `incomparable_membership` carried one divergence -- the predicate walking further than the operator did. Two were closed; this closes the third, on the `(left, List)` arm, with `operators::membership_pairing_refused(std::slice::from_ref(left), rhsl)`. The operator's walk for a scalar IS `is_one_of` over the same slice, so this is the same shared-walk shape as the element-loop repair: run the operator's loop and read what the pairings it built said, rather than restating the stop condition. LATENT AND REPAIRED ANYWAY, which is `membership_stops_after`'s own rule, followed rather than re-decided. That note reads: "LATENT, AND REPAIRED ANYWAY ... the suppression lives in a different function from the divergence, though, so changing what either short-circuit fires on -- or adding a third beside them -- makes it live with nothing in the tree to flag it." Verbatim true here: the verdict gate in `binary_operation` is what suppressed it, and the gate is not in this function. Fixing two arms and documenting the third would have left the module with two rules instead of one. The earlier decision to record rather than repair rested on "a change with no demonstrable before-and-after is a bad shape to review". That objection was about the element loop, where the zero was TEMPORARY -- the whole-value residual masked it, and landing that first made it demonstrable. Here the zero is structural, so waiting for it to become demonstrable is a decision never to fix it. The objection dissolves rather than being overridden, because a predicate-level test IS a before-and-after a reviewer can check; it is just not at the notice. THE MEASUREMENT, which is what the repair rests on. Over a 23,496-clause grid, closing this arm moves 66 predicate answers and ZERO notices: 12,771 before and after, 0 gained, 0 exit-code movement, 0 stdout movement. The predicate figure comes from an instrumented binary computing the arm under both policies in one run, so a change the gate hides is still counted, and every clause reached the predicate, so the denominator is complete. Why the gate caught all 66. The over-count needs a denylist entry that MATCHES the left value, or the walk does not stop early, and a later entry INCOMPARABLE to it, or the skipped pairing refuses nothing. A matched value fails `NOT IN`, so only a sibling can carry the clause -- but `compare_eq`'s comparability splits roughly into a numeric class and a textual one, and mixing them is what makes the skipped pairing refuse, so a sibling comparable to the offending entry sits in the other class from the matched entry and refuses against that instead. That argument explains the measurement rather than standing in for it. It is the shape of argument that was wrong twice on this branch -- the whole-value residual was called latent on a per-value reading of the gate, and the `(List, right)` arm records the identical error about its own repair. Six hand-built attempts at a non-refusing sibling all kept the notice, which is evidence and not proof. "Latent" was a property of the measured population, and that is why this is repaired rather than left resting on it. `a_short_circuited_scalar_pairing_earns_no_refusal` binds it, at the predicate because a cell watching a notice or an exit code would pass with the defect in. Proven red before landing: the fix call is unique in the file, the mutation was aimed at its line and content-checked before being applied, and reverting it to the nested loop reddens exactly that test, cells one and two flipping false to true while cell three holds. Cells one and two are the two ways `is_one_of` returns `Matched` -- `elem == each`, and `compare_eq` answering `Ok(true)` for an `Int` of 7 against a `Float` of 7.0 -- so a fix keyed on `PartialEq` alone cannot pass. Cell three is `99`, which matches nothing, so the operator walks the whole denylist and its refusal is owed; a wholesale suppression of the arm reddens there. Also folds in the element-loop divergence's provenance, into the TREE rather than a commit message: pre-existing at `1b81431c`, three lines above the line that range changed, not a regression of it. That sentence lived in a message, a history rewrite dropped it, and a reader opening `eval.rs` has the tree and not the log. Cited by construct, because the coordinates for that arm have already moved once this round. Suite 3187 to 3189, predicted and actual, one test function per build target; lib 1426 to 1427. Debug equal to release. Clippy clean at 156 units in a fresh target dir. Registry corpus byte-identical to the pre-fix binary on stdout and stderr, five DEPRECATION lines, script 0 and guard run 1 as pinned. All four original reproducers still silent with their exit codes unmoved.
The residual paragraph recorded 0 of 12,771 notices moving for the `(left, List)` over-count and did not record why that zero is a measurement. A run answering "no movement" because it observed nothing is indistinguishable from one that observed everything and found none, and the distinction is the whole weight of the latency claim. Two hand-checked clauses separate them, and both are now named. `Uint NOT IN [7, [9]]` over a `Uint` of 7 reported the two policies DISAGREEING, so the probe could see this arm at all. `some Mixed[*] NOT IN [7, [9]]` over `[7, 99]` reported them AGREEING, because the sibling `99` refuses against `[9]` on its own account and the over-count changes nothing there. Without the first, 0 of 12,771 would prove only that the probe was blind. Comment-only: zero non-comment lines change, so the suite cannot move. Gated anyway, because a comment-only change can still move `cargo fmt`, `typos`, and rustdoc doc-tests -- this branch has already taken a doc-test failure from a four-space indent inside `///`, which makes rustdoc compile the block as Rust.
`incomparable_membership`'s `(List, List)` arm walked every denylist entry against the whole left value. `contained_in`'s counterpart does not: its whole-list walk breaks on the first match, at `PartialEq` equality or at `compare_eq` answering `Ok(true)`. So the predicate counted pairings the operator never compared -- the fourth site of the class this branch has been closing, and the one the previous round missed. The previous round replaced this loop's GATE and left its BODY. Turning `!lhsl.is_empty()` into `whole_value_pairing_built` was right on its own terms -- it observes `!flat_subset` instead of re-deriving it -- but a guard that observes correctly can still sit in front of a walk that does not, and that is what shipped. Rather than add a fourth observer, `contained_in`'s own whole-list walk is now a call to `is_one_of`, which the predicate already calls through `membership_pairing_refused`. There is one walk where there were two, so the two cannot drift; the previous three sites were aligned by discipline and this one is aligned by construction. WHY THE SCALAR ARM'S LATENCY ARGUMENT DOES NOT COVER THIS. That argument rests on `compare_eq`'s comparability graph having no edge between its numeric component and its textual one, so a sibling comparable to the offending entry refuses on its own account. The `(List, List)` arm bridges them: a length mismatch answers `Ok(false)` before element kinds are ever examined. A length-1 sibling is therefore comparable to every length-2 entry, contributes nothing, and leaves the suspect's phantom refusal to carry the notice alone. Measured live before the fix, with all three controls. `Whole` of `[[[1],[2]], [[9]]]` against `NOT IN [[[1],[2]], ["a","b"]]` under `some` printed the notice at exit 0; the sibling alone and the suspect alone were both silent, and the suspect alone exited 19 -- which is the fact that makes the notice false, since the only value that could not be compared is the one that failed. The registry filter form printed it too. The new cell is discriminating on both of the operator's early returns. `[[1.0], [2.0]]` against `[[1], [2]]` is not `PartialEq`-equal but `compare_eq` zips it and answers `Ok(true)`, so a fix keyed on `PartialEq` alone reddens; and a third cell whose denylist matches neither entry proves that skipping the walk wholesale cannot pass either. Every element pairing in all three cells answers `Ok(false)` on a length mismatch, so nothing but the whole-value walk can set them.
The cell 2 that shipped in the previous commit does not discriminate what its own assertion text
claims. Two separate mistakes, one on top of the other, both caught by mutation rather than by
reading.
FIRST, IT MEASURED THE WRONG WALK. It paired `[7]` against `[[7.0]]`, which is `(Int, List)` and
refuses at ELEMENT granularity, so the cell was set by the element loop above rather than by the
whole-value walk it is named for. It passed, and it passed for a reason unrelated to the fix.
SECOND, ITS REPLACEMENT DISTINGUISHED NOTHING. Swapping in a `Float` entry to reach the walk's
`compare_eq` early return does not separate the two returns, because `PartialEq` has no
`(Int, Float)` arm either -- so `1` against `1.0` falls through to `compare_eq` from BOTH cells and
cell 2 became a second copy of cell 1. The intended discriminator was along for the ride.
Single-point RANGES are the asymmetry that works: `PartialEq` covers `(RangeInt, RangeInt)` and
nothing relating a range to a scalar, while `compare_eq` relates an `Int` to a range containing it.
So `[r[1,1], r[2,2]]` matches `[[1], [2]]` through `compare_eq` alone.
Measured both ways, on the parent commit and on this one, by replacing the shared walk's
`Ok(true) => return Membership::Matched` with `Ok(true) => {}`:
parent, `Float` cell 18 tests red, and this test is NOT among them
this commit, range cell 19 tests red, including this test
The 18 are the seventeen `a_range_in_a_list_denylist_denies_a_list_valued_property` cells and one
operator test, which already bound that early return -- so the parent's coverage of it was real but
came from elsewhere, and this test contributed nothing to it. The nineteenth is the point of the
change.
All three cells now share one left value and differ only in the denylist, and every ELEMENT pairing
in all three is `Ok(false)` on a length mismatch, so a `true` cannot have come from the element
loop. That property is what the first mistake violated, and stating it is what makes the cells
checkable rather than merely green.
The comment records both wrong turns with their measurements, because a reader who sees only the
range cell would reasonably reach for a `Float` first.
Test fixtures and comment only; no source change, and `git diff` against the parent touches one
file. Suite 3191 and lib 1428, both unchanged -- the cell count does not move, only what the cells
exercise. Debug equal to release, clippy clean at 156 units in a fresh target dir, registry corpus
byte-identical to the parent on both streams with the same five DEPRECATION lines.
…irings `incomparable_membership`'s `(List, right)` arm recorded a `pair_refused` per left-hand element unconditionally. For most right-hand kinds no such pairing exists: `contained_in` dispatches on the left value, so a list against a non-list reaches its `List` arm's catch-all and answers `NotComparable` carrying the two WHOLE values. Every element then contributed a refusal for a comparison nobody made, and the notice went out naming a reason that was false about the clause. With `Mixed` of `[["a"], 7]`, `some Mixed[*] NOT IN 5` exits 0 and printed the notice. `["a"]` is the only value that could not be compared and it FAILED -- alone it exits 19 with `Can not compare type ... Value=["a"], Value=5`, the whole values -- while `7` passed on an ordinary `Ok(false)`. The sibling alone is silent, which attributes it. NOT the class the four repairs above it closed. Those were the predicate walking FURTHER than the operator, a loop counting pairings a `break` had already skipped, each closed by asking the operator how far its walk went. `pair_refused` is correctly aligned here and this loop stops where it should. What was missing is the condition selecting which pairings exist at all. Worth keeping distinct: reading this as more of the same suggests another stop condition, and a stop condition fixes nothing here. `operators::element_pairings_built` is that condition, asked of the same `operators::is_literal` the dispatch uses. It is NOT `both_queried`: that flag names one of `InOperation::compare`'s four arms and collapses the other three, and two of the three it collapses do decompose. `(None, Some)` against a `String` expands the left-hand list into one `string_in` per element, and `(Some, None)` with no list among the right-hand values builds an element-wise `Vec::contains` diff. Both are refusals the operator really makes, so both are notices owed. A gate spelled `both_queried || matches!(right, String(_))` was measured and rejected. It is wrong in both directions: it drops the `(Some, None)` diff, which is silence where a notice is owed, and keeps `(Some, Some)` against a `String`, which is a false alarm. Over a 30,132-clause grid, 6,750 clauses reach the arm and that spelling drops 222 predicate answers it should keep while keeping 84 it should drop, 6 of the 84 live notices. Measurement. Grid: this condition removes 258 predicate answers, 8 of them observable notices, 0 gained, 0 exit-code movement. All 8 are whole-value refusals -- the message names the operands, e.g. `Value=[1,2]` against `Value="a"`. Registry corpus at pin 7f7340c2 under `test -d`: stdout 304,068 bytes and stderr 5,737 bytes byte-identical, the same 5 notices before and after. Positive controls: removing the loop entirely moves 1,110 answers and 26 notices, so the grid reaches it; and 0 both-queried clauses depend on the loop, confirming the shape refusal below subsumes that arm.
…nnot pin
Three things, all in eval_tests.rs, none of them behavioral.
1. THE RECORDED REASON WAS INVERTED. `the_whole_value_pairing_stops_where_the_operator_
stops` said `PartialEq` has "no `(Int, Float)` arm", so `1` against `1.0` reached
`compare_eq` from both cells. Backwards: `PartialEq`'s catch-all delegates to
`compare_values`, which carries explicit `(Int, Float)` and `(Float, Int)` arms, so
`Int(7) == Float(7.0)` is TRUE through `PartialEq` -- measured directly. The `Float` cell
stopped at the FIRST early return, not the second.
The rule to pick a discriminator by is correspondingly opposite, and that is why this is
worth correcting in the tree rather than leaving in a pushed message. "Choose a type
`PartialEq` lacks an explicit arm for" selects nearly every scalar pair and leads a reader
straight back to `Float`. The rule is: choose a pair `compare_values` REFUSES that
`compare_eq` relates by some other route. A single-point `RangeInt` against an `Int`
qualifies -- `compare_values` will not order a range against a scalar, `compare_eq`
answers range membership.
2. THE SECOND INSTANCE OF THAT DEFECT. `a_short_circuited_scalar_pairing_earns_no_refusal`
still carried the `Float`/`Int` pair for the same purpose. Measured on it as it stood:
deleting either early return left it GREEN, while the positive control -- making the `Err`
arm record no refusal -- reddened cell 3. Its cells 1 and 2 bound NEITHER return. Cell 2
now uses a single-point `RangeInt`, and deleting `Ok(true) => return Matched` reddens it at
cell 2, as it already did for the sibling.
3. WHAT THE CELLS CANNOT PIN, WHICH IS A PROPERTY OF THE OPERATOR. Measured over every
explicit arm of `PartialEq` plus its fall-through -- `(Map, Map)`, `(List, List)`,
`(Bool, Bool)`, `(String, Regex)`, `(Regex, Regex)`, the three range arms, `(Null, Null)`,
`(Char, Char)`, `(Int, Int)`, `(Int, Float)` -- wherever `PartialEq` answers true,
`compare_eq` answers `Ok(true)` too. No pair the first early return catches escapes the
second, so deleting `if elem == each { return Matched }` changes no verdict and reddens
nothing, in either table. `stops_on_partialeq` is renamed `stops_on_a_matched_entry` in
both tests, because it never showed which check matched, and both comments now say so.
`Bool` was the obvious candidate for the missing asymmetry and does not work either;
`compare_eq(Bool, Bool)` is `Ok(true)`.
Also carries the mutation-matrix caveat that the cherry-pick of the element-loop fix
predated: the `(None, None)` arm's non-separability at the predicate is now measured rather
than argued, and cell 4's real dependency on `membership_pairing_refused` is disclosed --
it isolates the element loop GIVEN that the subset matches, not unconditionally.
Verification: the three mutations above run against both short-circuit tests, each test
invoked alone so its diff block is unambiguous, with --test-threads=1 throughout. Parallel
output splices `test ... FAILED` lines, which under-counts failures and reads as green --
that is how an earlier matrix run reported a red cell as passing.
`element_pairings_built` answers per ARM of `InOperation::compare`, and for a LIST `rhs_value` the arm stops deciding: `contained_in`'s `(List, List)` arm reaches `elements_not_matched`, which asks `is_one_of` per left-hand element, and every arm can land there. So for a list right-hand value the honest answer is "always yes" and three of the four rows the doc states are wrong. The contract was written unconditionally and is false there. MEASURED, not reasoned, with counters at the only two sites that pair a list's elements against a right-hand value and ONE right-hand value per case so the count is attributable. An earlier run used two values and could not tell which one built the pairings; its `(None, None)` row read as agreeing for that reason alone. For a list `rhs_value`: (Some, Some) predicted false, observed 2 pairings via elements_not_matched DISAGREE (None, Some) predicted false, observed 2 DISAGREE (Some, None) predicted false, observed 2 DISAGREE (None, None) predicted true, observed 2 agree THREE disagreements rather than one, and `(None, None)` is not among them. It agrees, but not for the reason its row gives: its own element loop IS skipped by the `if !eachr.is_list()` guard, and `contained_in` builds the pairings instead. A reader who credits that row's own loop is wrong even where the answer is right, so the row now says which half is load-bearing. Latent, because the sole caller is `incomparable_membership`'s `(List, right)` arm and a `(List, List)` pair is consumed by the arm above it, so `right` is never a list there. That guarantee is real and invisible -- enforced by match-arm ordering and written down nowhere, in a `pub(super)` helper whose entire reason for existing is that a condition restated away from the dispatch can drift from it. One reordering turns three wrong answers live. So the precondition is stated in the doc AND asserted with `debug_assert!`, which is proven live rather than decorative: passing a list `rhs_value` panics under the debug profile and returns normally under release, both measured. No existing caller trips it -- both tables pass in debug. Also corrects the `(None, Some)` row's justification. It said `string_in`'s refusal is "the same partition" `pair_refused` makes. False: `string_in` answers `not_comparable` for `(Regex, String)`, `(String, Regex)` and `(Regex, Regex)` while `compare_eq` answers `Ok(true)` for all three, so `pair_refused` refuses none of them -- three disagreements out of four probed pairs, only `(Int, String)` agreeing. The claim is also unnecessary: the gate needs element pairings to EXIST there, not the two functions to classify them alike. It is unreachable, a `PathAwareValue::Regex` being built only by the rules parser and a rule literal on the left selecting a different arm, but an unreachable justification is still the wrong one to leave standing. Doc plus one `debug_assert!`; no behavioral change in release and no test added.
`an_element_pairing_the_arm_does_not_build_earns_no_refusal`'s "What these cells do
NOT pin" section said "Flipping each of the four arms in turn moves exactly one cell
here and one in the helper's table". That contradicted its own neighbour six lines
up, which records that forcing `(None, None)` to `false` reddens the helper's cell 1
and moves nothing in this table.
The tree held the correct fact, the reason for it, and a summary that generalized over
a set the reason had just excluded. Structural, not a measurement slip: every cell
carries a `Literal` on one side, so cells 1-2 reach `(None, Some)`, cells 3-4
`(Some, None)` and cell 5 `(Some, Some)`. `(None, None)` needs both operands queried
and no cell here is one. Verified by reading the five constructions against
`is_literal`'s `len() == 1 && Literal` rule, and independently by measurement --
`(None, None) => false` leaves this table all green, 0 of 5 cells, while the helper's
cell 1 moves `true -> false`. The reading it invited, that this table gives
`(None, None)` cell-level coverage, is exactly what the paragraph above exists to
prevent.
Also names the mutation, because "exactly one cell" is ambiguous for the two arms
carrying sub-conditions. `(None, Some)` branches on whether the right operand is a
`String` and `(Some, None)` on `any(is_list)`, so negating either flips both
sub-branches and moves two cells, while substituting `true` and then `false` moves one
each. Six substitutions across the four arms -- one apiece for the two arms already
constant, two apiece for the others -- is what was measured.
And discharges the inert-vs-untested ambiguity in the redundancy claim next door with
numbers rather than a green result: deleting `if elem == each { return Matched }`
leaves the lib target at 1430 passed / 0 failed, while deleting
`Ok(true) => return Matched` from the same loop produces 20 failures. The site is
reached; the first branch changes nothing when gone.
Records one limitation of that redundancy measurement. The walk asks `elem == each`
then `compare_eq(each, elem)`, operands reversed between the two, and the asymmetry
cell 2 relies on is one-directional. Both directions were checked for the two pairs
the cells turn on and one direction only for the remaining arms; a later 2116-ordered-
pair probe in the loop's own order found no violation and 15 pairs where `compare_eq`
matches while `PartialEq` does not. The conclusion holds, and the narrower basis is
written down because "every arm" invites a reader to assume both orders were walked.
Comment-only: every changed line is a doc comment, no test added or removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #717, which was merged at
567c0d7. These are the commits made on that branch afterwards and sonot part of it. Rebased onto current
main; the tree differs from the pre-rebase branch only in thelockfiles
ef17f36patched.Same class of defect as #717: a rule that answers compliance for a check that never ran, or a run that
exits non-zero and names no finding.
One CI job is expected to be red, and it is not this branch's doing.
aws-guard-rules-registry-integration-tests-*fails because a change here — failing a run when anexpectation names a rule that produces no verdict — is correct and the registry has 30 such expectations, all
in
rules/aws/aws_cloudformation, naming 11 rules their file never defines. Measured: exactly 1 of 49registry directories changes its exit code, and stdout is byte-identical over the whole corpus, 9,156 lines
either way. The fix is aws-cloudformation/aws-guard-rules-registry#288, which takes
cfn-guard test -d ruleson that repository from exit 1 to exit 0 against this branch's binary. Both registryjobs are also now pinned to a SHA instead of tracking
main, so a commit in another repository can no longerturn this one's CI red without a change here; read that pin as a lock on a known state rather than a green
one, and move it forward once #288 lands.
A typo could delete a rule and report compliance
extract_messagesearched all remaining input for the closing>>, so one forgotten tag consumed everyrule up to the next rule's tag as message text and those rules ceased to exist. A file whose second rule
the template violated reported PASS at exit 0 — no finding, no diagnostic on any channel, and a parse
tree holding only the first rule with the rest of the file inside its custom message.
A message is now delimited by the grammar the language actually uses, rather than by a bound on the search.
A
>>on the opening line closes the message; otherwise the message ends at the first later line whosetrimmed text is exactly
>>; otherwise it is the error this code already raised for a missing tag.Two earlier attempts at this each bounded the search, and each was wrong. They are worth stating because
the shape of the mistake is the same both times. Stopping at the first line whose first token is
}orrulefinds a forgotten tag whose next
>>is in a later rule, but when the next>>belongs to the next clause ofthe same block no such line sits between the tags, so that clause was still swallowed at exit 0 -- in a rule
body, in a type block and in a
whenblock. It also rejected a legitimate body quoting example JSON, becausesuch a body has a line starting
}. Stopping at the next<<fixed all of those and broke a third case: withno second
<<in the file the search ran on to a>>inside a comment and closed the message there, soparsed as rules one and three, with rule two absent from the parse tree, and exited 0 against a template rule
two violates. Taking the minimum of the two bounds does not work either: a comment carrying
>>at the end ofthe same block is silent under both, since neither a
}line nor a second<<sits between the tags.Each of those bounds inferred document structure from a token that may legitimately appear inside a message
body. A terminator line infers nothing -- it is what a closing tag looks like in 231 of the 233 messages that
exist. The census is the whole basis for the design: of the 233 messages in the AWS rule registry and this
repository's fixtures, 231 are block form with the closing
>>alone on its line, 2 are inline with both tagson one line, nothing follows
>>on its line in any of them, and there are no other shapes.Two behaviour changes fall out. The JSON-quoting body now parses, because a brace in a body is text, so this
removes a limitation rather than documenting one. And a block body whose closing
>>shares its line with bodytext, as in
<< Violation: XthenFix: Y >>, is now a parse error. That is a narrowing, and it cannot beavoided: the swallowed clause in the example above is itself a line ending in
>>, so anything that acceptsthe one accepts the other. No message in the corpus is written that way. It is asserted as a rejection so it
is not reopened by loosening the terminator.
All four broken shapes exit 5 now, for one reason rather than four. Measured against the previous bound over
2,494 fixture pairs and 276 rules files: zero exit-code, stdout and stderr differences, and byte-identical
parse trees, so all 233 messages parse exactly as before. Regression tests cover the four broken shapes, both
accepted forms, the empty message, and the rejectedthree shapes.
Five more readings that were not what the author wrote
[x]gaveAllIndicesand[ x ]gaveAllValues, becausethe named branch of
all_indicesdid not skip whitespace. Two spaces were then the difference between aworking rule and an unevaluatable one:
Tags.%k[x] == "prod"passed at exit 0 whileTags.%k[ x ]bailedat exit 19, since the
%varinterpolation arm accepts one query part and not the other.whenwas atagin front of acut. It matched the first four characters ofwhenCreated, thewhitespace requirement after it failed, and the cut made that unrecoverable — so any clause whose first
identifier began
whenorWHENwas a parse failure.rule whenever { ... }defined fine and awheneverreference did not: a rule you could declare and never call.let v = 1thenlet v = 999failed a rule that the reverse order passed, and two assignments of different kindsignored their order entirely, because literals, queries and function calls go into separate maps consulted
in a fixed order. Rejected now in both scopes.
count.guardin this repository was a live instance: itbound
restwice and asserted%res == 3after each, so the first assertion tested the second count —and would have failed had it been live, because there are five properties and three collection entries.
Both counts are now named and asserted.
keyswas reserved for every clause rather than for key filters. Acutafter the keyword stopped[ keys EXISTS ]from reaching the branch that parses it. The identical clause parsed one slot later,which is what showed nothing ambiguous forced the rejection.
inand the sevenIS_*operators were plain tags.Size IS_INTEGERparsed asIS_INTplus areference to a rule called
EGER— loud with no such rule, PASS at exit 0 with one. The same guardtrue,falseandnullalready had.Three smaller ones:
i64::MINwas the single integer the parser refused, a rule reference could not end itsblock (
rule b { a }was rejected while the same rule over three lines parsed), and the normative grammarcomment documented
NOT_INand a clause-levelKEYS, neither of which was ever accepted.The data loader, where every rule's answer starts
A data file of nothing but comments aborted the process. Exit 101,
panicked at guard/src/rules/libyaml/event.rs:63: not implemented, on four shapes: one comment line, the samewithout a trailing newline, several comments with blank lines between them, and a fully commented-out template.
An empty file and a whitespace-only file already exited 255 saying the file is empty, so the degenerate case was
handled and this one was not. The event arriving at the wildcard arm is
YAML_NO_EVENT, and it arrives becausethe loop had no exit for a stream that never started a document:
DocumentEndwas the only arm that returnedand
StreamEndshared the no-op arm, so the next pull ran past the end of the stream and libyaml answered itwith a success.
StreamEndnow ends the loop and reports the file as holding no document, worded as the messagean empty file already gets. The wildcard arm remains and no longer aborts; it returns an error carrying the
event type, so a future miscount stays diagnosable.
TrueandTRUEwere read as strings, so a gate never fired. Underwhen Properties.PublicAccess == true, against a bucket withEncrypted: false:true,yesandonexit 19and catch the violation, and
TrueandTRUEexit 0 leaving it unchecked.is_bool_trueandis_bool_falsematched lowercase only, while the
nullarm two lines below lowercased first, so within one function null wascase-insensitive and bool was not. YAML 1.1 makes all 22 spellings boolean; the 1.2 core schema admits six. The
loader already read
yes,onand the single letters, which only 1.1 makes boolean, so the vocabulary wasalready 1.1's with the capitalised spellings missing, and those four are boolean under both schemas so they were
missing whichever version was meant.
test_handle_bool_happy_pathhas enumerated all 22 as the intended setsince it was written — and could not fail, because its assertion sat inside an
if let MarkedValue::Bool(..)and14 of the 22 cases passed while asserting nothing.
handle_type_refusedstr::parse::<bool>, so an explicit!!bool yeswas stricter than a bareyes; both now agree. Case-insensitive matching was rejected becausetRuEis boolean under neither schema, and narrowing to the 1.2 set was rejected because it would takeyesand
onfrom documents that write them, which is a separate argument.A query for
Tagswas answered out ofTag. A rule askingProperties.Tags.Nameof a bucket withProperties.Tag.Namereported aComparisonErrorover/Properties/Tag/Namecarrying that value: the rulenamed one property and was answered about another. One-way, since
TagagainstTagsreportedMissingProperty. The cause is the class-case entry in the converter list the lookup falls back to when a key isabsent as written.
cruet's class case is Rails' class-name rule, PascalCase and singular, soto_class_case("Tags")isTag; every other entry respells a key rather than naming a different one, andto_pascal_casealready covers PascalCase. It is not a plural rule either —AnalysisbecomesAnalysiandMetadatabecomesMetadatum. Removing it costs no reach, measured by instrumenting the fallback: class casewon 57 lookups across the registry's 193 paired files and pascal, title and train produced the identical string
for all 57.
A duplicated key took the last value and said nothing.
Encrypted: falsefollowed byEncrypted: trueevaluates as
true, the reverse order asfalse, and{"Encrypted":false,"Encrypted":true}the same way, allwith nothing on stderr. A reviewer reading a template top-down sees the first value and the tool judges the
last. YAML 1.2.2 makes this a loading failure rather than a merge — a mapping's keys must be unique per §3.2.1.1
and §3.2.1.3, and §3.3 lists non-unique keys among the loading failure points — so rejecting the document would
be defensible, and it would also fail every template carrying one today, which is a maintainer's call. This
reports the fact and leaves the verdict: one line naming the key's path and both locations, the last value still
winning, no exit code moved. The duplicate is not visible in the loader, which keys its maps on name and
location so both entries survive; they collapse a layer up where the map is rebuilt on the name alone, which is
also the only point that holds the path the key was reached by. Detection is per-mapping by construction, so a
property name repeated across two resources — which nearly every real template does — cannot register. Counted
before building: one occurrence in 71 corpus files, in the fixture that exists for it, and none in 256 registry
files.
cfn-guard teststill cannot report duplicates in its embedded inputs, becauseserde_yamlcollapses arepeated key during deserialization before any of this runs.
An
unreachable!()in the FFI error table.get_codeinguard-ffiansweredError::InternalErrorwithunreachable!(), in the table every error crossing the boundary is fed through and the same one that gained acode for the no-document error above. It is not reachable through
cfn_guard_run_checksas the code stands —that path parses through serde and never touches the libyaml loader, the one construction site that is on the
path is mapped to
ParseErrorfirst, and all four non-string key shapes come back asParseErrorwithoutpanicking. It is fixed regardless:
Errorandrun_checksare public, the remaining variants are constructedin a module this path avoids only because it writes through a
BufWriterof its own, and a table total overevery variant should not rest on an unreachability the next commit can invalidate. The arm gets a code and the
table stays exhaustive rather than gaining a catch-all, since exhaustiveness is what surfaced it.
A leading space before a numeric index or a quoted key was a parse error, while every other bracket form
tolerated one.
Names[0]andNames[0 ]parsed;Names[ 0],Names[ 0 ],Names[ -1 ]andResources[ "MyBucket" ]did not. This is the[x]-versus-[ x ]defect a second and third time: that fixadded whitespace skipping to the named branch of
all_indicesand left the numeric index and the quoted keyalone. The tests compare the resulting query part rather than merely that the file parses, because falling
through to a different part is how the first version of this defect did its damage.
Filters and captures in the query engine
[*]and.*both expand a map and hand each entry's value onward. That is right for a collection andwrong for a single object whose fields the filter names: for
{ Statement: { Effect: Allow, Action: "s3:*" } }the values reaching the filter were the stringsAllowand
s3:*,Effect == 'Allow'matched neither, and the filter selected nothing. An assertion over an emptyselection is a SKIP, so the two spellings disagreed —
.*failed and[*]exited 0.A filter capture leaked across iterations of a block. An iteration that captured nothing under a name fell
through to the parent scope, which by then held the earlier iterations' merged keys, so a non-compliant
resource passed on a compliant neighbour's key. Adding a compliant resource is what made the non-compliant
one pass, which is why such a rule looks correct when tested one resource at a time.
BlockScopenow carriesthe capture names its own clauses declare, read out of the rule text rather than learned at runtime — the
first attempt learned them as keys were captured, which made the verdict depend on document order.
A bare bracket is a capture too, and was not counted as one.
Properties.Tags[ tk ]written without a pipegoes through
all_indicesand becomesAllIndices(Some("tk")), which captures the entry's key at retrievalexactly as a filter does, but the name collector read
FilterandMapKeyFilteronly. No block declared thename, so the lookup deferred to the parent, which held the keys earlier iterations had merged up. A resource
that captured nothing was credited with a previous resource's key and document order decided the verdict: with
the capturing clause in one
whenblock and the reading clause in a siblingwhenblock that only a resourcewithout
Tagsenters, one order exits 0 and the other 255. All orders now agree at 19. Pre-existing;origin/maingives the same three codes, because no scope there held the name at all. The function's own doccomment claimed
AllValuesandAllIndiceswere collected through arms of their own, and there were none.Resources[*][ nm | ... ]then%nmended the run at 255 whileResources[ nm | ... ]— the same filterwithout the wildcard — worked. After a wildcard the map was already expanded when the filter ran, so the
name was discarded and the capture never made. The filter now runs at the expansion site, where the key is
still in hand.
Reporter aborts and findings with nowhere to go
Four reachable
unreachable!()in the CloudFormation reporter and a reachabletodo!()in the Terraform onetook the process down at exit 101, the latter on an everyday rule (
INagainst a plan). One was reachableonly after fixing the panic in front of it, which is why "not reproduced" is not the same as unreachable.
One rule that could not be evaluated discarded the whole file's report, in all four output paths and in
cfn-guard test: a file whose second rule read a variable that does not exist printed one error line andthrew away five real findings from its third rule.
Findings the per-resource output cannot place are reported under their own heading,
Findings that belong to no resource:, each entry naming the rule it came from. The existingCould not be evaluated:is for a blockor rule that failed on a condition nobody could decide; these clauses were decided and merely have nowhere to
be printed, and filing a missing property under "could not be evaluated" would report an ordinary failure as
an undecidable one.
Which findings those are is decided by what the reporter rendered, not by predicting it from paths — and
the first version of this section predicted, wrongly, in both directions. It asked once per rule on the
premise that one placed sibling renders the whole rule;
pprint_clausesin fact gates every clauseindividually, so a rule with one located finding and one pathless one showed the first and lost the second
from the console entirely while the JSON carried its reason. And it could not see a block whose query failed
at the document root, because it required
unresolvedto be absent andMissingBlockValueis the one offour block-report constructors that sets it — so a rule querying a top-level property against a
CloudFormation template exited 19 saying nothing, taking the author's own
<< >>message with it.rendered_contextsis the union of the resource clause sets, which is by construction what the per-resourceoutput will show, and every clause and block is asked against it. Deciding per clause without that set
reintroduces a different fault, because the evaluator emits two reports for one comparison it resolved one
way and could not resolve another — the context set is what separates the unresolved twin from a genuinely
unreported sibling. Alongside: the author's
<< >>message now reaches the section (.or_elseover the twomessage slots was dead in its second half),
NoValueForEmptyCheckno longer prints the negation of whathappened, and a rule name matched by prefix no longer renders one rule's block under another's findings.
Two further gaps in that section, both found by reviewing it rather than by a test. It keyed "was this shown
under a resource?" on the trimmed context string, so two clauses of one rule whose text is identical were
indistinguishable and the placed one made the pathless one count as shown; a rule asserting
Properties.Tags existsboth inside a resource block and at the document root reported one of the two and printed no section atall. What separates that from the legitimate twin the context set exists to suppress took establishing: both are
two
ClauseReportnodes, adjacent siblings, with byte-identical context and no recorded source span, and whatdiffers is the check carried,
UnResolvedbesideInResolvedfor the twin againstUnResolvedbesideUnResolvedfor the collision. The gate is now node identity plus that one case named for what it is, and itsuppresses a strict subset of what the old one did, so a finding the old code printed cannot be lost.
And the 320-character cap on an explanation was applied before trimming, so a message's own indentation spent
its budget. A message is written inside the clause carrying it, so every line arrives indented to that clause's
depth, which made the length an author could write depend on how deeply the rule was nested; 9 of the corpus's
233 messages exceed the cap as written and only 5 do once trimmed. It now measures the content, and an
explanation that fits once trimmed comes out whole and unmarked, because an ellipsis is a claim that content was
dropped. One consequence is visible in four registry outputs: a short message no longer prints the whitespace
around it, so the evaluator's sentence joins it a line earlier rather than after a blank one. No exit code moves
and nothing is lost, but the separation between an author's message and the machine's reason was incidental and
is now absent.
Both console reporters also aggregated findings into a
std::collections::HashMapand iterated it to writethe report. That hasher is seeded per process, so ten runs of one binary against a three-resource template
produced five distinct outputs.
Parser boundaries
Items[4294967296]wrapped toItems[0]and compared the wrong element. Non-finite float literals (1e400)and literals that underflow to zero were accepted. Duplicate rule names and repeated parameter names were
accepted silently. A map key that reads as an integer was treated as an index rather than a key.
cfn-guard testreported success for suites it never ranThree defects in the same family as the rest of this branch, found by following up why an expectation in the
AWS rule registry had never been checked.
A test suite could be run against the wrong rules file, and its failure reported as success. Test files
under a
tests/directory are paired with rules files by prefix, taking the first match in filename-sortorder, so a shorter stem claims a longer stem's tests. With
s3.guardands3_encryption.guardin onedirectory,
tests/s3_encryption_tests.ymlwent tos3.guard, which does not define the rule its expectationsname, and
s3_encryption.guardwas reported as having no tests associated. A suite that expects PASS where thetemplate violates the rule exits 7 under
test -r/-tand exited 0 undertest -dover the same twofiles. Paired on the longest matching prefix now. The looseness is deliberate and is kept: requiring a
_testssuffix would stop
<prefix>.ymland<prefix>-1.jsonpairing, and a character-class boundary cannot separatethe readings because
_is both the conventional separator and a word character.min_by_keyover thereversed length rather than
max_by_key, so a tie still goes to the first file in sort order as it did before;ties are reachable, since
x.guardandx.rulesetshare a prefix.A test file matching no rules file was discarded without a word. That is what a rename leaves behind.
s3_bucket.guardbesidetests/s3_tests.yml, left over from when the rules file wass3.guard:test -dprinted only that the rules file had no tests associated, which reads as benign, and exited 0. Renaming the
test file and changing nothing else gives exit 7. Each unmatched file is now named on stderr with its path.
An unchecked expectation was invisible to every structured reporter. An expectation naming a rule the file
does not define is not checked; the plaintext reporter says so on stderr and
json,yamlandjunitsaidnothing. A test file whose every expectation names a nonexistent rule produced
<testsuites tests="0" failures="0" errors="0">and exited 0, so a pipeline gating on that report had nothingto gate on. Reported now as a skip in a field of its own,
unchecked_expectations, not folded intoskipped_rules-- those are opposite directions of one mismatch, rules with no expectation againstexpectations with no rule, and a consumer reading one array could not tell which had happened. A skip and not a
failure, because escalating it changes the exit code of a shipped CLI for every user with a typo in a test
file, and that is a maintainer's call; making the fact visible is not. The field is omitted when empty, so
every report over a suite with nothing unchecked is byte for byte what it was.
Run against the AWS rule registry at
7f7340c, unmodified, the two diagnostics name three test files thathave never run and eleven expectation names that answer to no rule. Two of those three files are the only test
file their rules file has, so
s3_bucket_default_lock_enabledandcloudfront_accesslogs_enabledhave no testcoverage at all: one filename drops a trailing
d, the other begins with a space. Both suites pass oncewired up. Those are fixed in that repository rather than here.
Merging alongside #719
Both PRs merge into
origin/maincleanly on their own. Only the combination conflicts, and no singleresolution direction is correct for it — of the five conflicted paths, three want this branch's side, one
wants #719's, and two want a union. One of them is dangerous to get wrong and the damage is silent.
Measured by trial merge into
origin/mainat3e265bb, withrereredisabled so a recorded resolutioncould not report a false clean, and judged by the merge output text rather than by the unmerged-path list.
guard/src/rules/parser.rsextract_messagegrammarfind_substring(">>")guard/src/rules/path_value.rsQueryResolverand its twoselectfunctionsguard/src/rules/mod.rsadd_merged_capture_keytrait methodguard/src/commands/reporters/validate/common.rsFileReportandMessagesguard/src/commands/reporters/test/structured.rshas_unchecked_expectationsandnumber_of_unchecked_expectationsbuild_junit_test_caseswith an elided lifetimeparser.rsis the one that matters. #719 branched before any of the three attempts atextract_messageand its
Apply rustfmtcommit reformatted the signature, which is what makes it conflict rather than merge.Its side is the original unbounded search. Resolving that hunk in #719's favour reinstates the defect where one
forgotten
>>consumes every following rule as message text and the run reports PASS at exit 0 — the headlinefinding of this PR, and silent when it recurs.
path_value.rsis the mirror image: this branch carries that engine only because it does not delete it, anddeleting it is #719's entire purpose. It does not touch the duplicate-key detection added here, which sits
earlier in the file, well outside the conflict region.
The order matters and only one order is dangerous. If #719 merges first, this branch rebases onto it and
the resolution happens here, so no trap reaches anyone. If this branch merges first, the conflict lands in
#719 and the
parser.rshunk is the one that must not be resolved the obvious way.This table is re-derived rather than carried forward: it named two conflicted paths when this branch was 27
commits shorter, and both the count and the resolution directions have changed since. Re-derive it again if
either branch moves.
A rule could not be composed out of other rules
rule MAIN { H_A H_B }failed whenever a rule it referenced was not applicable. Two helper rules eachguarded by a
whenon their own resource type, and a template carrying one of those types but not the other,gave
H_APASS,H_BSKIP andMAINFAIL at exit 19 — reporting a violation on a template that violatesnothing. A template carrying neither type failed as well.
eval_guard_named_clauseclaimed every assertion SKIP as FAIL, in both polarities, which also made the armbelow it unreachable for assertions.
eval_parameterized_rule_callcarried the identical line forr(...),with a comment claiming to mirror the first; both are fixed together and one test asserts both, so a change to
one cannot pass on the other's coverage.
This is why decomposing a ruleset over disjoint resource types has not been done: neither workaround holds. A
conjunction of references fails whenever any type is absent. A disjunction returns PASS when any one helper
passes, so a template with a violating IAM Role beside a clean DynamoDB table passes a rule that should fail
it — a false negative in a compliance rule. The AWS rule registry's existing helper idiom composes with
ORonly because its helpers are alternative paths to one conclusion, and that does not transfer.
A negated reference is included:
rule deny when Resources.*.Type exists { not inner }failed on a templatewith one S3 bucket and no KMS key. It now reports
the rule did not apply because it referenced rule [inner], which did not apply to this input. A negated reference used as awhengate is deliberately unchanged,because a gate that closes silently disables the rule it guards; the registry's one negated reference is a
gate, and that decision is what protects it.
Exit codes said the tool broke when the rules file was wrong
Two paths reported an authoring mistake as exit 255, the internal-failure code, where 5 means rules-file
error:
parse-treeon a rules file it cannot parse —validatesaid 5 for the same file, so two subcommandsdisagreed about whose fault it was.
validateon a rules file referencing a variable noletdeclares. Forgetting aletis a common mistake,and the caller was told the tool had failed. The undeclared name is now reported with
Could not resolve variable by name <name> across scopes, so it can be acted on.The reason a 255 survived here is a test that passed for the wrong reason.
guard/tests/parse_tree.rshad twocases sharing one expected output that is an I/O error. One points at
dne.guard, absent on purpose. Theother pointed at
validate/rules-dir/malformed-rule.guardwhile the file lives atvalidate/malformed-rule.guard— one directory up. So it failed atFile::open, matched the shared I/Ostring, and passed green.
parse-tree's parse-error exit code therefore had no coverage at all. The fixture,never having been run, also turned out not to contain a syntax error: it is an undeclared-variable file, which
is how the second defect above came to light. The repository had no genuinely malformed rules fixture; there
is one now, the cases assert what their names claim, and the exit codes are documented, which they were not.
A test expectation could name a rule that does not exist
cfn-guard testprintedNo rule named X is in this file, so its expectation was not checkedand exited0. An expectation naming a rule that is not there asserts nothing, so a rule renamed without its test file
being updated leaves a test that silently stops testing while the suite stays green.
It is now exit 1 —
TEST_ERROR_STATUS_CODE, notTEST_FAILURE_STATUS_CODE, following the line thisrepository already draws at
guard/tests/test_command.rs: "an expectation that could not be evaluated is adifferent answer from an expectation that was not met." All four output formats report it; junit uses its
<error>element, which consumers treat as red exactly as they treatfailures.The question that decided whether a blanket failure is right is whether an expectation for a rule in a
sibling rules file can legitimately be checked in directory mode. It cannot:
OrderedTestDirectory::fromreduces the candidate rules files with
min_by_key, which yields exactly one claimant, so a test file isnever evaluated against a second rules file. A directory whose test files are named to cover several rules
files at once will now go red, and it was never checking what it appeared to.
This has a consequence outside this repository, measured rather than assumed.
aws-cloudformation/aws-guard-rules-registryhas 30 such expectations inrules/aws/aws_cloudformation/tests/cfn_no_explicit_resource_names_tests.yml, across 11 distinct names, in asuite that reports clean. Its
.github/workflows/{ci,build,publish}.ymleach installcfn-guardunpinnedfrom
mainand runcfn-guard test -d ./rules/, so when this change is released that repository's CI andits publish pipeline go red until those expectations are dealt with.
aws-guard-rules-registry#286
removes them and is green under every
cfn-guardversion, released or not, so landing it first leaves no redwindow in either repository.
A regular expression's own diagnostic was built and thrown away
Both failure returns in
parse_regex_innerwerenom::Err::Error, whichalttreats as recoverable, so thefancy_regexmessage was constructed and discarded and the next value production's complaint was reportedinstead.
== /a(/exited 5 sayingexpecting either a property access "engine.core" or value like "string",with no mention of the parenthesis. It now says
Could not parse regular expression: Parsing error at position 2: Opening parenthesis without closing parenthesis. The file's own convention two functions above alreadyrequired
Failurehere and cross-referencesparse_range; two sites followed it and these did not. Twoexisting tests pinned the
Errorvariant byassert_eq!, and moving those assertions is the point of thechange rather than collateral damage.
Two changes that alter what a rules file means
Both are deliberate, both are measured, and neither is a bug fix you can take without noticing it.
A string or regex literal ending in a backslash.
parse_string_innerandparse_regex_innereach decidedwhether a delimiter was escaped by testing whether the text in front of it ended with
\, which cannot tell\from\\. When it guessed wrong the scanner resumed past the closing delimiter, so the next quote orslash anywhere in the file became the terminator — an apostrophe in a comment was enough — and every clause
and rule in between was absorbed into a string value, unevaluated and unreported, at exit 0.
Fixing it means a backslash consumes the character after it, whichever character that is. That is the rule
every language with a backslash escape uses, and it makes a backslash's meaning independent of its position.
The consequences: in a string
"a\\b"is now one backslash rather than two, and in a regex\\closesitself so a following
/ends the literal. There is no reading in which\\/both terminates in/^x\\/and does not terminate in
[A-Za-z0-9\\/+=]— the swallow and the fixture are one construct, so one of themhas to move.
Measured blast radius:
\\inside a string literal, a backslash immediately before a quote, and\\/inside a regex each occur zero times across all 313
.guardfiles and 435 KB in this repository and theAWS rule registry. The two exceptions are the two copies of
advanced_regex_negative_lookbehind_rule.guardhere, whose
\\/is rewritten to\/— the spelling that always meant what the file intended. A rulesfile in the wild using
\\/in a regex now fails loudly at parse rather than misreading silently.Mixed range bounds,
r[0,20.5], are now accepted — and that commit is separable. It is not a defect fixand can be dropped without touching anything else.
docs/CLAUSES.mdpromises thatintegerandfloat"compare against each other as numbers. That includes range membership", but its examples are
r[5.0, 100.0]and
r[5, 100]— homogeneous bounds with a mixed value. Mixed bounds are not promised, andRangeTypeholds one type, so refusing them was a representable-shape limitation rather than a defect. The case for
accepting them is that
Size in r[1,2]holds for aSizeof1.5whileSize in r[0,20.5]did not parse atall, so the gate was narrower than the thing it gated; the integer bound widens to float and is refused if
widening would move it. Accepting more input invalidates no existing file. But it changes the accepted grammar
of a public DSL, which is a maintainer's call, so it is its own commit.
Verification
cargo test --release --no-fail-fast: 1489 pass, 0 fail, 2 ignored across 16 targets, from 1379 at thebranch point.
cargo clippy --release --all-targets -- -D warningsandcargo fmt --all -- --checkbothclean.
The whole blast radius across both repositories, and every item in it is intended:
.guardfile parsed with the branch point's binary and this one, compared on exitcode and on the sha of the full tree. 210/210 in the AWS rule registry identical. 105 of 108 here identical;
the three that differ are the deliberately-unparsable fixtures going 255 to 5, with byte-identical tree
output — the exit-code fix, not a parse change.
tests/subdirectory, run ascfn-guard test --dirat clean upstream, before and after. One changed:rules/aws/aws_cloudformation,0 to 1, stdout unchanged. That is the ghost-expectation fix firing on the 30 dead expectations described
above, and nothing else in the registry moves.
git archivesnapshots at explicit refs rather than alive checkout, after two measurements in this work turned out to be reading a tree that had moved
underneath them.
the language semantics rather than from the binary.
cfn-guard testpairs plus1,433
validatepairs — byte-identical, with sensitivity proved rather than assumed by showing a fixture inthe same shape does move across the two binaries.
A caveat on all of it, worth more than the numbers. These are regression instruments. Every one of them
reported zero changes for most of the parser fixes here — correctly, because no fixture contains a forgotten
>>, or a string ending in a backslash, or a transposed range. A differential answers "did any of theseinputs change answer", not "is there an input that gets a wrong answer". Every defect in this branch was
found by reading what a function commits to and then constructing the input that violates it, and each fix
carries a test built the same way.
Two flavours of empty selection answered differently
{}— noResourceskey at all{"Resources":{}}Rows 1 and 2 are the same logical situation as row 3 — an empty selection with
not existsover it — andanswered PASS where row 3 answered SKIP. A rule about DynamoDB tables reported compliance for a document
containing nothing, and correctly reported itself inapplicable for a document containing one unrelated resource.
Across sixteen operator spellings twelve disagreed, and the disagreement was not uniformly permissive: for
exists,!empty,==andinthe empty-document flavour answered FAIL where the other answered SKIP. Thehighest-stakes row was a
whencondition, where anot existsgate opened on one flavour and closed on theother — changing whether the guarded body ran at all rather than only what was reported.
Two fixes were implemented and rejected before this one, both caught by measurement. Treating any
all-unresolved result as an empty selection turns
Resources.*.Properties.BucketEncryption existsfrom FAIL intoSKIP — "this bucket has no encryption configured" is the entire point of a compliance rule, so that reading gives
up the tool's main job. Keying off "an expansion with query parts after it" produced every target row correctly
and then failed four existing tests that encode requirements rather than the defect: an IAM role carrying no
Tagsmust failProperties.Tags[*].Value == /.../, andsome Tags[*].Key == /PROD/overTags: []must fail.Those four tests are unchanged.
The predicate that satisfies all of it: an empty expansion yields an empty selection only when a filter is
still pending at or after the point the collection came up empty. A filter is how the language says "the subset
of things this rule is about", so a filter that has not yet run over an empty input means the rule has no
subjects. A bare wildcard is a projection over a property of a subject already chosen, so an empty projection is
an answer about that subject. Searching from the current index rather than from zero is what keeps
Resources.*[ Type == 'X' ].Properties.Tags[*].Key == /PROD/failing on an empty tag list — that filter hasalready run, the resource was selected, and its empty
Tagsbelongs to it.This routes the empty-collection case into the SKIP path that already exists for filters, which
eval.rs's own comment describes as happening "when the query has filters in them".QueryPart::Indexisexcluded deliberately:
Items[0]names one element rather than enumerating a set.Blast radius: all 44 directories of the AWS rule registry run as
cfn-guard test --dir, and every one of its210 rules files run against three empty-ish documents — 630 verdict pairs, zero changes. Every registry rule
is gated by
when %var !empty, so none takes the path this changes; the defect was reachable by hand-writtenungated rules.
Known limitations
Reviewing this branch found more than reading it did
Later passes over this branch's own commits turned up four things that no amount of reading the diff would
have reached, because each only exists when a documented command is actually run. They are worth listing
separately because three of them are in the tooling around the change rather than in the change.
cargo test --allaborted on this branch. Fifteen of the 34 tests that parse at or nearMAX_NESTING_DEPTHexceeded libtest's 2 MB thread stack in the default debug profile — SIGABRT, cargo exit101 — while
--releasepassed.pr.ymlrunscargo test --verbose, which is debug, on ubuntu, macos andwindows, so this was red on three platforms. The deep parses now run on an explicitly sized thread, so they
no longer depend on the harness's default. What hid it is worth naming:
MAX_NESTING_DEPTH's doc commentjustified the bound with "about 2.3x to spare", measured on an optimized build and written without saying so.
Unoptimized, the same ladder tops out below the bound. Every stack figure in that comment now states its
profile, and the argument for 128 rests on the corpus instead, because a corpus fact survives a change of
profile.
cfn-guard testexited 0 on a rules file it could not parse, injson,yamlandjunit, whileprinting
errors="1"in the same run. A CI job gating on the exit code saw success on a broken ruleset,which is the worst failure direction available for a policy tool.
single-line-summaryalready exited 1 onthe same bytes, so the two format families disagreed. All four now route through the same
TestResult::get_exit_code()rather than assigning codes by hand at each site.The four output formats disagreed about content and about codes in three more places, found by
enumerating the grid rather than by fixing the reported case: a stale-rule-name diagnostic was dropped in
single-line-summarywhen the same case also carried an unparseable expectation string, and thedirectory path answered 7 where three formats and the single-file path answered 1 for identical input. There
are now tests asserting that every format agrees over one rules file and over a directory — the invariant,
not an instance. Both of the earlier fixes here had been to an instance, and one of them created the same
defect pointing the other way.
The Windows registry CI step decided pass/fail from stdout.
if (<command>)in PowerShell tests thecommand's output for truthiness, not its exit code, and that command prints thousands of lines — so the step
logged "have passed" for runs that had exited non-zero, and would have failed a successful run that printed
nothing. Its bash twin uses
if <command>; then, which does test exit status, so the two legs of one checkwere checking different things. Both now test
$LASTEXITCODE. The same expression appeared in theparse-treeloop, where it was correct only by luck.Four doc comments also stated numbers that could not be reproduced, which matters here because a false
comment in this area misdirected several of these investigations. The corpus depth maximum was 6 and is 7,
reached by two files rather than four; the loader's two corpus figures were each high by one, and one of them
named "the deepest CloudFormation template" in a corpus that contains no template files. A ratio was still
derived from a threshold that had already been corrected. And a call-graph component count was removed rather
than corrected: the figure depends on how a parser passed to a higher-order combinator is attributed, four
defensible models give four different answers, and the sentence named none — so no reader could check it. Two
user-facing refusal messages that quoted corpus statistics now make their point qualitatively, because a
message asserting a maximum over a repository whose release cadence this one does not control is wrong again
on someone else's schedule.
Three questions are left open on purpose, each measured and recorded rather than settled by accident: a key
filter after
[*]versus.*; what a filter applied to an already-indexed value (Rules[0][ ... ]) shouldmean; and
::MODULEstripping in a type-block name. Each needs a decision about what the language means, andguessing one is worse than leaving it visible.