Fix evaluator and reporter defects where a rule reported compliance, or nothing at all, for input it should have rejected

[awsmadi]

Twenty-six commits in the rule evaluator. Every one is a case where a rule reported compliance, or reported nothing at all, for input it should have rejected — plus the reporter fixes needed to make those failures visible.

The class matters more than the count. SKIP and PASS both exit 0, so a clause that quietly stops failing is indistinguishable from one that passed, and a rule that never fires looks exactly like a rule that holds. Most of what follows is that shape, reached by different routes: a dropped negation, a reference that resolved to nothing, a role that did not propagate, a comparison between an integer and a float, a condition scoped to the wrong object, an index applied twice.

The first twelve commits came from auditing the PASS/FAIL/SKIP paths by reading, and two of them were found by @satyakigh in review. The last twelve came from measuring instead: drive branch coverage over the evaluator, then treat every line that decides a verdict and has never executed as a bug candidate. That turned up four more defects, all pre-existing, and confirmed one block of code as unreachable rather than untested. The two halves were briefly separate PRs; they are one story and the split made the diff harder to read, not easier.

Commits

Commit	Defect
2b215cc	usize underflow drops the source snippet from violation reports
a54e4ca	clause-level not on a binary comparison is parsed, displayed, then discarded
ed342ea	not <rule> reports compliance when the referenced rule never ran
3f8466e	comparisons against a reference that resolves to nothing are skipped rather than decided
8c51b77	replaces two strictness booleans with a ClauseRole type; fixes a gate that disarmed the block it guarded
0e140b3	clause-level not on a parameterized rule call is discarded
70b9006	a when nested inside a when re-labels its body, so a gate's FAIL becomes an absorbed SKIP
de431f7	five of the six message-bearing record variants never reach a reporter
70feb75	integer and float operands are not comparable, which silently disarms any gate comparing them
7ea5c43	288-cell comparison matrix, the type block's status fold, and the root-scoped type-block when
a023d09	176-cell unary operator matrix across every value shape
eeddc92	a type block's when conditions are evaluated against the file root, not each resource
ca818b0	a skipped rule reaches the reporters as a bare name, so it cannot explain itself
b104090	four evaluator arms no rules file can reach, in the function you would edit to fix a comparison
cc5a2f5	asserts the status decisions no test in the suite reached
d56965c	documents the type block and the numeric comparison rule
f3c919f	a condition that cannot be decided skips the rule with no diagnostic at all
46156f0	a record context built from std::any::type_name, which varies by compiler version
9900d0b	an index after an interpolated map key is applied twice, so the query resolves to nothing
dcf52ad	a failure message prints an entire allow-list, because the cut-off used max where min was meant

Each commit stands alone and can be reviewed or dropped independently. The table omits a few commits that only adjust tests or apply rustfmt.

2b215cc — underflow in the violation reporter

emit_code computed its first context line as max(1, line - 2), which evaluates the subtraction before the clamp. On an unsigned line number of 0 or 1 that underflows: debug builds panic and exit 101; release builds wrap to ~usize::MAX, seek past EOF, and silently omit the snippet while the exit code stays a correct 19.

Only the violation path is affected, so the tool misbehaves precisely when it has something to report. The trigger is input formatting rather than content — a minified single-line template puts the violated property on line 0 or 1.

Extracted into context_start_line() using saturating_sub, with unit tests for lines 0-2, the normal case, and a never-returns-zero property. Reachable from the library entry point, so it also affects guard-ffi, where unwinding out of an extern "C" function is undefined behaviour.

a54e4ca — clause-level not dropped on binary comparisons

parser.rs accepts a leading not before a clause and stores it as GuardAccessClause::negation, but the binary evaluation path never passed it to binary_operation; only the unary path consumed it. grep -n negation over eval.rs returned three uses, none in the binary path.

The result is that not <query> == <value> evaluated as plain <query> == <value>, the exact inverse of the author's intent. A rule written to reject an insecure value accepted it and rejected the secure one. The report compounded this by rendering the clause with the not, so nothing in the output indicated it had been dropped.

Composed with the operator's own not-flag by XOR at the call site. This matches invert_closure in the superseded evaluate.rs, which applies both flips independently — that reference implementation is the reason I read this as a regression in the v3 evaluator rather than a design decision. No existing test covered clause negation on a binary access clause.

ed342ea — not <rule> on a rule that never ran

eval_guard_named_clause collapsed SKIP into the same match arm as FAIL, so under negation it produced PASS. not <rule> therefore reported compliance on the strength of a dependent rule that never ran, and because the enclosing rule then reported PASS rather than SKIP, the output contained no trace of the omission.

The two contexts a named-rule reference is reached from need different answers, so this threads a role: in a rule body the reference is an assertion and a skipped dependency fails closed; in a when condition, gating on a rule that did not apply is deliberate and is asserted by the existing cross_rule_clause_when_checks. A first attempt failed SKIP unconditionally and broke that test, which is what surfaced the distinction.

3f8466e — comparisons against an empty reference

CmpOperator::compare treated an empty operand on either side as reason to skip the clause. A rule whose reference list resolved to nothing therefore reported compliance for input it should have rejected.

The two empty operands are different situations:

• Empty LHS — the query selected nothing, so the clause has nothing to say about this input. Genuinely inapplicable, and what lets one ruleset run across templates that do not all contain the resource type being checked. Unchanged, and checked first so it keeps precedence when both sides are empty.
• Empty RHS — there are values to check but no reference to check them against. Polarity decides: a positive comparison is unsatisfiable and fails; a negated one is vacuously satisfied and must not fail, since a deny-list is legitimately empty whenever the input contains none of the denied values.

The vacuous case reports SKIP rather than PASS. eval_conjunction_clauses treats PASS as short-circuiting (continue 'conjunction) but SKIP as absorbing, so a vacuous PASS satisfies an entire or block and abandons its sibling disjuncts unevaluated — X != %empty_denylist or X == true would pass a violating input because the first disjunct is vacuously true and the real check never runs. An intermediate version of this change did exactly that. Per-clause soundness is not sufficient; the status has to compose correctly under the folds.

The verdict is also context-dependent: in a when condition the unsatisfiable case stays SKIP, because a failing condition makes the gate not-PASS and eval_rule then treats the rule as inapplicable and skips the whole body.

The unsatisfiable case emits a status rather than per-value comparison records, because there is no right-hand value to report against and building from: out of a raw LHS entry panics when the LHS is a QueryResult::Literal — three reporters treat that as unreachable inside a comparison.

8c51b77 — ClauseRole instead of two booleans

The two preceding commits threaded strict_skip: bool and strict_empty_rhs: bool. That encoding was unsafe twice over: a boolean argument carries no meaning at the call site, and a missing one is invisible.

Converting to an enum turned every unthreaded path into a compile error, which surfaced a defect the boolean version hid. eval_when_clause threaded the flag into its Clause and NamedRule arms but not ParameterizedNamedRule, and everything downstream defaulted to assertion strictness. A parameterized rule invoked as a gate therefore evaluated its body strictly, failed, made the gate not-PASS, and eval_rule skipped the guarded body entirely — a violating input exited 0 where v3.2.0 exited 19. Four shapes reached it: a plain parameterized gate, one whose gate body held an empty-reference comparison, one nested in a rule body, and one on a type block.

The compiler identified all twelve threading sites, including three filter-predicate calls in eval_context.rs that no amount of reading had turned up: a filter selects values, so it is a test rather than an assertion.

0e140b3 — clause-level not dropped on parameterized calls

The same defect class as a54e4ca, in the one arm it did not reach. The parser stores a leading not on a parameterized invocation, but eval_parameterized_rule_call returned the invoked rule's status unchanged and never read the flag, so not r(...) behaved identically to r(...). Demonstrated by three fixtures: the negated and un-negated forms were outcome-identical at exit 0, while the non-parameterized not r correctly exited 19 — so the mechanism existed and only this arm bypassed it.

Applied after evaluating the invoked rule, mirroring eval_guard_named_clause so both spellings agree.

This commit also tightens three assertions added by earlier commits. They used assert_ne! and so admitted both SKIP and FAIL — exactly the distinction that decides whether a guarded body ran, since both exit 0. The exact statuses were measured rather than guessed.

70feb75 — a number is a number

compare_values in path_value.rs matched Int/Int, Float/Float, String/String, Char/Char and Null/Null, with no arm across the two numeric types. Every mixed comparison fell to the NotComparable catch-all.

The visible half is a wrong FAIL: Size > 10 against a template carrying Size: 50.5 reported "PathAwareValues are not comparable float, int" and failed a compliant volume.

The half that matters is a wrong PASS. Put the same comparison in a when condition and eval_rule maps the non-PASS to Status::SKIP, which exits 0 and takes the guarded body with it:

rule large_volumes_are_encrypted when Resources.*[ Type == 'AWS::EC2::Volume' ].Properties.Size > 10 {
    Resources.*[ Type == 'AWS::EC2::Volume' ].Properties.Encrypted == true
}

template	result
Size: 50, Encrypted: false	FAIL, exit 19
Size: 50.5, Encrypted: false	SKIP, exit 0, nothing reported

Changing one character in a template turns the encryption rule off. Reproduced on the merge-base 57bbdbf, so it predates this work; the reason nobody hit it in review is that the Ge/Gt/Lt/Le paths had no test that reached them with operands of differing types.

The comparison is exact rather than (i as f64).partial_cmp(f). i64 above 2^53 does not survive a round trip through f64, so the lossy spelling answers Equal for 2^53+1 against 2^53. Casting the other way is safe once the float is bounds-checked, and the bound is 2^63 rather than i64::MAX as f64 because the latter rounds up to 2^63 and would admit a float that then saturates on the cast. NaN stays NotComparable, which is what Float/Float already answered.

Mutation-verified twice: removing the arms restores NotComparable, and substituting the lossy spelling fails on the 2^53+1 case specifically. No existing test or golden file depended on the old behaviour.

7ea5c43 — the comparison matrix, and two type-block findings

the_comparison_matrix_over_operand_types_is_pinned evaluates 288 clauses end to end: six operators against eight left-hand operand types and six right-hand literals. What the grid asserts is the absence of a wrong PASS — a FAIL cell is a pairing that is undecidable or genuinely false, and PASS appears only where the comparison is decidable and true. Two cells look like mistakes and are not, so the doc comment names them: a list on the left distributes element-wise, so [1,2,3] < 50 passes, and an absent property fails rather than skips.

the_type_block_status_fold_is_pinned covers eval_type_block_clause's fold, which has the same shape as eval_conjunction_clauses and absorbs a per-resource SKIP the same way. No test reached any of its arms, including the one that decides whether a violating resource is reported at all.

a_skipped_type_block_is_indistinguishable_from_a_clean_run records a defect rather than fixing it. Inside one construct the scoping is not consistent: the block's clauses are resource-relative, but the when conditions are evaluated against the enclosing resolver, before the per-resource ValueScope exists. So

AWS::EC2::Volume when Properties.Size > 10 { Properties.Encrypted == true }

reads as "every volume over 10 GiB must be encrypted" and instead looks for Properties at the file root, finds nothing, and skips — reporting not_applicable and exit 0 for every template it is ever run against, including the ones it was written to catch. when Resources.A.Properties.Size > 10 works. Both spellings are asserted, since the contrast is the whole finding.

I wrote an explanation onto that skip record and then removed it. It could not reach a reader: skipped rules arrive at the reporters as a HashSet<String> of names, so the message would have been recorded and discarded — which is the defect #717 just removed from five variants. every_recorded_explanation_has_a_rendering_path caught it, which is what that test is for. Surfacing it properly means the skip set has to carry reasons, and that changes the report signature every reporter implements.

Both halves are now fixed, in eeddc92 and ca818b0.

a023d09 — the unary operators

176 cells: eight operators in both polarities against eleven operand shapes, including the empty and absent cases that separate EMPTY from EXISTS. unary_operation and the is_* family had no test walking them across shapes, which is how the negation arm arrived here uncovered.

Two behaviours are pinned on purpose. EMPTY on a container answers; on a scalar it is an incompatible-type error rather than a status, which is right because both statuses would be wrong — an int is not empty, but calling it non-empty implies the question made sense. And every negation must invert its positive form: a negated operator answering the same status as the positive one has stopped discriminating, which is the shape of the role-propagation defect #717 opens with.

eeddc92 — type block conditions are per resource

The conditions moved inside the loop over matched resources, so they are evaluated against the same scope as the clauses they guard. That also makes them per resource: one the condition exempts contributes to neither the pass nor the fail count, so an exempt resource cannot shield a violating one, and a block that applied to nothing answers SKIP rather than PASS.

The cost is the mirror image and it is real. A condition written as a literal root-anchored path — when Resources.A.Properties.Size > 10 — resolved before and does not now, because ValueScope::query starts at the resource. Accepted for two reasons: a condition over one named resource does not belong on a block that iterates all of them, and the idiom real rulesets use is unaffected, since ValueScope::resolve_variable delegates to the parent and let volumes = ... followed by when %volumes !empty still resolves at the root. The test asserts all three spellings so the trade is visible rather than inferred.

Nothing in the repository depended on the old behaviour: a search for type blocks carrying a when found two occurrences, both fixtures added by this branch.

Rejected alternative: having ValueScope fall back to the parent root when a query does not resolve locally would keep both spellings working, but every unresolved path would retry against a different scope, so a typo in a resource-relative path would silently resolve somewhere else — trading a visible skip for an invisible wrong answer.

ca818b0 — a skipped rule can explain itself

find_skip_reason walks a rule's own record subtree for a block-shaped record that skipped with a message, and the three paths that report skips were widened to carry it: FileReport gained not_applicable_reasons, the console summary prints the reason under the rule, and GenericReporter::report takes a SkippedRules map instead of a HashSet<String>.

Both new serialised fields are skip_serializing_if empty, so a run with nothing to explain produces byte-identical output and no consumer of the JSON or YAML document sees a shape change. BTreeMap rather than HashMap, and the console printer sorts, because the underlying map is unordered and would otherwise reshuffle its own lines between runs.

Two explanations use the mechanism, and they are the pair worth telling apart. A type block whose query matched nothing says the type is absent, which is the ordinary reason for a rule not to apply. One whose condition exempted every matched resource says so, and that is the one worth a second look — a rule that never fires looks exactly like a rule that passes, because both report SKIP and exit 0.

The counting in every_recorded_explanation_has_a_rendering_path changed too. It counted the literal string message: Some and undercounted the moment a message was built in a match arm, reading 14 where the real figure was 15. It now counts by exclusion: every message: field that is not None, not a type annotation, and not a forward of the author's own custom_message. That cannot be fooled by a new spelling, and it separates evaluator explanations from the custom-message feature, which always rendered.

b104090 — the unreachable arms, removed by narrowing

MapKeyFilterClause::comparator was a (CmpOperator, bool), which can express every operator, while map_keys_match parses four. real_binary_operation is reached only from QueryPart::MapKeyFilter and carried Ge, Gt, Lt and Le arms that recorded zero executions against the 288-clause matrix using those very operators.

Deleting them and keeping the wider type would have left them one parser change from returning and converted a currently-correct dead path into unreachable!(). A comment would have left the file permanently uncoverable and the trap intact — someone tracking an ordering bug finds the arm calling compare_ge, edits it, and sees no effect. A MapKeyComparator enum makes them impossible instead.

One user-visible consequence, an improvement rather than a regression: parse-tree serialises a map key filter's comparator as Eq rather than the two-element sequence [Eq, false], and != as NotEq. Regenerating the three affected golden files changed exactly one site — the other comparators in those documents belong to AccessClause, which keeps the pair. Hand-editing them first produced a wrong diff, because AccessClause also has a compare_with field and so cannot be told from a map key filter by its neighbours.

cc5a2f5 — the decisions nothing reached

A clause-level not in front of EMPTY, where the clause's negation composes with the operator's own; a negated parameterized call whose invoked rule reached a verdict; and a disjunction in which every disjunct skipped. All correct, which is worth stating as plainly as a bug would be — an audit that reports only its finds cannot be told apart from one that stopped early.

The negation fixtures are fussier than they look. A first version used a plain key path, which produces the right answer through a different code path entirely: unary_operation handles EMPTY on a filter-terminated or lone-variable query in a separate early-return block, and the clause-level flip lives inside it. The test passed while the arm stayed at zero. Coverage caught that the assertion was vacuous; reading the source would not have.

d56965c — the language docs

Two of these changes alter what a rule means, so they belong in the docs. Type blocks with a when condition were undocumented — the only prior mention of type blocks anywhere in docs/ is a passing note about variable scoping — so COMPLEX_COMPOSITION.md now covers the construct, the per-resource scoping, the three outcomes, and how to migrate a root-anchored condition. CLAUSES.md gains the numeric comparison rule under Binary Operators, with the when case spelled out because that is where the old behaviour cost enforcement rather than merely reporting a wrong verdict. KNOWN_ISSUES.md item 2 still holds for genuinely different kinds, so a note narrows it to exclude the numeric pair.

Every rule sample was run through parse-tree, and the worked example was executed against five templates to confirm each documented outcome. The last branch to write a docs claim about evaluator behaviour did not check it, and the claim was wrong.

f3c919f — a condition that cannot be decided

The quietest wrong answer of the set, and the one that shows why the others matter. CloudFormation templates carry numbers as strings routinely, and when ...Properties.Size > 10 against Size: "50" cannot compare a string to 10. The condition does not pass, so the rule is reported not applicable, so its body never runs. Exit 0, nothing named. Size: 50 fails the same rule.

The rule still does not enforce, and it cannot be made to from here. On a condition both FAIL and SKIP drop the block being guarded, and a gate FAIL is worse — the condition fold counts it and it outranks sibling conditions that passed, dropping a body those siblings would have kept enforced, which is why the empty-reference arms in 3f8466e are role-aware rather than failing outright. Telling "could not decide" from "decided false" where it matters needs a status meaning "could not tell", and Status has three variants, none of which is that.

What is available is saying so, which ca818b0 makes possible. The change is entirely reporter-side, and the discriminator is precise rather than heuristic: only two things record an explanation on a comparison — an empty reference and incomparable operands — and both mean the clause could not be decided, while the ordinary failure arm records no message. So a rule that legitimately does not apply stays silent, which is what makes the message worth reading.

46156f0 — a context that changed with the compiler

eval_conjunction_clauses built its record context with std::any::type_name, which is documented as being for diagnostics with no stability guarantee. This use was not confined to diagnostics: the result reaches verbose output, which four golden-file tests compare byte for byte. rustc 1.77.2 renders GuardClause<'_> and later versions drop the elided lifetime, so the suite passed on the pinned toolchain and failed on every newer one.

Not hypothetical debt: those four tests had to be skipped to measure branch coverage at all, since branch counters require nightly and one failing binary aborts the run before the later ones execute. Taking the path before the generic arguments is stable on both, and no golden file changed — they were already written for the normalised form.

9900d0b — an index applied twice

The query resolver handles a variable used where a map key is expected, and peeks at the following part: an index there says which of the resolved keys to use. Having consumed it, the recursion advanced by one anyway, so the index was applied a second time, to the value the key had just selected.

let names = Resources.Pointer.Properties.Targets[*]
rule r { Resources.%names[0].Type == 'AWS::S3::Bucket' }

%names[0] picked BucketA correctly, then [0] was applied to BucketA itself, the query resolved to nothing, and every part after [0] was discarded. Two things kept it hidden: the form without an index always worked, so the pair reads as a rule-authoring mistake; and an unresolved query is reported as a retrieval failure against the input, so the output blames the template. In a condition it is the same wrong PASS as the rest — the last case in the test is that shape, SKIP at exit 0 with an unreported violation inside.

dcf52ad — a failure message that printed everything

binary_error_in_msg computed its cut-off as max(values.len(), 5), which is never below the number of values, so the loop meant to stop early never did and the branch reporting a Total was unreachable. A rule comparing against a denylist of five hundred entries printed all five hundred, in every failure message, for every non-compliant resource. The dead branch is what gives the intent away: it exists to say how many there were when not all are shown.

Review response

@satyakigh raised two findings on the first revision. Both were real, and both are fixed above rather than argued with.

Role propagation was incomplete. 8c51b77 converted the strictness booleans to a ClauseRole so that an unthreaded path becomes a compile error, but eval_when_condition_block then hardcoded ClauseRole::Assertion for the guarded block, on the reasoning that a guarded block holds the rule's own assertions however the conditions were evaluated. That is wrong for a when nested inside another when: the inner body was re-labelled, so a gate's FAIL arrived at the leaf as a SKIP, and eval_conjunction_clauses absorbs SKIP rather than counting it, so one passing sibling answered PASS for the whole conjunction and the run exited 0.

70b9006 makes the parameter required rather than defaulted. A default reads as a deliberate choice at each call site and reintroduces the bug the first time someone adds a third caller. Both requested forms are covered: nested_when_inherits_the_enclosing_role pins the one-level case against its unnested twin, and the_role_reaching_a_leaf_clause_survives_every_nesting generates the 12-cell product of {negated, positive} leaf x {0,1,2} nesting depths x {assertion, gate} role and asserts each cell, so the behaviour is pinned at every depth rather than at the one that happened to be broken. SKIP is the signature of a leak, and the matrix distinguishes it from both PASS and FAIL.

The gate fixture uses two ANDed conditions on purpose. With a single condition, eval_rule maps every non-PASS condition to SKIP, so a gate that FAILs and one that SKIPs are indistinguishable and the test passes even against an unconditional FAIL — which an earlier version of it did.

The empty-reference explanation never rendered. This one also makes an earlier claim of mine wrong rather than merely incomplete. I told this review that the failure "names the cause and the remedy", and wrote the same into docs/CLAUSES.md. The message was constructed and stored, and report_all_failed_clauses_for_rules returned whatever a record's children reported — which for a record whose failure is described by its own message and which has no failing children is nothing at all.

de431f7 fixes it for every affected variant, not just the one that prompted the report. Six record types carry a message; only ClauseValueCheck had a rendering path. GuardClauseBlockCheck, TypeCheck and WhenCheck now fall back to their own explanation when no child reported, BlockGuardCheck prefers its stored message over a hardcoded sentence, and Disjunction does the same. The console reporter grew a section for explanations that carry no unresolved value, because populate_hierarchy_path_trees cannot attribute those to a resource and so pprint_clauses never reached them.

Three tests, per the requested shape: the two reporter-output cases assert the text reaches console and structured JSON, a negative control asserts a clean run prints no such section (so the first is not satisfied by a reporter that prints unconditionally), and every_recorded_explanation_has_a_rendering_path counts the message: Some construction sites against a per-variant table so a seventh message-bearing record cannot be added without wiring a reporter. That last test has already earned its place twice: once when rebasing surfaced a fourteenth site, and once when it refused a message I tried to add to a skipped type block, where records never reach a reporter at all.

Both fixes are mutation-verified. Reverting the role change fails exactly the two role tests; reverting the reporter fallback fails both output cases; suppressing only the console section fails only the console case.

Verification: output, not just verdicts

A second differential covers full output, driven by a generator rather than the repository's 45 rule
files: 6 clause families x 12 rule shapes gives 206 distinct rules, each against 14 input families in
2 output formats, for 5768 comparisons. Deterministic from a fixed seed, and every rules file also
gets a variant with an unrelated always-failing rule prepended -- the shape that caught the abort
regression, since with one rule per file an abort and a clean skip both leave nothing reported.

All 2365 output differences and all 326 exit-code differences are attributed to a named commit.
Zero unclassified. Zero abort-losses.

The generator immediately falsified one claim from the exit-code pass, and that is the useful part.
That pass asserted no 19 -> 0 transition -- a reported violation becoming a silent pass -- and the
generator produced 104. All are intended: a54e4ca and 0e140b3 apply a clause-level not that was
previously discarded, and 70feb75 makes an integer comparable with a float, so both flip verdicts in
both directions. The invariant is now "every 19 -> 0 is attributable to a construct whose meaning
intentionally inverted", checked per pair rather than asserted; the 28 numeric cases were each
confirmed by re-running the merge-base and checking its FAIL was specifically a mixed-numeric
NotComparable. Unexplained: 0.

It also found something the verdict differential could not. The console reporter emitted failing
rules in HashMap order: twenty runs of the merge-base over three failing rules produced five
distinct reports, six with --show-summary all. Pre-existing, but there is no useful sense in which
output is unchanged when the baseline disagrees with itself, so 07bcd4f sorts them. And 968 of the
differences are a violation's Code: snippet appearing where the merge-base printed a header with
nothing under it -- 2b215cc is a much larger fix than its diff suggests, and 7cefd51 closes a
second route to the same blank snippet.

The durable half is in CI: generated_rule_shapes_hold_the_evaluator_invariants runs 252 cells over
invariants that need no oracle -- canary isolation, determinism, and that a clause and its negation
never both pass. Two of those invariants were wrong when first written, in ways worth recording. "not X must differ from X" is false: an undecidable comparison fails closed in both polarities, which
docs/CLAUSES.md already documents. And the test's assertion that some cell must produce an
incompatible-type error failed on the first run, because no generated clause reached that path -- so a
documented exception was describing code nothing exercised.

Regression verification

A differential against the merge-base, since twenty-seven commits in an evaluator is a lot to take on
trust. Both binaries built from source, exit codes compared per (rules file, data file) pair.

Every rule file that existed before this branch, against every data file in the repository — 45 x
60 = 2700 pairs — produces an identical exit code. No shipped ruleset, example or fixture changes
its verdict.

Over an extended corpus including this branch's own fixtures, 911 pairs give 14 differences, all
0 -> 19, and each one is a fixture whose purpose is to demonstrate a documented change: thirteen on
the empty-reference rule and one on the type block over a template whose volume exceeds the threshold
and lacks Encrypted. Nothing moves 19 -> 0 (a reported violation becoming a silent pass) or
19 -> 255 (a violation discarded when a file aborts).

The exercise earned its keep by finding a regression this branch had introduced, which is why
e814a88 exists. Moving a type block's conditions per-resource removed an early return that had been
masking an error path, and an unresolvable type block query returned Err — which aborts the whole
rules file, so a violation an unrelated rule had already found stopped being reported. 19 became
255 and the finding vanished.

Worth recording how it was missed: the first corpus had one rule per file, and with a single rule an
abort and a clean skip are indistinguishable by exit code, since both leave nothing reported. Adding
canary variants — every rules file with an always-failing rule prepended — is what exposed it. A plain
type block reached the same Err on the merge-base, so the abort predates this work and is fixed
along with it.

What the differential does not cover: output, which deliberately changed in three places and is
covered by golden-file tests instead; rulesets outside this repository, where the type-block when
migration above does apply; and shapes a hand-built corpus does not contain, which is the class that
hid the regression in the first place.

Coverage, and a caveat about measuring it

Stable 1.77.2, the pinned toolchain, emits no branch counters, so the branch column needs nightly. But nightly's --branch instrumentation splits sub-expressions into separate regions — eval.rs reports 842 regions on stable and 1855 on nightly — and marks struct-literal fields and closing punctuation inside covered blocks as uncovered. At eval.rs:2070-2071, two fields of a single struct literal report 45 executions and 0. So nightly's region and line percentages carry an artifact floor and are not comparable to anything CI reports; only its branch column is meaningful. The two toolchains also cannot share a target directory, since stable's llvm-cov rejects nightly-built artifacts outright.

Region and line coverage on the pinned toolchain, before and after:

file	regions	lines
eval.rs	67.93% -> 72.93%	74.03% -> 77.88%
eval/operators.rs	74.34% -> 76.60%	74.84% -> 77.90%
eval_context.rs	68.01% -> 69.48%	72.09% -> 73.02%
reporters/validate/cfn.rs	69.48% -> 69.86%	77.03% -> 81.36%

Branch coverage, which only nightly reports: eval.rs 71.88% -> 79.03%, eval_context.rs 42.95% -> 48.73%, cfn.rs 63.33% -> 71.43%, operators.rs 82.14% (unchanged). The whole suite now passes on nightly as well, so those runs no longer skip anything — before 46156f0 four tests failed there and aborted the run.

The gains are modest next to the findings, and that is the honest shape of the result: the wrong-PASS class lives in the fold and role paths, which are a small share of the line count. What remains uncovered is mostly not status-deciding — query_retrieval_with_converter (173 lines of query shapes) and report_all_failed_clauses_for_rules (153 lines of reporter variants) in eval_context.rs, operators::compare (101), and the reporter's error-message formatting helpers (47 in cfn.rs).

Rather than chase the percentage, the sweep enumerated every line in these four files that produces or folds a PASS, FAIL or SKIP and had never executed, then closed the reachable ones. What is left is 13 error-propagation arms that require the resolver itself to fail, one unreachable!(), and the EmptyRhs arm already commented as unreached because its wrapper resolves the case into two other variants. None of them decides a verdict for input a rules file can express. cfn.rs has no uncovered status decisions at all.

Verification

• 317 unit tests and 110 integration tests pass, with nothing failing, on both the pinned 1.77.2 and nightly. 45 tests are new, carrying roughly 500 assertion cells. Every new test was verified to fail without its fix; two early ones passed against the reintroduced bug and were rewritten.
• Differential against a pristine v3.2.0 build: 491 rule/template pairs across guard/resources and guard-examples produce zero exit-code differences. All 24 shipped example rulesets are unaffected.
• Adversarial fixtures: ~90 hand-built shapes covering both polarities crossed with conjunction, disjunction in both orders, when gates, nested when, type blocks, some blocks, parameterized rules, multi-rule files, and map-key filters. No case where v3.2.0 exits 19 and this branch exits 0.
• cargo clippy -- -D warnings, cargo fmt --check, typos and shellcheck install-guard.sh are clean.

An earlier revision of this description reported 15 failures in validate and 17 in test_command as pre-existing. That is no longer accurate and the count was also wrong about the cause. Two separate things were happening: a test harness that resolved fixture paths through $HOME rather than the crate directory, fixed in b6d384d, and four golden files that compare output containing std::any::type_name, which is not stable across compiler versions. The latter pass on the pinned 1.77.2 and fail only on a newer toolchain, so CI is unaffected — but they will break on a compiler bump, and that is worth knowing independently of this PR.

Intended behaviour changes

All of these are the point rather than side effects, but they are behaviour changes and belong in release notes.

Rulesets that pass today because a reference resolved empty, or because a not was silently dropped, will now fail — those were false passes. The not <query> <op> <value> form in particular changes meaning in both directions, because it previously evaluated as its own inverse.

Rules comparing an integer against a float now decide instead of failing, so a rule that failed a compliant template stops doing so, and a gate that silently skipped now applies.

Type block conditions need migrating in one case. A when on a type block is now evaluated against each matched resource rather than against the document root. The natural spelling starts working:

AWS::EC2::Volume when Properties.Size > 10 { Properties.Encrypted == true }

That previously looked for Properties at the root, found nothing, and reported the rule not applicable for every input. The reverse also holds: a condition written as a literal root-anchored path no longer resolves.

# was fine, now does not resolve
AWS::EC2::Volume when Resources.MyVolume.Properties.Size > 10 { ... }

# either express it relative to the resource
AWS::EC2::Volume when Properties.Size > 10 { ... }

# or move it to the enclosing rule, whose conditions are still root-scoped
rule r when %volumes !empty {
    AWS::EC2::Volume when Properties.Size > 10 { ... }
}

The variable idiom is unaffected, since variables resolve through the parent scope. Nothing in this repository used the root-anchored form — a search found two occurrences of a type block carrying a when, both fixtures added here — but external rulesets may. Documented in docs/COMPLEX_COMPOSITION.md.

Two output changes. parse-tree serialises a map key filter's comparator as Eq rather than the two-element sequence [Eq, false], and != as NotEq. And a rule reported as not applicable may now carry a line explaining why, in the console summary and as an optional not_applicable_reasons field in the structured formats. Both serialised additions are omitted when empty, so a run with nothing to explain produces byte-identical output to before.

Not addressed

Recording these so reviewers know they were considered rather than missed. All are pre-existing in v3.2.0 and none is a regression from this branch:

• A property whose value is an empty collection satisfies a match_all block clause: Resources.*[ Type == 'AWS::S3::Bucket' ] { Properties.Tags == "Owner" } passes a bucket with Tags: []. The ALL fold sees zero per-value results and falls through to PASS, having discarded the polarity. Tags[*] and some variants fail correctly, so it is specific to the bare form.
• A misspelled resource type in a filter or type block silently skips the rule and exits 0, with no diagnostic. Likewise a glob matching zero data files.
• A failing filter predicate collapses to an empty selection, so the enclosing assertion goes vacuous. This commit relabels filter predicates as gates, which is semantically right, but the label is currently inert because these paths already treat FAIL and SKIP identically.
• EMPTY on a boolean is always answered "not empty", because the implementation computes to_string().is_empty(). A fix for this was prepared and dropped from this PR: it converts the case to an error, and that error aborts the entire rules file, so a violation in an unrelated rule stops being reported. Correcting it needs a per-clause failure rather than a propagated error, which is a separate change.
• Status::and is the only correct three-valued operator in the codebase and none of the fourteen evaluation folds use it; each is a hand-rolled counter. That inconsistency is the structural root of several items above, including the disjunction asymmetry described under 3f8466e. Worth addressing, but far too large to sit alongside these fixes.

Notes for reviewers

The ClauseRole distinction in 8c51b77 is the load-bearing idea. An unevaluatable clause has to fail as an assertion and skip as a gate, because a failing gate makes its rule inapplicable and drops every check inside it — so the same clause needs two different answers depending on why it is being evaluated. If a future change adds a third evaluation context, that enum is where it belongs.

I would rather over-explain the reasoning than have a reviewer reconstruct it, so the commit messages are long and each records what was tried and rejected. Happy to squash, split, or drop any commit.

[awsmadi]

You're right that the current behaviour under-enforces, and the code comment claiming "the weaker status costs nothing in the standalone case" is wrong. What it costs is the enforcement of the clause: a rule whose only check is Property != %empty_reference exits 0 having compared nothing.

One useful thing fell out of looking at this. The literal-vs-query distinction you're asking for is already structural rather than something that needs new plumbing. EmptyRhs is raised from rhs.is_empty() in CmpOperator::compare, and that can only be true for a query-derived reference, because a literal [] produces exactly one QueryResult::Literal. So an intentionally empty literal collection never reaches this path at all, and the change you're describing reduces to making EmptyRhsVacuouslyTrue a definite failure.

Which is why I want to check the semantics with you before pushing it, because I don't think that distinction separates the cases we actually care about. Consider:

let banned = Resources.*[ Type == 'AWS::Foo' ]   # template has no Foo
Properties.Tags not in %banned

That reference is query-derived and resolved to nothing, and the template is compliant. Failing it rejects correct input, and docs/QUERY_AND_FILTERING.md:222 documents empty query results as a skip rather than a failure. An empty literal [] and a filter that legitimately matched nothing are both intentional. What actually marks a mistake is that the author expected the reference to be non-empty, and the ruleset never says so.

Two ways to take it, and I'd rather you pick:

1. Fail closed on query-derived empties, as you describe. It closes the hole and it's a small change. It will also fail the compliant filter-matched-nothing case above, which I believe is common in practice, so it's a deliberate semantics change worth calling out in release notes.

2. Make the author's expectation explicit. Keep the current status, and treat a required-non-empty reference as something the ruleset declares (an accompanying %banned !empty clause, say), so the expectation is checkable rather than inferred from how the reference was written.

Either way I'll add the CLI-level regression test you asked for, asserting that the standalone negated rule does not reach a successful exit without enforcement.

The other two comments are addressed in the latest push:

• Boolean EMPTY: removed the Bool arm from element_empty_operation so a boolean reaches the same IncompatibleError every other unsupported type already reached. You were right that no test followed the comment, and right that the comment described a fix that wasn't there. boolean_empty_is_an_incompatible_type now covers all four combinations of value and polarity, both axes deliberately: the old code made !EMPTY a silent pass and EMPTY a silent fail, so one polarity alone would leave the other unguarded, and true vs false is exactly the axis the old implementation was insensitive to.

• Parameterized gate SKIP: a non-negated call whose invoked rule SKIPs now stays SKIP instead of falling through to FAIL. Worth noting why the test has two conditions, since with one the bug is invisible: FAIL and SKIP both make eval_rule drop the guarded body, and the difference only shows up because eval_conjunction_clauses absorbs SKIP but counts a FAIL. So a single inapplicable gate condition was poisoning the whole when and dropping a body the passing sibling should have kept enforced. Negated calls still fall through deliberately, matching the fail-closed reasoning in eval_guard_named_clause.

Both fixes are mutation-checked: reverting either one fails its new test and nothing else. The gate test took two attempts. The first fixture used !empty, which fails rather than skips on an unresolved query, so it reached the same arm by a different route and passed without exercising the SKIP path.

Also fixed cargo fmt --check, which this branch was failing.

[awsmadi]

@Zee2413 ^^

[Zee2413]

You're right that the current behaviour under-enforces, and the code comment claiming "the weaker status costs nothing in the standalone case" is wrong. What it costs is the enforcement of the clause: a rule whose only check is Property != %empty_reference exits 0 having compared nothing.

One useful thing fell out of looking at this. The literal-vs-query distinction you're asking for is already structural rather than something that needs new plumbing. EmptyRhs is raised from rhs.is_empty() in CmpOperator::compare, and that can only be true for a query-derived reference, because a literal [] produces exactly one QueryResult::Literal. So an intentionally empty literal collection never reaches this path at all, and the change you're describing reduces to making EmptyRhsVacuouslyTrue a definite failure.

Which is why I want to check the semantics with you before pushing it, because I don't think that distinction separates the cases we actually care about. Consider:

let banned = Resources.*[ Type == 'AWS::Foo' ]   # template has no Foo
Properties.Tags not in %banned

That reference is query-derived and resolved to nothing, and the template is compliant. Failing it rejects correct input, and docs/QUERY_AND_FILTERING.md:222 documents empty query results as a skip rather than a failure. An empty literal [] and a filter that legitimately matched nothing are both intentional. What actually marks a mistake is that the author expected the reference to be non-empty, and the ruleset never says so.

Two ways to take it, and I'd rather you pick:

1. **Fail closed on query-derived empties**, as you describe. It closes the hole and it's a small change. It will also fail the compliant filter-matched-nothing case above, which I believe is common in practice, so it's a deliberate semantics change worth calling out in release notes.

2. **Make the author's expectation explicit.** Keep the current status, and treat a required-non-empty reference as something the ruleset declares (an accompanying `%banned !empty` clause, say), so the expectation is checkable rather than inferred from how the reference was written.

Either way I'll add the CLI-level regression test you asked for, asserting that the standalone negated rule does not reach a successful exit without enforcement.

The other two comments are addressed in the latest push:

* **Boolean `EMPTY`**: removed the `Bool` arm from `element_empty_operation` so a boolean reaches the same `IncompatibleError` every other unsupported type already reached. You were right that no test followed the comment, and right that the comment described a fix that wasn't there. `boolean_empty_is_an_incompatible_type` now covers all four combinations of value and polarity, both axes deliberately: the old code made `!EMPTY` a silent pass and `EMPTY` a silent fail, so one polarity alone would leave the other unguarded, and `true` vs `false` is exactly the axis the old implementation was insensitive to.

* **Parameterized gate SKIP**: a non-negated call whose invoked rule SKIPs now stays SKIP instead of falling through to `FAIL`. Worth noting why the test has two conditions, since with one the bug is invisible: `FAIL` and `SKIP` both make `eval_rule` drop the guarded body, and the difference only shows up because `eval_conjunction_clauses` absorbs `SKIP` but counts a `FAIL`. So a single inapplicable gate condition was poisoning the whole `when` and dropping a body the passing sibling should have kept enforced. Negated calls still fall through deliberately, matching the fail-closed reasoning in `eval_guard_named_clause`.

Both fixes are mutation-checked: reverting either one fails its new test and nothing else. The gate test took two attempts. The first fixture used !empty, which fails rather than skips on an unresolved query, so it reached the same arm by a different route and passed without exercising the SKIP path.

Also fixed cargo fmt --check, which this branch was failing.

Option 1, fail closed on query-derived empties.

The two error modes aren't symmetric: a wrong FAIL is visible and gets investigated; a wrong SKIP exits 0 and is indistinguishable from PASS in CI, which is the deny-list bypass this PR exists to close. Option 2 leaves every ruleset without an explicit !empty guard silently defeatable, which is the current state.

The legitimate empty-reference case keeps an escape hatch under option 1: an author who expects a possibly-empty reference can gate with when %reference !empty { ... }, and the gate role keeps that a SKIP. So the idiom from option 2 still works — option 1 just makes the unstated-expectation case fail closed instead of open.

Two asks with the change:

1. Make the failure message state that the reference resolved to no values, so the FAIL is actionable.
2. Call it out in release notes as a deliberate behavior change, alongside the CLI-level regression test you offered.

[awsmadi]

@Zee2413 all four were real, and all four are fixed. Measured at the current head, 99ffeff, with your fixtures.

Boolean !EMPTY as a gate

Fixed in 4c8c650. Your diagnosis was exactly right, including the part that made it my mistake rather than a pre-existing one: making the boolean clause fail closed as an assertion without doing the same at the gate turned a bug that over-reported into one that under-reported.

base 57bbdbf   exit 19   guarded FAIL   (only because !EMPTY on a boolean was unconditionally true)
head b116f48   exit  0   guarded SKIP   (the regression you found)
head 99ffeff   exit 19   guarded FAIL   with "could not be evaluated" as the reason

The regression test is an_unevaluatable_gate_fails_the_rule_closed in guard/tests/validate.rs, end to end through the CLI. It asserts three things, because two of them would have passed while the third was broken: the exit code, that the rule with the unevaluatable gate is named in the report, and that an unrelated failing rule is still named too. Failing closed is only correct if it does not take the rest of the file down with it.

Why this could not be expressed as a status is worth stating, since it shapes the fix: eval_rule collapses both FAIL and SKIP on a condition into a rule-level SKIP, because "the condition did not match" is the ordinary gating idiom and has to stay a skip. So an unevaluatable condition travels as an error and is caught at the rule boundary. #720 replaces that with a value that can say "could not tell", which is the structural version of the same fix.

Your point about the generated corpus was the more useful half of the finding. You were right that its canary checks cannot see this, and the reason generalises: canary isolation, determinism and negation-discrimination are all satisfiable by an answer that is wrong and self-consistent. All 252 cells passed while this defect was present. So the corpus now also carries an explicit list of the 22 shapes where an undecidable comparison still disarms its body, checked as exhaustive in both directions, and 30237d8 adds a 440-cell oracle test that states the expected verdict for every operator and operand shape rather than asserting relationships between them. That is the part that would have caught this by construction.

Oversized indexes

Fixed in 3417a81. Your five clauses, against {"Items":["safe","other"]}:

Clause	base 57bbdbf	head 99ffeff
Items[4294967295] == "other"	exit 0	exit 19
Items[4294967296] == "safe"	exit 0	exit 19
Items[4294967297] == "other"	exit 0	exit 19
Items[9223372036854775807] == "other"	exit 0	exit 19
Items[-4294967296] == "safe"	exit 0	exit 19

An index that cannot be represented now names no element rather than wrapping onto one, so the clause fails instead of comparing something the author never asked about.

The indexed-interpolation path you asked about is covered too:

let names = Resources.Pointer.Names.*
rule r { Resources.%names[<index>].Properties.Name == "a" }

Every oversized index there exits 19 on this branch. That path also had a second defect, unrelated to yours and found separately: the index was applied twice, once to select which resolved key to use and again to the value that key had selected, so %names[0] exited 19 against input where it should pass. It now exits 0.

Range equality

Fixed in b5153de. a_range_is_equal_to_itself covers all three range variants rather than the two in your report, because the defect was the missing arm rather than anything about a particular type: RangeInt, RangeFloat with positive and with negative bounds, and RangeChar. Each asserts both == and hash equality, so the Eq and Hash contracts are covered together.

Two adjacent tests keep the surrounding behaviour pinned: equality_is_symmetric_for_ranges, for the symmetry hole whose fix opened this reflexivity one, and mixed_numeric_range_membership_is_decided, since membership across integer and float is where the arms are consulted from.

JUnit skip reasons

Fixed in b5153de as well. The reasons are joined with an explicit delimiter, and junit_keeps_multiple_skip_reasons_separate splits the <skipped> body on it and asserts two reasons rather than checking that both rule names appear somewhere. That distinction matters for exactly the reason you gave: an assertion that both names are present passes on the concatenated string too. Your note about some evaluator reasons ending in whitespace is recorded in the test, because it explains why the CLI fixtures did not catch it.

Also since b116f48

Six more commits. The first three came out of verifying the rest against the rules registry rather than from reading code, which is the pattern I would keep:

• d9de29c — cfn-guard test listed rule names in HashMap order, so ten consecutive runs of the same command over the same files produced ten different reports. Found because the registry differential reported ten rule files as changing their output when nothing had changed; both binaries were simply unstable. All four output formats now agree run to run.
• 78f1f11 — the deprecation notices this branch adds reached validate but not test, which is backwards: the notice is addressed to whoever wrote the rule, and test is the command they run. Not one of the 190 registry rule files reported a notice before this.
• 90066da — an expectation naming a rule the file does not contain was dropped in silence. cfn_no_explicit_resource_names_tests.yml in the registry has 11 such expectations, none of which name a rule that exists anywhere in the registry, and cfn-guard test exits 0 on it.
• d4fea74 — corrects the oracle's expectation for an empty collection, which had one answer where it needs two: a claim over nothing is a failure, and the negated form is vacuously true and asserts nothing.
• 99ffeff — the rule-level failure line announced every such failure as a "Parameterized Rule". That was accurate at the merge-base, where the only failures carried on the rule rather than on a clause comparison came from parameterized calls; it stopped being accurate once this branch gave ordinary rules a rule-level message. I found it reading the output of your own repro while writing this reply, which makes it the third diagnostic on this branch that said something false, so the regression test now asserts the absence of the word as well as the presence of the reason.
• 576dd22 on Give the evaluator a value that can say "nothing to compare", and fix the empty-collection wrong passes #720 and efeed83 here — comments only. 53 em dashes in a crate whose merge-base has no non-ASCII byte, and four commit hashes that rebasing had already invalidated.

The registry differential itself: 190 rule-file/test-file pairs, 1,956 expectation checks, 0 exit-code changes and 0 content changes against the merge-base. Five rule files now print the NOT IN deprecation notice, which is the one behaviour change on this branch with demonstrated real-world exposure and the reason that change is deferred rather than made here.

Verification at 99ffeff: 818 tests passing, clippy -- -D warnings and clippy --all-targets -- -D warnings clean, fmt --check clean, typos clean.

@satyakigh the formal CHANGES_REQUESTED is still yours, from the review whose body reads "Oops". Your substantive finding, the boolean EMPTY abort, is fixed by the second of the two options you offered: incompatible-type errors are clause-local, with the multi-rule CLI regression test. If you still think that clause-local change belongs in its own PR, say so and I will split it out; otherwise a re-review when you have time would unblock this.

[awsmadi]

The nested-gate finding is real, and the diagnosis was exact. Fixed, along with two more instances of the same conflation that came out of chasing it. Measured at c5027fc.

The finding

I could not reproduce it twice before I could, and the failed attempts are worth one line each because they say something about the shape. My first repro put the undecidable clause behind a rule parameter (%prop !EMPTY), which turns out not to be undecidable at all — that is the second defect below. My second used a root-anchored condition inside a type block, which does not resolve on this branch by design.

With the clause on a direct path inside a when nested in a parameterized rule's body, and a control that proves the shape can reach and fail the body:

                              base    this branch    fixed
param gate + nested when      19      0              19
control (decidable guard)     19      19             19

eval_when_condition_block converted the undecidable answer to Status::FAIL. One level out, a FAIL on a condition is indistinguishable from a condition that was decided and did not match, and eval_rule maps that to a rule-level SKIP. Exactly as described: FAIL means both "the condition is false" and "the condition could not be evaluated", and converting the second one early loses what the outer rule needs.

The fix splits that arm by role, which is what the arm in unary_operation already did. An assertion still answers FAIL — that is what stops one undecidable clause from aborting the whole file, the earlier finding on this PR — and a gate keeps the error, so the enclosing condition site fails its own rule closed instead of deciding the rule does not apply.

Two more of the same, found by sweeping rather than reading

The named-rule spelling was worse, and the exit code hid it. rule guarded when inner_gate with the same nested condition reported guarded SKIP where the merge-base reports FAIL. The cause was one layer further in: rule_status evaluated a named rule once, always as an assertion, and cached it under the rule's name alone, so the gate reference read a verdict computed for a different question. Keyed on (rule, role), the reference asks its own question and the undecidable answer survives.

What makes this one instructive is why it stayed hidden. inner_gate is also a top-level rule, so its own failure exits the file 19 whether or not the rule it gates ever runs. Any invariant of the form "the body fails, so the file must not exit 0" passes here while the guarded check is silently dropped. The tests now assert on the rule, not the file.

EMPTY on a lone variable could not fail. Chasing the first failed repro: %flag !EMPTY was answered by res.is_null(), not by emptiness. Against Tags: [], Name: "", Enabled: true, Size: 50:

clause	correct	before
%tags !EMPTY (list is empty)	FAIL	passed
%tags EMPTY (list is empty)	pass	FAILED
%name !EMPTY (string is empty)	FAIL	passed
%flag !EMPTY (bool — no answer)	FAIL	passed
%size !EMPTY (int — no answer)	FAIL	passed

Both polarities wrong on a value whose emptiness is the entire question, and two of them are the silent always-pass the boolean fix removed for the direct path. A let binding and a rule parameter both take that shortcut, and no test or matrix cell reached the operator that way.

The shortcut itself has to stay, and that boundary took two attempts: an empty selection resolves to zero values and is answered before the per-value loop, so removing the arm would turn %vols !empty — the most common gate in the registry — from a failure into a silent skip. A query ending in a filter also keeps resolution semantics, because Condition[ keys == 'aws:IsSecure' ] !empty means "that key is present" and the value behind it may be a boolean. block_evaluation, an upstream test, is what said so; my first version failed it.

The reporting half

The fixed shapes exited 19 and said nothing: Number of non-compliant resources 0 with no reason anywhere. The explanation was recorded and the JSON reporter had always printed it — only the console dropped it, because that output is organised by resource and a rule that failed on its own condition points at no value. Now:

Could not be evaluated:
  rule guarded
    The rule's condition could not be evaluated, so the rule fails rather than being treated as
    not applicable: Attempting EMPTY operation on type bool ... at /Resources/Vol/Properties/Enabled

Both regression tests assert the reason as well as the verdict. They passed on the verdict alone while the console said nothing, which is the same gap one level out.

What generalises

The 252-cell corpus asserts the right invariant and could not catch any of this, because it varies the operator and the operand and wraps exactly one clause in exactly one gate. Every escape in this family composed instead: the answer crossed more than one conversion site, and each site that converted early lost what the next one needed. There were three such sites on this branch.

So there is now a sweep over two axes the matrix holds fixed — where the clause sits relative to the guard (gate condition, nested when, named reference, parameterized call) and how the value reaches it (direct query or let variable) — 18 cells, each with a control that proves the shape can fail the body at all. Half the cells in my first version of that sweep were dead for want of a control, which is why they are there.

It found a defect in my own fix within a minute of existing: the lone-variable arm returned an error for a gate between start_record and end_record, leaving the recorder unbalanced. Verified live afterwards — reverting the role split fails the sweep and names the cell.

Verification

824 tests, clippy -- -D warnings and clippy --all-targets -- -D warnings clean, fmt --check and typos clean. The 440-cell matrix moves zero cells. The registry differential is unchanged: 190 rule-file/test-file pairs, 1,956 expectation checks, zero exit-code changes and zero content changes — so none of these fixes moves a published rule, and none needed a deprecation notice.

The stacked PR carries the same three fixes, and its own instance of the last one: its cache was keyed on (rule, role) but stored to_status(role), so Unevaluatable became SKIP for a gate and the reference read "did not apply" for a rule that had not decided anything. It stores the answer now.