core: list the items that failed an all or any check

[jbpros]

Closes #68.

A failing all or any row shows the whole list it read, and the reader has to work out which items failed. Each such row now has failed_items.

Example

Input:

{ "deployments": [ { "id": "dep-1", "deployed_by": "bob",
    "pull_requests": [
      { "number": 7, "commits": [ { "sha": "a1", "author": "ann" } ] },
      { "number": 8, "commits": [ { "sha": "b1", "author": "ann" }, { "sha": "b2", "author": "bob" } ] } ] } ] }

Policy (DEV-0501 from #113):

segregation_of_duties:
  subject_type: deployment
  from: [deployments, { each_as: deploy }]
  id: [id]
  checks:
    not_self_deployed:
      op: all
      path: [pull_requests]
      each: [commits]
      check: { op: compare, left: [author], right: [$deploy, deployed_by], cmp: ne }

The row, from this branch:

"passed": false,
"cause": "value",
"inputs": [
  { "name": "pull_requests[].commits",
    "value": [ [ {"author": "ann", "sha": "a1"} ], [ {"author": "ann", "sha": "b1"}, {"author": "bob", "sha": "b2"} ] ] },
  { "name": "$deploy.deployed_by", "value": "bob" }
],
"failed_items": [
  { "path": "pull_requests[1].commits[1]", "cause": "value", "value": {"author": "bob", "sha": "b2"} }
]

The rules

• Where it goes: a field of its own on the row, not an entry in inputs, because it's a result rather than something ergo read. inputs doesn't change, so the row still shows everything that was checked.
• What's in it: the items that made the check fail, in list order, each with its path, cause and value. The path is written like a path in an expression, with the item's position added.
• Bad inner lists: with each, an inner list that's missing, null, not a list or empty is listed in place of its items, for example pull_requests[1].commits with cause absent.
• When it's []: when the check passed, and when the list itself can't be read, because the row's cause already says why.
• Checks written wrong: every item and every bad inner list is listed with cause ill_formed. When a ref can't be read, every item gets the ref's cause.
• Nested checks: only the outer items are listed. An approver who approved before the last commit is listed, but the commit isn't.
• Substitutes: it lists the main check's failed items, even when the substitute passed.
• Which rows have it: only rows of all and any checks. Rows of any_of, $applies and custom operators don't.
• Violations keep the field.

Changes the report: rows of all and any checks, and their violations, have a new failed_items field.

One change from what we discussed: a passing any gets [] rather than its failed items. Listing them meant checking every item, where any otherwise stops at the first item that passes, and that made those rows 3.5 times slower. With [], the rule is simply "items that made the check fail".

Checks

• opa check --strict, opa fmt --list, opa test (829/829), --strict-builtin-errors eval, Regal, the comment check and coverage: all clean.
• OPA 1.20.2 in Docker: formatted, 829/829. Wasm target in Docker: 829/829. JS Wasm runtime: 829/829.
• 2,500 random policies with all/any checks (using each, as, refs, substitutes and broken checks) through main and the branch:
  • Everything except failed_items is identical to main.
  • failed_items is on exactly the all/any rows, is [] on every passing row, never lists a passing item, has no repeated paths, and matches in violations.
  • Every failing row with an empty failed_items has a top-level list that's missing, empty or not a list.
• An item is listed exactly when the same all fails on a list holding just that item: 0 mismatches in 1,500 random checks (with refs that can't be read, any_of, nulls and wrong types). On the first version of this branch it found 33, from the bug fixed in 70f9fea.
• Speed: 2,000 subjects with three list checks of 20 items each take 4.15s, against 3.85s on main.
• Downstream: sdlc-policies fails the same 4 tests as main, and npm-bump passes 6/6.

#35 and the playground (#124) should show failed_items once this lands.

🤖 Generated with Claude Code