Skip to content

fix(cli): group sampled check findings by element and rule - #5174

Merged
miguel-heygen merged 5 commits into
mainfrom
fix/check-group-sampled-findings
Oct 8, 2026
Merged

miguel-heygen merged 5 commits into
mainfrom
fix/check-group-sampled-findings

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

A held low-contrast element previously appeared once for each failed sample. Check now reports one finding per source file, element selector, and rule, with the sorted occurrence times in JSON and terminal output. Real low contrast still fails the check; its worst sample supplies the color and screenshot anchor. Layout findings retain their existing persistence grading, and the explicit no-collapse option retains individual layout observations.

Findings in separate composition files stay separate even when their selectors match. Sample-level contrast checked/passed counts remain measurements, while finding counts represent grouped problems.

Validation: four reporter regressions fail on main, and the same-class connector identity regression also fails before its fix. The check suite passes 76 tests in three consecutive serial runs; browser audit tests pass 162 tests in three runs, and layout utility tests pass 39 tests. CLI typecheck and commit checks pass. Actual source CLI browser runs on a synthetic broken fixture produce one overlap and one contrast error, with five contrast times listed. The intentional-layout-marker control still reports the same real contrast error; JSON and terminal output both show it once. Both commands retain exit 1, and the disjoint-scene control stays clean with exit 0. These controls were repeated at the final head.

Layout findings use the existing unique-selector owner so separate same-class elements remain separate. A real-browser two-connector fixture reports two distinct findings, each listing its repeated times and retaining its own geometry.

Frame and caption observations now reach the same reporter before consolidation, so stationary failures retain all sampled times and separate elements with identical text stay separate. A stationary-frame regression fails before this fix.

Grouped snapshot evidence keeps its representative time paired with its original geometry; firstSeen records the earliest occurrence separately. A sparse-before-dense regression reproduces the old mismatched crop and verifies the corrected crop request.

@miguel-heygen
miguel-heygen marked this pull request as ready for review October 7, 2026 08:57

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 809aca4b (full PR).

  • Grouping. groupSampledFindings keys on [sourceFile, selector] plus code, merges the sample times, and keeps the higher severity. For contrast it keeps the worst ratio (checkPipeline.ts:1343-1347). contrastFailureHeld now keys on the element instead of selector|text, so an element whose text changes between samples (a counter, say) still counts as held, and its real failures stay errors. Layout issues only group under collapse-static, which is on by default.
  • Selectors. uniqueSelectorFor (layout-audit.browser.js:94) falls back to an nth-of-type path when the preferred selector matches more than one element. That stops two look-alike elements from merging into one row.
  • Tests. check.test.ts, layout-audit.browser.test.ts and layoutAudit.test.ts pass locally, 276/276. Mutants:
    • caught: grouping key ignores code; contrast keeps the first sample instead of the worst; held key reverted to selector|text.
    • survived: severity never escalates within a group. For contrast that mutant is equivalent, because held is per element, so every sample has the same severity. For layout codes whose severity can vary by sample (panel_out_of_canvas warning/info) it isn't pinned.

Nits, none blocking:

  1. textOverflowIssues and clippedTextIssue call uniqueSelectorFor before they know there's an issue. That runs a querySelectorAll for every text element on every sample, and recurses up the tree when the selector isn't unique. On a large DOM, computing it only once a finding exists would avoid that cost.
  2. With --no-collapse-static, caption-zone and frame findings used to be deduplicated to their first sample. They now print one row per sample. That's probably intended for the opt-out, but it's worth a line in the changelog.
  3. An nth-of-type selector can change between samples if the timeline adds or removes siblings, which would split one element's rows.

CI: no required check is red. Build and the Windows render were still pending when I approved.

— Somu

@miguel-heygen
miguel-heygen marked this pull request as draft October 7, 2026 09:23
@miguel-heygen
miguel-heygen marked this pull request as ready for review October 7, 2026 09:30

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at a9d71ccf: the fix commit over my approved 809aca4b (1 runtime line plus a test).

collapseStaticLayoutIssues no longer overwrites time with firstSeen (layoutAudit.ts:266). A collapsed issue now keeps the time of the sample its rect/bbox came from, and groupSampledFindings carries that whole finding through, so the time and position always describe the same sample. Range labels still come from firstSeen and lastSeen (layoutAudit.ts:150), so the printed span doesn't change.

The three CLI suites pass locally (277/277). Putting the time: firstSeen line back makes the new test fail, so it's pinned. My earlier nits still stand and aren't blocking. No required check was red when I approved.

— Somu

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Edit accuracy: accurate 2059 (base branch 2059), smooth 1528 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 8, 2026
Merged via the queue into main with commit 188475a Oct 8, 2026
81 checks passed
@miguel-heygen
miguel-heygen deleted the fix/check-group-sampled-findings branch October 8, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants