Skip to content

feat: check quality and target in the regression gate - #39

Merged
nucliweb merged 1 commit into
mainfrom
feat/harden-regression-gate
Oct 2, 2026
Merged

nucliweb merged 1 commit into
mainfrom
feat/harden-regression-gate

Conversation

@nucliweb

@nucliweb nucliweb commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Summary

Closes the three regression-gate items on the ROADMAP.

Check Before After
Size bytes > +2% fails unchanged
Quality not checked ssimulacra2 drop > 0.5 points fails as QUALITY, even if the file is smaller
Target not checked a run target different from the baseline's fails with an explicit error, comparing nothing
CSV split on commas writer quotes fields (RFC 4180), parser honours quotes
  • baseline.json now stores { bytes, ssimulacra2 } per image and codec. It is regenerated from the latest main gate run's results.csv (amd64, like CI); bytes change only in the AVIF and HEIC pairs the trixie move had already shifted (≤0.41%).
  • regression.yml defines TARGET once for the comparison and the check; the comment reads "no size or quality regression".
  • check-regression.js gains --quality-tolerance, applies --target in check mode, and prints quality next to size.
  • regression.js on main held a raw NUL byte as its key separator, so git treated the file as binary and never showed its diffs. It is now the \u0000 escape: same behaviour, plain text.
  • Docs: both READMEs describe the three checks, and the baseline regeneration instructions use the CI CSV, since sizes differ slightly across architectures.

Test plan

  • TDD: 10 tests in regression.test.js (7 failed before the change); node --test 51 pass, 0 skipped
  • A CSV with JPEG XL at −0.8 points and −2.2% bytes is reported as QUALITY, exit code 1
  • Checking at target 85 against the target-90 baseline fails with an explicit message, exit code 1
  • A real compare-codecs run on an image named sunset, beach.png writes a quoted field that parses back correctly
  • The workflow's commands inside the Docker image (arm64) pass; cross-architecture quality differences are ±0.04 points, well inside the 0.5 tolerance
  • actionlint passes
  • Gate on this PR (amd64) passes against the regenerated baseline

The gate compared bytes only, so a codec could pass as smaller by settling at a lower quality within the search tolerance, and a changed target would compare unrelated numbers without warning. The baseline now records each codec's achieved ssimulacra2 score, and the check fails on a drop of more than 0.5 points or a target that differs from the baseline's; the workflow defines the target once for both steps. The CSV writer quotes fields and the parser honours quotes, so an image name with a comma no longer breaks the results. The baseline is regenerated from the latest amd64 CI run, and the key separator in regression.js is now an escape instead of a raw NUL byte, so git diffs it as text.
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codec comparison ✅ no size or quality regression

📥 Download the codec report (unzip and open report.html locally). Available until 2026-12-31, when GitHub deletes the artifact.

Size check
Size regression check (target 90, size tolerance 2.0%, quality tolerance 0.5 ssimulacra2)

  ok         kodim04.png  JPEG XL  base 120534 → 120534  (+0.00%)  ss2 90.23 → 90.23  (+0.00)
  ok         kodim04.png  jpegli  base 130656 → 130656  (+0.00%)  ss2 90.02 → 90.02  (+0.00)
  ok         kodim04.png  HEIC  base 141631 → 141631  (+0.00%)  ss2 89.77 → 89.77  (+0.00)
  ok         kodim04.png  AVIF  base 150019 → 150019  (+0.00%)  ss2 90.11 → 90.11  (+0.00)
  ok         kodim04.png  WebP  base 173748 → 173748  (+0.00%)  ss2 89.81 → 89.81  (+0.00)
  ok         kodim04.png  mozjpeg  base 201372 → 201372  (+0.00%)  ss2 90.13 → 90.13  (+0.00)
  ok         kodim19.png  JPEG XL  base 136558 → 136558  (+0.00%)  ss2 90.02 → 90.02  (+0.00)
  ok         kodim19.png  jpegli  base 147032 → 147032  (+0.00%)  ss2 90.02 → 90.02  (+0.00)
  ok         kodim19.png  AVIF  base 148498 → 148498  (+0.00%)  ss2 89.74 → 89.74  (+0.00)
  ok         kodim19.png  HEIC  base 155322 → 155322  (+0.00%)  ss2 89.94 → 89.94  (+0.00)
  ok         kodim19.png  WebP  base 173918 → 173918  (+0.00%)  ss2 89.96 → 89.96  (+0.00)
  ok         kodim19.png  mozjpeg  base 213006 → 213006  (+0.00%)  ss2 89.92 → 89.92  (+0.00)
  ok         kodim23.png  JPEG XL  base 74113 → 74113  (+0.00%)  ss2 89.59 → 89.59  (+0.00)
  ok         kodim23.png  jpegli  base 97039 → 97039  (+0.00%)  ss2 90.25 → 90.25  (+0.00)
  ok         kodim23.png  AVIF  base 111556 → 111556  (+0.00%)  ss2 89.74 → 89.74  (+0.00)
  ok         kodim23.png  mozjpeg  base 130349 → 130349  (+0.00%)  ss2 89.85 → 89.85  (+0.00)
  ok         kodim23.png  WebP  base 140374 → 140374  (+0.00%)  ss2 89.04 → 89.04  (+0.00)
  ok         kodim23.png  HEIC  base 157442 → 157442  (+0.00%)  ss2 90.22 → 90.22  (+0.00)

OK: no size or quality regressions.

@nucliweb
nucliweb merged commit 5e5ade7 into main Oct 2, 2026
2 checks passed
@nucliweb
nucliweb deleted the feat/harden-regression-gate branch October 2, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant