Skip to content

Replace Percy with nitpix for visual testing - #3873

Open
T4rk1n wants to merge 7 commits into
devfrom
nitpix
Open

T4rk1n wants to merge 7 commits into
devfrom
nitpix

Conversation

@T4rk1n

@T4rk1n T4rk1n commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Visual snapshots are now captured locally by dash.testing (real browser screenshots into NITPIX_SNAPSHOT_DIR, one PNG per width) and diffed in CI by plotly/nitpix: baselines live on the nitpix-baselines orphan branch, PRs get a review comment with before/after/diff images and a nitpix/visual commit status, and changes are approved by commenting "/nitpix approve" — no test-matrix re-run needed.

  • dash/testing/browser.py: percy_snapshot() keeps its name and signature (public API) but saves PNGs locally instead of calling the Percy SDK. convert_canvases is now a no-op: real screenshots capture canvases.
  • testing.yml: drop PERCY_* env, percy-exec wrapper and percy-finalize; snapshot-producing jobs (test-main, html-test, dcc-test, 3.12 leg only) upload nitpix-snapshots-* artifacts instead.
  • nitpix.yml (new): workflow_run diff/report job in base-repo context, so fork and Dependabot PRs are fully supported — the old percy/dash placeholder-status hack in post-test-status.yml is removed.
  • nitpix-approve.yml (new): handles /nitpix approve comments.
  • table-visual-test still uses percy-storybook (separate token/mechanism); converting it is a follow-up.

Visual snapshots are now captured locally by dash.testing (real browser
screenshots into NITPIX_SNAPSHOT_DIR, one PNG per width) and diffed in CI
by plotly/nitpix: baselines live on the nitpix-baselines orphan branch,
PRs get a review comment with before/after/diff images and a
nitpix/visual commit status, and changes are approved by commenting
"/nitpix approve" — no test-matrix re-run needed.

- dash/testing/browser.py: percy_snapshot() keeps its name and signature
  (public API) but saves PNGs locally instead of calling the Percy SDK.
  convert_canvases is now a no-op: real screenshots capture canvases.
- testing.yml: drop PERCY_* env, percy-exec wrapper and percy-finalize;
  snapshot-producing jobs (test-main, html-test, dcc-test, 3.12 leg only)
  upload nitpix-snapshots-* artifacts instead.
- nitpix.yml (new): workflow_run diff/report job in base-repo context, so
  fork and Dependabot PRs are fully supported — the old percy/dash
  placeholder-status hack in post-test-status.yml is removed.
- nitpix-approve.yml (new): handles /nitpix approve comments.
- table-visual-test still uses percy-storybook (separate token/mechanism);
  converting it is a follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

@camdecoster camdecoster self-assigned this Jul 28, 2026
# Conflicts:
#	.github/workflows/testing.yml
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Dash performance benchmarks

✅ all within thresholds

scenario metric p90 (ms) median growth baseline p90 note
✅ callback_chain chain_ms 281.5 278.5 0.95x 499.1
✅ callback_chain graph_ms 1.2 1.2 1.0x 2.4
✅ callback_fanout fanout_ms 62.9 56.1 0.84x 92.5
✅ deep_nesting render_ms 42.6 40.1 1.35x 56.8
✅ full_children_replace replace_ms 3075.9 1144.1 17.04x 4697.1
✅ initial_render_large render_ms 448.6 381.6 1.23x 694.4
✅ initial_render_small render_ms 84.2 65.7 1.39x 104.0
✅ patch_append_nested append_ms 106.0 77.4 2.15x 192.3
✅ patch_append_toplevel append_ms 99.2 60.2 2.76x 140.2
✅ patch_scalar_update_large update_ms 144.7 124.5 0.94x 202.9
✅ wildcard_all_resolve wildcard_ms 175.3 159.8 0.9x 313.6
✅ wildcard_all_resolve graph_ms 0.7 0.7 1.0x 1.3

growth = late-third / early-third per-op time; ~1 is flat, a large value means the per-op cost scales with accumulated state.

machine scale vs baseline: 0.68x - divided out of the baseline ratios so they compare like for like (the absolute warn/fail ceilings are left un-scaled); calibrated on initial_render_small.

@sonarqubecloud

Copy link
Copy Markdown

@camdecoster camdecoster left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add an update to the changelog? Some downstream users might be relying on Dash installing percy.

Do you think a test should be added to verify the snapshot pipeline is working? Right now, if a call to nitpix fails, there's no feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants