Visual Regression Testing

[joshblack]

About

Currently, the project uses Chromatic which uses the project's storybook to generate snapshots. This setup currently has several limitations or challenges that we're running:

• We are currently hitting the snapshot limit for the free tier
  • The current tier is 5,000 snapshots/month
  • The next tier would be 35,000 snapshots/month at $149
• This setup only captures a story in the default theme
• We would like to capture snapshots at different component states which may not be represented through controls or require specific interactions
• We would also like to run axe against these kinds of scenarios, when applicable
• The review workflow runs outside of GitHub and unfortunately does not work for Pull Requests from forks

Approach

We are looking into using Playwright as an alternative for our Visual Regression Testing setup, along with using it as a way to report on Automated Accessibility Testing from axe. This approach will allow snapshots in the following situations:

• For stories in storybook
• For specific control values in storybook stories
• Across different themes
• Between various interactions or states
• Across breakpoints
• (If needed) Across different browsers

This approach will also include running and reporting axe results, in particular with custom rules that mirror dotcom's.

Checklist

• Install and configure Playwright in the project
• Add initial exploration for test structure and approach to snapshots and running axe
• Add visual regression testing and automated accessibility checking to the ci workflow
• Generate initial coverage for stories in storybook
  • ActionList
  • ActionMenu
  • Autocomplete
  • Avatar
  • AvatarPair
  • AvatarStack
  • Breadcrumbs
  • Button
  • ButtonGroup
  • Checkbox test(e2e): add Checkbox, FilteredActionList, and Token e2e tests #2802
  • CheckboxGroup
  • CounterLabel
  • Dialog
  • FilteredActionList test(e2e): add Checkbox, FilteredActionList, and Token e2e tests #2802
  • FilteredSearch
  • FilterList
  • Flash
  • FormControl
  • Header
  • Heading
  • IconButton
  • Label
  • LabelGroup
  • Link
  • LinkButton
  • NavList
  • Overlay
  • Pagehead
  • PageLayout
  • Pagination
  • PointerBox
  • Popover
  • ProgressBar
  • RelativeTime
  • Radio
  • RadioGroup
  • SegmentedControl
  • Select
  • SelectPanel
  • Spinner
  • SplitPageLayout
  • StateLabel
  • SubNav
  • ToggleSwitch
  • TabNav
  • Textarea
  • Text
  • TextInput
  • TextInputWithTokens
  • Timeline
  • Token test(e2e): add Checkbox, FilteredActionList, and Token e2e tests #2802
  • Tooltip
  • Truncate
  • UnderlineNav
  • Dialog
  • InlineAutocomplete
  • TreeView
  • PageHeader
  • UnderlineNav
• Document the e2e workflow for visual regression testing for contributors
  • Methodology
  • Approach in CI
  • Approval workflow
  • Updating snapshots workflow
  • Running locally
  • Hiding elements
  • Test flakiness, how to reliably interact and test with React 18
  • Document the @vrt and @aat tags for VRT and axe accordingly

What's next

• Add matrix test to components
• Add interaction states, like focus or open

[joshblack]

Stories

Component	Title	Name	Link	VRT	AAT
ActionMenu	Components/ActionMenu/examples	Groups And Description	Link
	Components/ActionMenu/examples	Menu With Actions	Link
	Components/ActionMenu/examples	Mixed Selection	Link
	Components/ActionMenu/examples	Multiple Selection	Link
	Components/ActionMenu/examples	Single Selection	Link
	Components/ActionMenu/examples	Single Selection With Placeholder	Link
	Components/ActionMenu/fixtures	Actions Story	Link
	Components/ActionMenu/fixtures	Controlled Menu	Link
	Components/ActionMenu/fixtures	Custom Anchor	Link
	Components/ActionMenu/fixtures	External Anchor	Link
	Components/ActionMenu/fixtures	Memex Add Column	Link
	Components/ActionMenu/fixtures	Memex Iteration	Link
	Components/ActionMenu/fixtures	Memex Keyboard Shortcut	Link
	Components/ActionMenu/fixtures	Memex Table Menu	Link
	Components/ActionMenu/fixtures	Memex View Options Menu	Link
	Components/ActionMenu/fixtures	Mnemonics Test	Link
	Components/ActionMenu/fixtures	Overlay Props	Link
	Components/ActionMenu/fixtures	Tab Test	Link
	Components/ActionMenu/fixtures	Within Focus Zone	Link
AvatarStack	Components/AvatarStack	AvatarStack	Link
AvatarToken	Components/AvatarToken	Default	Link
	Components/AvatarToken	Interactive	Link
	Components/AvatarToken	With On Remove Fn	Link
ButtonGroup	Components/ButtonGroup	Default	Link
	Components/ButtonGroup	Playground	Link
ConfirmationDialog	Components/ConfirmationDialog	Basic Confirmation Dialog	Link
	Components/ConfirmationDialog	Shorthand Hook	Link
	Components/ConfirmationDialog	Shorthand Hook From Action Menu	Link
Dialog	Components/Dialog	Basic Dialog	Link
	Components/Dialog	Stress Test	Link
	Components/Dialog	With Custom Renderers	Link
FilteredActionList	Components/FilteredActionList	Default	Link
IconButton	Components/IconButton	Playground	Link
	Components/IconButton/Features	Danger	Link
	Components/IconButton/Features	Default	Link
	Components/IconButton/Features	Disabled	Link
	Components/IconButton/Features	Invisible	Link
	Components/IconButton/Features	Large	Link
	Components/IconButton/Features	Medium	Link
	Components/IconButton/Features	Primary	Link
	Components/IconButton/Features	Small	Link
IssueLabelToken	Components/IssueLabelToken	Default	Link
	Components/IssueLabelToken	Interactive	Link
	Components/IssueLabelToken	With On Remove Fn	Link
Label	Components/Label	Label	Link
LinkButton	Components/LinkButton	Playground	Link
	Components/LinkButton/Features	Danger	Link
	Components/LinkButton/Features	Default	Link
	Components/LinkButton/Features	Invisible	Link
	Components/LinkButton/Features	Large	Link
	Components/LinkButton/Features	Leading Visual	Link
	Components/LinkButton/Features	Medium	Link
	Components/LinkButton/Features	Outline	Link
	Components/LinkButton/Features	Primary	Link
	Components/LinkButton/Features	Small	Link
	Components/LinkButton/Features	Trailing Visual	Link
	Components/LinkButton/Features	With React Router	Link
NavList	Components/NavList	Simple	Link
	Components/NavList	With Next JS Link	Link
	Components/NavList	With React Router Link	Link
	Components/NavList	With Sub Items	Link
PageLayout	Components/PageLayout	Custom Sticky Header	Link
	Components/PageLayout	Default	Link
	Components/PageLayout/Interactions	CustomStickyHeader	Link
	Components/PageLayout/Interactions	NonStickyPane	Link
	Components/PageLayout/Interactions	StickyPane	Link
	Components/PageLayout	Nested Scroll Container	Link
	Components/PageLayout	Pull Request Page	Link
	Components/PageLayout	Resizable Pane	Link
	Components/PageLayout	Scroll Container Within Page Layout Pane	Link
	Components/PageLayout	Sticky Pane	Link
RelativeTime	Components/RelativeTime	Count Down Timer	Link
	Components/RelativeTime	Default	Link
	Components/RelativeTime	Micro Format	Link
	Components/RelativeTime	Recent Time	Link
SegmentedControl	Components/SegmentedControl/SegmentedControl.Button	Playground	Link
	Components/SegmentedControl/SegmentedControl.IconButton	Playground	Link
	Components/SegmentedControl	Playground	Link
	Components/SegmentedControl/Features	[Example] Associated with a label and caption	Link
	Components/SegmentedControl/Features	[fullWidth: narrow]	Link
	Components/SegmentedControl/Features	[fullWidth: regular]	Link
	Components/SegmentedControl/Features	[variant: narrow] Action menu	Link
	Components/SegmentedControl/Features	[variant: narrow] Hide labels	Link
	Components/SegmentedControl/Features	Controlled	Link
	Components/SegmentedControl/Features	Default	Link
	Components/SegmentedControl/Features	Full width	Link
	Components/SegmentedControl/Features	Icon only	Link
	Components/SegmentedControl/Features	With Icons	Link
SelectPanel	Components/SelectPanel	Multi Select	Link
	Components/SelectPanel	SelectPanel, Above a Tall Body	Link
	Components/SelectPanel	SelectPanel, Height and Scroll	Link
	Components/SelectPanel	SelectPanel, Height: Initial, Overflowing Items	Link
	Components/SelectPanel	SelectPanel, Height: Initial, Underflowing Items	Link
	Components/SelectPanel	SelectPanel, Height: Initial, Underflowing Items (After Fetch)	Link
	Components/SelectPanel	Single Select	Link
	Components/SelectPanel	With External Anchor	Link
SplitPageLayout	Components/SplitPageLayout	Default	Link
	Components/SplitPageLayout	Settings Page	Link
ToggleSwitch	Components/ToggleSwitch/examples	Controlled	Link
	Components/ToggleSwitch/examples	Default (uncontrolled)	Link
	Components/ToggleSwitch/examples	statusLabelPosition="end"	Link
	Components/ToggleSwitch/fixtures	Associated with a caption	Link
	Components/ToggleSwitch/fixtures	Small	Link
Token	Components/Token	Default	Link
	Components/Token	Interactive	Link
	Components/Token	with leadingVisual	Link
	Components/Token	With On Remove Fn	Link
Tooltip	Components/Tooltip/Default	Icon Button Tooltip	Link
TreeView	Components/TreeView	Default	Link
	Components/TreeView/Features	Async Error	Link
	Components/TreeView/Features	Async Success	Link
	Components/TreeView/Features	Async With Count	Link
	Components/TreeView/Features	Contain Intrinsic Size	Link
	Components/TreeView/Features	Controlled	Link
	Components/TreeView/Features	Empty Directories	Link
	Components/TreeView/Features	Files	Link
	Components/TreeView/Features	Files Changed	Link
	Components/TreeView/Features	Nested Scroll Container	Link
	Components/TreeView/Features	Stress Test	Link
MarkdownEditor	Drafts/Components/MarkdownEditor	Custom Buttons	Link
	Drafts/Components/MarkdownEditor	Default	Link
	Drafts/Components/MarkdownEditor	Lazy Loaded Suggestions	Link
PageHeader	Drafts/Components/PageHeader/Examples	Files Page	Link
	Drafts/Components/PageHeader/Examples	Files Page On Narrow Viewport	Link
	Drafts/Components/PageHeader/Examples	Pull Request Page	Link
	Drafts/Components/PageHeader/Examples	Pull Request Page On Narrow Viewport	Link
	Drafts/Components/PageHeader/Examples	Webhooks	Link
	Drafts/Components/PageHeader/Examples	Webhooks On Narrow Viewport	Link
	Drafts/Components/PageHeader/Features	Has Large Title	Link
	Drafts/Components/PageHeader/Features	Has Title Only	Link
	Drafts/Components/PageHeader/Features	With Actions	Link
	Drafts/Components/PageHeader/Features	With Actions That Have Responsive Content	Link
	Drafts/Components/PageHeader/Features	With Context Bar And Actions Of Context Area	Link
	Drafts/Components/PageHeader/Features	With Description Slot	Link
	Drafts/Components/PageHeader/Features	With Leading And Trailing Actions	Link
	Drafts/Components/PageHeader/Features	With Leading And Trailing Visuals	Link
	Drafts/Components/PageHeader/Features	With Leading Visual Hidden On Regular Viewport	Link
	Drafts/Components/PageHeader/Features	With Navigation Slot	Link
	Drafts/Components/PageHeader/Features	With Parent Link And Actions Of Context Area	Link
	Drafts/Components/PageHeader	Playground	Link
UnderlineNav	Drafts/Components/UnderlineNav/Examples	Profile Page	Link
	Drafts/Components/UnderlineNav/Examples	Pull Request Page	Link
	Drafts/Components/UnderlineNav/Examples	Repos Page	Link
	Drafts/Components/UnderlineNav/Features	Counters Loading State	Link
	Drafts/Components/UnderlineNav/Features	Default	Link
	Drafts/Components/UnderlineNav/Features	Overflow On Narrow Screen	Link
	Drafts/Components/UnderlineNav/Features	Overflow Template	Link
	Drafts/Components/UnderlineNav/Features	With Counter Labels	Link
	Drafts/Components/UnderlineNav/Features	With Icons	Link
	Drafts/Components/UnderlineNav/Interactions	KeepSelectedItemVisible	Link
	Drafts/Components/UnderlineNav/Interactions	KeyboardNavigation	Link
	Drafts/Components/UnderlineNav/Interactions	SelectAMenuItem	Link
	Drafts/Components/UnderlineNav	Playground	Link
	Drafts/Components/UnderlineNav/UnderlineNav.Item	Playground	Link

Code snippet

'use strict'

const data = require('../storybook-static/stories.json')
const components = new Map()

for (const story of Object.values(data.stories)) {
  const segments = story.title.split('/')
  const skiplist = new Set(['Behaviors', 'Hooks', 'Private components', 'Deprecated components', 'Layout components'])

  if (skiplist.has(segments[0])) {
    continue
  }

  if (segments[1] === 'Forms') {
    continue
  }

  if (segments[0] === 'Components') {
    const name = segments[1]
    if (!components.has(name)) {
      components.set(name, [])
    }
    components.get(name).push(story)
    continue
  }

  if (segments[0] === 'Drafts' && segments[1] === 'Components') {
    const name = segments[2]
    if (!components.has(name)) {
      components.set(name, [])
    }
    components.get(name).push(story)
    continue
  }
}

console.log(`| Component | Title | Name | Link | VRT | AAT |
| :--- | :--- | :--- | :--- | :--- | :--- |`)
for (const [name, stories] of components) {
  let didLog = false
  for (const story of stories) {
    console.log(
      `| %s | ${story.title} | ${story.name} | [Link](https://primer.style/react/storybook/iframe.html?args=&id=${story.id}&viewMode=story) | | |`,
      didLog ? '' : name,
    )
    if (didLog === false) {
      didLog = true
    }
  }
}

[joshblack]

Issue Checklist

• Component has test under e2e/components/{Name}.test.ts
• Outermost block is test.describe('{Name}')
  • Each story for that component is in the first level test.describe('{StoryName}')
  • Each story iterates through themes
    • Each theme is a test.describe(theme) block
      • The first test is named default @vrt
        • It visits the storybook story with the given theme
        • It screenshots the page
      • The second test is named axe @aat
        • It visits the storybook story with the given theme

[joshblack]

Renaming storybook stories

Checklist

1. First off, yay! 🥳
2. Make sure to review the Contribution guidelines for primer/react
3. For a quick setup, run npm run setup and then npm start
4. Check if component has an e2e test, this will be under e2e/components

If it does not have an e2e test

• You may choose to add one
• If so, update script/generate-e2e-tests.js with the component and stories that you would like to have tested
• Run node script/generate-e2e-tests.js to generate the test files
• Push to your Pull Request
• Add update snapshots to your Pull Request

If it does have an e2e test

• You may choose choose to either update or remove
• Either way, remove the e2e test file from e2e/components/ComponentName.test.tsx
• If you would like to update the file, update script/generate-e2e-tests.js where the component is defined
• Run node script/generate-e2e-tests.js to generate the test files
• Push to your Pull Request
• Add update snapshots to your Pull Request

[langermank]

Adding a note here for the easiest way to add component stories to the script/generate-e2e-tests.js file:

• After you have written or renamed all the component stories, run npm run build:storybook
• Locate stories.json in /docs/public/storybook
• Format document (for your own sanity)
• Search for the component name
• Copy and paste into script/generate-e2e-tests.js, cleaning it up to match other code (remove the unnecessary parts).
• Continue with the steps above (run the test script or use the vscode task called Generate e2e tests)

[github-actions]

Hi! This issue has been marked as stale because it has been open with no activity for 180 days. You can comment on the issue or remove the stale label to keep it open. If you do nothing, this issue will be closed in 7 days.