Skip to content

[RFC] feat(test-runner-visual-regression): make "update" strict and add "buildCache" for more repeatable tests #1339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 11, 2021

Conversation

Westbrook
Copy link
Member

What I did

  1. Make the update property in the VRT config "strict". It will no only update if set to true, allowing newly added regressions (tests without a baseline) to fail the test pass.
  2. Add the buildCache property to the VRT config. This has the test save all of the current test output in one place so that it is easier to create a new cache from updated tests listings, including when tests have been intentionally removed.

Questions

  1. Using update strictly means that a user with new screenshots can't add those shots while testing if existing tests fail. Is this a common use case? Should we add an update: 'auto' option? Or possibly a strict flag so that the current functionality can be accessed without breaks, but users can alter their workflow to the new version if desired?
  2. buildCache currently leverages the "failed" workflow. In that way should we update the naming conventions throughout to the saveCurrent or similar to clarify the usage of the code throughout?
  3. Placing "current" next to "failed" may make it more difficult for a user to parse the output of their tests. Should these two outputs be treated differently?

@Westbrook Westbrook requested a review from LarsDenBakker March 6, 2021 16:03
@changeset-bot
Copy link

changeset-bot bot commented Mar 6, 2021

🦋 Changeset detected

Latest commit: 805e731

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@web/test-runner-core Patch
@web/test-runner-visual-regression Minor
@web/test-runner Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@Westbrook Westbrook requested a review from daKmoR March 8, 2021 20:25
@LarsDenBakker
Copy link
Member

Looks good to me!

@Westbrook Westbrook merged commit b146365 into master Mar 11, 2021
@Westbrook Westbrook deleted the westbrook/vrt branch March 11, 2021 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants