Add Visual Regression testing to e2e test suite for Nav Unification #48444
Open
Description
As the Nav Unification feature will be dependent on upstream to Menus in WordPress.org it would be prudent to have some automated tests to validate that layout/visuals don't change between WordPress.org releases.
To do this we should implement visual regression tests in the e2e test suite in Calypso.
A npm package exists for this
http://v4.webdriver.io/guide/services/visual-regression.html
The idea should be:
- Only capture specific areas of the page - do not try visual diffing the entire page.
- Remove all transient elements such as notifications, popups and count badges.
- Snapshot the sidebar and store as baseline.
- Each tests should capture the area and then compare to baseline. If changes then fail tests.
- Human should review failure and mark as "pass/fail". "Pass" means this becomes the new baseline.
- Ideally we should also do the same comparison on a .org site so we can be notified if/when the sidebar meny on .org changes between releases.
cc @bsessions85 and @cathymcbride