-
Notifications
You must be signed in to change notification settings - Fork 884
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide guidelines on validation testing of vega charts #4515
Comments
Right now canvas based charts are validated using visual regresion testing. Unfortunately these tests only exist in the selenium environment and not in the cypress test runner. We have two options here, I'll start with the one i prefer, addng visual regression testing to the functional test repo and adding tests there, the other is to use the existing selenium tests to create these tests. Here is an example of the selenium test that performs a visual regression test on the chart alone: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/test/functional/apps/dashboard/dashboard_snapshots.js |
Since Selenium is on its way out. We need to come up with a cypress solution. Found this but I'm not sure if it's best practice: https://github.com/cypress-visual-regression/cypress-visual-regression Can we consider chucking Vega and using a library with easier testability? For what purpose is this test? |
This isn't really a Vega/Vega-lite problem, it's a limitation of the rendering technique and test frameworks. SVG, Canvas, and WebGL visualization rendering all have their pros and cons, but SVG-based rendering is not inherently better. The bigger question is about the general test strategies we should use for validating visualization rendering. I agree with @ashwin-pc's suggestion to add visual regression testing to the cypress tests as the preferred way forward here. |
I'm currently experimenting with cypress-image-snapshot built off of |
@ashwin-pc / @joshuarrrr please see this FTRepo PR which adds |
Currently vega charts are all rendered using vega view, which when rendered on a page, just produces a single
canvas
html element. This limits being able to validate what is actually being rendered and if certain datapoints are visible as expected or not.vislib
-implemented charts do not have this problem as each datapoint / legend / axis are separate elements and can be validated by a test runner (cypress).A few potential options are:
Ideally we can come up with a final plan and have it documented, then begin adding consistent validation testing on all existing tests that are generating vega-implemented charts.
The text was updated successfully, but these errors were encountered: