-
Notifications
You must be signed in to change notification settings - Fork 8.5k
Add more info about a11y tests #76045
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
ade0902
add more info about a11y tests
c8e076a
Apply suggestions from code review
219c82f
Adding info on test failures and skipped tests
d5a13d8
Update development-accessibility-tests.asciidoc
f60b996
Apply suggestions from code review
faf965a
wip
e7dc3b8
fix codeblock formatting
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
101 changes: 91 additions & 10 deletions
101
docs/developer/contributing/development-accessibility-tests.asciidoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,104 @@ | ||
| [[development-accessibility-tests]] | ||
| == Automated Accessibility Testing | ||
|
|
||
|
|
||
| To write an accessibility test, use the provided accessibility service `getService('a11y')`. Accessibility tests are fairly straightforward to write as https://github.com/dequelabs/axe-core[axe] does most of the heavy lifting. Navigate to the UI that you need to test, then call `testAppSnapshot();` from the service imported earlier to make sure axe finds no failures. Navigate through every portion of the UI for the best coverage. | ||
|
|
||
| An example test might look like this: | ||
| [source,js] | ||
| ---- | ||
| export default function ({ getService, getPageObjects }) { | ||
| const { common, home } = getPageObjects(['common', 'home']); | ||
| const a11y = getService('a11y'); /* this is the wrapping service around axe */ | ||
|
|
||
| describe('Kibana Home', () => { | ||
| before(async () => { | ||
| await common.navigateToApp('home'); /* navigates to the page we want to test */ | ||
| }); | ||
|
|
||
| it('Kibana Home view', async () => { | ||
| await retry.waitFor( | ||
| 'home page visible', | ||
| async () => await testSubjects.exists('homeApp') | ||
| ); /* confirm you're on the correct page and that it's loaded */ | ||
| await a11y.testAppSnapshot(); /* this expects that there are no failures found by axe */ | ||
| }); | ||
|
|
||
| /** | ||
| * If these tests were added by our QA team, tests that fail that require significant app code | ||
| * changes to be fixed will be skipped with a corresponding issue label with more info | ||
| */ | ||
| // Skipped due to https://github.com/elastic/kibana/issues/99999 | ||
| it.skip('all plugins view page meets a11y requirements', async () => { | ||
| await home.clickAllKibanaPlugins(); | ||
| await a11y.testAppSnapshot(); | ||
| }); | ||
|
|
||
| /** | ||
| * Testing all the versions and different views of of a page is important to get good | ||
| * coverage. Things like empty states, different license levels, different permissions, and | ||
| * loaded data can all significantly change the UI which necessitates their own test. | ||
| */ | ||
| it('Add Kibana sample data page', async () => { | ||
| await common.navigateToUrl('home', '/tutorial_directory/sampleData', { | ||
| useActualUrl: true, | ||
| }); | ||
| await a11y.testAppSnapshot(); | ||
| }); | ||
| }); | ||
| } | ||
| ---- | ||
|
|
||
| === Running tests | ||
| To run the tests locally: | ||
myasonik marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| [arabic] | ||
| . In one terminal window run | ||
| `node scripts/functional_tests_server --config test/accessibility/config.ts` | ||
| . In another terminal window run | ||
| `node scripts/functional_test_runner.js --config test/accessibility/config.ts` | ||
| . In one terminal window run: | ||
| + | ||
| [source,shell] | ||
| ----------- | ||
| node scripts/functional_tests_server --config test/accessibility/config.ts | ||
| ----------- | ||
|
|
||
| . When the server prints that it is ready, in another terminal window run: | ||
| + | ||
| [source,shell] | ||
| ----------- | ||
| node scripts/functional_test_runner.js --config test/accessibility/config.ts | ||
| ----------- | ||
|
|
||
| To run the x-pack tests, swap the config file out for | ||
| `x-pack/test/accessibility/config.ts`. | ||
|
|
||
| After the server is up, you can go to this instance of {kib} at | ||
| `localhost:5620`. | ||
|
|
||
| The testing is done using https://github.com/dequelabs/axe-core[axe]. | ||
| The same thing that runs in CI, can be run locally using their browser | ||
| plugins: | ||
| You can run the same thing that runs CI using browser plugins: | ||
|
|
||
| * https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US[Chrome] | ||
| * https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox] | ||
| * https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox] | ||
|
|
||
| === Anatomy of a failure | ||
|
|
||
| Failures can seem confusing if you've never seen one before. Here is a breakdown of what a failure coming from CI might look like: | ||
| [source,bash] | ||
| ---- | ||
| 1) Dashboard | ||
| create dashboard button: | ||
|
|
||
| Error: a11y report: | ||
|
|
||
| VIOLATION | ||
| [aria-hidden-focus]: Ensures aria-hidden elements do not contain focusable elements | ||
| Help: https://dequeuniversity.com/rules/axe/3.5/aria-hidden-focus?application=axeAPI | ||
| Elements: | ||
| - #example | ||
| at Accessibility.testAxeReport (test/accessibility/services/a11y/a11y.ts:90:15) | ||
| at Accessibility.testAppSnapshot (test/accessibility/services/a11y/a11y.ts:58:18) | ||
| at process._tickCallback (internal/process/next_tick.js:68:7) | ||
| ---- | ||
|
|
||
|
|
||
| * "Dashboard" and "create dashboard button" are the names of the test suite and specific test that failed. | ||
| * Always in brackets, "[aria-hidden-focus]" is the name of the axe rule that failed, followed by a short description. | ||
| * "Help: <url>" links to the axe documentation for that rule, including severity, remediation tips, and good and bad code examples. | ||
| * "Elements:" points to where in the DOM the failure originated (using CSS selector syntax). In this example, the problem came from an element with the ID `example`. If the selector is too complicated to find the source of the problem, use the browser plugins mentioned earlier to locate it. If you have a general idea where the issue is coming from, you can also try adding unique IDs to the page to narrow down the location. | ||
| * The stack trace points to the internals of axe. The stack trace is there in case the test failure is a bug in axe and not in your code, although this is unlikely. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.