[Feature] "Accessibility testing" guidance docs

[dbjorge]

My team (Accessibility Insights) would be interested in contributing an official "Accessibility testing" page to the Guides section of https://playwright.dev/docs, comparable in size/scope/structure to the existing "API testing" guides available for Python and Java.

We wanted to open this issue to build consensus on whether this would be reasonable to include among Playwright's docs and what the desired structure/placement/etc should be.

Motivation

We are hopeful that providing official, first-party guidance will make it more likely that folks might think to use Playwright to incorporate accessibility testing earlier in the development cycle.

We think that these issues are good examples demonstrating community interest for such a guide:

• [Feature] Using Playwright to automate some of the guided tests for accessibility Insights for Web  #13338
• [Question] Does Playwright have support WCAG 2.0 testing? #11621
• integration for the aXe accessibility testing engine [Feature] #4750
• Accessibility Testing[Question] #1638

Using axe-core

We propose that the guidance should be based on using Deque Labs' axe-core, an industry-standard accessibility testing engine used by Accessibility Insights and Lighthouse, among many others.

axe-core itself is a testing engine which runs in the context of (each frame of) the page being scanned. Most folks using axe-core with browser automation frameworks do so via a language- and framework-specific "integration library", which encapsulates the process of using the browser automation framework to inject axe-core into each frame of the page under test, pass through any necessary configuration options, invoke it, and retrieve results back to the test context in a format suitable for writing test assertions and/or producing standalone reports.

Today, we are aware of the following major integration libraries for axe-core and Playwright:

• (Node.js) @axe-core/playwright (src, npm): Official integration library published by Deque
• (Node.js) axe-playwright (src, npm): Unofficial third-party integration library which predates @axe-core/playwright and has slightly more downloads as of writing
• (Java) com.deque.html.axe-core:playwright (src, maven): Official integration library published by Deque

We are not aware of any existing integration libraries for Playwright with .NET or Python.

Proposed documentation

For those languages where officially supported integration libraries already exist (Node.js and Java), we'd like to contribute guides explaining how to use them to do accessibility testing. These would probably follow a similar structure to the existing API testing guide for Python and Java, with an outline along these lines:

• Accessibility testing
  • Writing an accessibility test
    • Configuring Axe
    • Running an accessibility scan
    • Verifying the scan results
    • Complete example
  • Handling known issues
  • Using a test fixture
  • Exporting a report as an attachment

Similar to the API Testing guides, the proposed guides would have very similar content but would probably need a different variation written for each language to account for the use of different integration libraries in each language.

We would probably want to include a disclaimer somewhere (the bottom of the initial overview text, maybe) warning that automated accessibility testing can only catch a limited subset of accessibility issues, and that we recommend using it in combination with regular manual assessments with Accessibility Insights for Web. This is the disclaimer text we typically use.

Implementation

Our understanding is that to implement this we would:

1. Create a PR in this repo adding the guide content as docusaurus-flavored markdown files at:
   • /docs/src/accessibility-testing-java.md
   • /docs/src/accessibility-testing-js.md
2. Create a PR in https://github.com/microsoft/playwright.dev which adds them to the "Guides" section of the docs sidebar by following the example of chore: add selenium-grid to sidebars playwright.dev#463, updating:
   • /java/sidebars.js
   • /java/versioned_sidebars/version-1.XX-sidebars.json for the vnext version of Playwright
   • /nodejs/sidebars.js
   • /nodejs/versioned_sidebars/version-1.XX-sidebars.json

[pavelfeldman]

/cc @michael-siek @Zidious

[pavelfeldman]

This plan looks good to me. I would start with Node / JavaScript, and once we are happy with it, project it to the other languages. That narrows down the focus and works well for us in our core environment.

I looked at @axe-core/playwright and it looks great. There are a couple of things that could be optimized:

• using page.addInitScript to inject axe into all the pages and frames, once and for all, in a way that it'll live through the navigations
• there might be a more efficient way of traversing frames and glueing partial results. // @michael-siek

But those are very much implementation details and should not stand in our way of documenting and promoting it.

[pavelfeldman]

Note that you don't need to change /versioned_sidebars/. You only add it into the current ("next") version and once we cut subsequent release, it'll become versioned.

[dbjorge]

Updates:

• The JavaScript guide is now published for vNext at https://playwright.dev/docs/next/accessibility-testing
• Since this issue was created, we've become aware of a new 3rd party integration library for Playwright .NET + Axe (src, nuget)

[yury-s]

We are not aware of any existing integration libraries for Playwright with .NET or Python.

• Since this issue was created, we've become aware of a new 3rd party integration library for Playwright .NET + Axe (src, nuget)

Do I understand correctly that you are also planning to provide similar documentation for .NET but not for Python (since there is no axe python bindings) or the plan is still to have the docs only for js and Java for now?