-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Add Cypress Rules documentation #6215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 12 commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
a45d9e6
update supported providers for the Results API
marktnoonan b487069
wip docs for Cypress rules
marktnoonan e6e44af
Merge branch 'main' into marktnoonan/app-quality-updates
marktnoonan 4ca2f08
add detail, update changelog
marktnoonan b46c8fe
changelog updates
marktnoonan b44267c
improve alt text
marktnoonan 95c8f83
update wording for website crawling
marktnoonan cd38608
tweaks
marktnoonan f7c62a4
lint fix
marktnoonan 3076894
Merge branch 'main' into marktnoonan/app-quality-updates
marktnoonan 72618dc
fix typo in rule id
marktnoonan 14cb12c
Merge branch 'marktnoonan/app-quality-updates' of https://github.com/…
marktnoonan 5adad81
remove temp file
marktnoonan 0d33a0c
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan adede79
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan c8d6bd7
fix a typo
marktnoonan 6fd74b7
update rule text
marktnoonan 00dd572
Merge branch 'main' into marktnoonan/app-quality-updates
marktnoonan ae485fc
add more detail
marktnoonan b09ae57
add more detail
marktnoonan 05e7cbc
lint fix
marktnoonan 2fe3013
add more detail
marktnoonan 993b19e
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan 3b0dba2
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan c0cf9bb
remove unfinished example
marktnoonan 087bc82
changelog
marktnoonan e47a6e7
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan a5000ae
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan a09a9c3
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan 3378f2b
add overall explainer at the top
marktnoonan 55bbf33
Merge branch 'marktnoonan/app-quality-updates' of https://github.com/…
marktnoonan d6c96d0
Update docs/accessibility/core-concepts/cypress-rules.mdx
marktnoonan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,82 @@ | ||
--- | ||
title: 'Cypress Rules | Cypress Accessibility Documentation' | ||
description: 'Review the main areas to pay attention to when first reviewing an accessibility report for a Cypress run.' | ||
sidebar_position: 70 | ||
sidebar_label: Cypress rules | ||
sidebar_custom_props: { 'new_label': true } | ||
--- | ||
|
||
<ProductHeading product="accessibility" /> | ||
|
||
# Cypress rules | ||
|
||
In addition to running the default ruleset of the Axe-Core® library, we also create custom rules that take advantage of the addition layer of information available in a test automation context. These rules are identified by a "Cypress Rule" badge, to distinguish them from the main Axe-Core® rule set. | ||
|
||
The first custom rule that we have implemented is called "Interactive elements must be semantically correct." At launch, it is a non-blocking **manual-review** rule that will either pass if no issues are found, or return an inconclusive state. | ||
|
||
## Interactive elements must be semantically correct | ||
|
||
**Rule ID**: `cy-semantic-html-warning` | ||
|
||
<DocsImage | ||
src="/img/accessibility/core-concepts/cypress-rule.png" | ||
width="60%" | ||
alt="A custom Cypress accessibility rule called 'Interactive elements must be semantically correct' has two failed elements displayed. One element is open and message reads 'This element should use semantic HTML or ARIA to support the interactions that take place during tests. Detected interactions: mousedown, mouseup, click." | ||
/> | ||
|
||
This custom rule is intended to surface potential usability problems with the kinds of elements that Axe-Core® traditionally does not evaluate as interactive, such as div, span, SVG, and image elements. | ||
|
||
This works based on the interactions that take place during testing. Often an inaccessible custom component will have Cypress tests that perform interactions the way a user does, and that allows us to detect a mismatch between the underlying purpose of the element in code, and the usage of the element in the context of a test. | ||
|
||
### How to pass this rule | ||
|
||
You should look at the purpose and nature of the specific elements flagged by this rule and decide what interactive element is appropriate. | ||
|
||
For example: | ||
|
||
- If the element that gets flagged is a span that has been styled to look like a button and submits a form, the correct solution would be standard HTML button element. This communicates the expected action and is activated with the space bar or enter key. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- If the same element is styled with underlined text, and when clicked it resets the URL to navigate to a new page, the correct solution is a link element with a `href` attribute. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
In each case, the solution is to choose the element that provides the best information to the user, so they can independently understand what the action does, and activate it. | ||
|
||
After you have updated to a suitable HTML solution, elements will then be able to be correctly parsed and analyzed by the standard Axe-Core® checks. | ||
|
||
:::info | ||
**Tip**: Follow the [first rule of ARIA use](https://www.w3.org/TR/using-aria/#firstrule) and implement interactive controls with standard semantic HTML where possible, relying on ARIA `role` attributes only when no suitable HTML element is available. | ||
::: | ||
|
||
### Why this matters | ||
|
||
In the Web Content Accessibility Guidelines (WCAG), interactive elements are required to met a wide range of Success Criteria (SCs) in order to follow the POUR principles, which stands for Perceivable, Operable, Understandable, and Robust. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Interactive elements that are not semantically correct often don't appear in automated scans, but do appear in manual testing, because they are usually not operable with a keyboard or correctly understandable when described by a screen reader. | ||
|
||
Custom components implemented this way have a direct impact on the independence of your disabled users. | ||
marktnoonan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Related WCAG Success Criteria | ||
|
||
An incorrectly implemented interactive element may be failing multiple Success Criteria from WCAG, for example: | ||
|
||
- [SC 1.3.1 Info and Relationships](https://www.w3.org/TR/WCAG21/#info-and-relationships) | ||
- [SC 2.1.1 Keyboard](https://www.w3.org/TR/WCAG21/#keyboard) | ||
- [SC 4.1.2 Name, Role, Value](https://www.w3.org/TR/WCAG21/#name-role-value) | ||
|
||
### Status | ||
|
||
This rule is available in all Cypress Accessibility projects, though it is not turned on by default at launch. Reach out to Cypress if you would like to have this rule activated for you early. | ||
|
||
### Possible outcomes | ||
|
||
For each run, this rule can be in one of four states. Most importantly, it will never fail, which means it will not affect your accessibility score. | ||
|
||
| Outcome | Description | | ||
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | ||
| **Pass** | No interactive elements with semantic issues were detected during the test run. | | ||
| **Inconclusive** | Interactive elements with potential semantic issues were detected. You should review each element and determine the correct solution. | | ||
| **Not Applicable** | The test run did not interact with any elements. | | ||
| **Ignored by configuration** | The rule was not executed during this test run because it was intentionally turned off. | | ||
|
||
### False positives | ||
|
||
We expect a certain number of false positives to be detected during the initial rollout. If you see these, please use the "Submit Feedback" button in the product to let us know. Our goal with any new rules is to follow the general Axe-Core® philosophy that minimizes false positives. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -34,6 +34,7 @@ Fetching Accessibility results for a run supports fetching results for the follo | |
- [GitLab](/app/continuous-integration/gitlab-ci) | ||
- [Jenkins](/app/continuous-integration/overview#Jenkins) | ||
- [AWS CodeBuild](/app/continuous-integration/aws-codebuild) | ||
- Drone | ||
|
||
Please reach out to Cypress Support to request support for a different provider. | ||
|
||
|
@@ -59,6 +60,25 @@ If you check this in as a dependency, your installation will fail when we update | |
|
||
Write a script using the `getAccessibilityResults` utility to retrieve the results and perform one or more assertions to verify if the changes are acceptable. This script will be executed in CI. | ||
|
||
#### Basic example | ||
|
||
This snippet uses the `getAccessibilityResults()` helper to log out the results. It assumes your Project ID and Record Key variable are set. The following should work in any of the supported CI Providers out of the box: | ||
|
||
```javascript title="scripts/verifyAccessibilityResults.js" | ||
// Assuming these environment variables are set: | ||
// CYPRESS_PROJECT_ID=your-id | ||
// CYPRESS_RECORD_KEY=your-record-key | ||
|
||
const { getAccessibilityResults } = require('@cypress/extract-cloud-results') | ||
|
||
getAccessibilityResults().then((results) => { | ||
// use `console.dir` instead of `console.log` because the data is nested | ||
console.dir(results, { depth: Infinity }) | ||
}) | ||
``` | ||
|
||
#### How to assert that only known rules are failing in the run | ||
|
||
The Cypress App [repository](https://github.com/cypress-io/cypress) uses the Results API to ensure no new violations have been introduced. You can reference [this script](https://github.com/cypress-io/cypress/blob/develop/scripts/verify-accessibility-results.js) as a real example. | ||
|
||
```javascript title="scripts/verifyAccessibilityResults.js" | ||
|
@@ -133,7 +153,23 @@ getAccessibilityResults({ | |
}) | ||
``` | ||
|
||
#### `getAccessibilityResults` Arguments | ||
#### How to use a previous run as a baseline | ||
|
||
```javascript title="scripts/verifyAccessibilityResults.js" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nitpick] The baseline example duplicates the basic example above verbatim; consider consolidating or differentiating the two to reduce redundancy. Copilot uses AI. Check for mistakes. Positive FeedbackNegative Feedback |
||
// Assuming these environment variables are set: | ||
// CYPRESS_PROJECT_ID=your-id | ||
// CYPRESS_RECORD_KEY=your-record-key | ||
|
||
const { getAccessibilityResults } = require('@cypress/extract-cloud-results') | ||
|
||
getAccessibilityResults().then((results) => { | ||
// use `console.dir` instead of `console.log` | ||
// to ensure nested data prints out correctly | ||
console.dir(results, { depth: Infinity }) | ||
}) | ||
``` | ||
|
||
### `getAccessibilityResults` Arguments | ||
|
||
`getAccessibilityResults` uses the following attributes to identify the Cypress run and return the Accessibility results: | ||
|
||
|
@@ -444,5 +480,33 @@ phases: | |
+ - CYPRESS_INTERNAL_ENV=staging CYPRESS_PROJECT_ID=[slug] CYPRESS_RECORD_KEY=[KEY] node ./scripts/verifyAccessibilityResults.js | ||
``` | ||
|
||
</TabItem> | ||
|
||
<TabItem value="Drone" > | ||
```diff title=".drone.yaml" | ||
kind: pipeline | ||
name: default | ||
|
||
environment: | ||
CYPRESS_PROJECT_ID: example_project_slug | ||
CYPRESS_RECORD_KEY: | ||
from_secret: example_record_key_secret | ||
|
||
steps: | ||
|
||
- name: test | ||
image: node:latest | ||
commands: | ||
- npm install | ||
- npx cypress run --record | ||
|
||
* - name: validate | ||
* image: node:latest | ||
* commands: | ||
* - npm install --force https://cdn.cypress.io/extract-cloud-results/v1/extract-cloud-results.tgz | ||
* - node ./scripts/verifyAccessibilityResults.js | ||
|
||
``` | ||
</TabItem> | ||
</Tabs> | ||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@marktnoonan Did you mean to put the new rules in the changelog?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
adding now, ty!