How to test widgets in different states?

[Jym77]

When writing the initial color contrast rule, widgets were voluntarily left out since they need to be tested in different states (e.g. a link can be visited or not, an input can have a valid or invalid value, …) and the rule first needed to tackle the important difficulty of the color contrast itself…

I recently tried to write the corresponding rule for widgets (#1455). My idea is that the rule should check a widget in all states it can be through its life on the web page. In order to approximate that, I am using the CSS pseudo-classes (e.g. :visited, :focus, …) and check widgets that match a set of these.

This does leave out various states of "ARIA+JS" widgets that change appearance due to scripting rather than selectors. For example, a <span class="link" role="link" onclick="this.class='visited'"> would only be checked in its default state, not in its visited state since that is not reflected in pseudo-classes (links are maybe not too often in that situation, but other widgets are and links tend to be an easier example to reason about…)

Wilco thinks this is not good enough. I think this is not perfect but is valuable (and will flag real issues, even if not all of them). See further discussion at #1455 (comment)

---

So, we probably need more brainstorming and ideas on how to shape out that rule.

[Jym77]

Trying to sum up questions and situation before the 21 January join meeting.

---

Problem

Some components of a page evolve during the life cycle of the page (depending on its state). For example, a widget can be focused or not, a link can be visited or not, an input field can show its placeholder or not, can be valid or invalid, an image can differ depending on media query CSS selector, … We call these "states" of the component. These states may compose (a link can be both visited, focused and hovered). Aspect of the component may change depending on its state.

There is little doubt that many of the SC must be valid whatever the state. For example, a link text must have good contrast with its background no matter whether it's been visited or not, an image alt text must be descriptive for each of the possible images loaded by the media query, …

It is less clear whether we want a rule to check a component only in one state, or in several states.

---

Solutions

There are essentially 3 ways to approach this as part of rules:

1. A rule checks a page in the state it currently is. So no mention of the state is done in the rule. It is up to the tester (and implementer) to provide different states to the tool if they want to. It is the tester's responsibility to decide which states are relevant for a given page and rule. +: easy to write. -: different testers might end up with different results.
2. A rule checks a page in a given state. The rule only applies for pages that are in the state they specify, e.g. in their Applicability. It is up to the tester to put the page in the wanted state. +: granular results; -: not clear how to specify the states (see below).
3. A rule checks a page globally and try all states when needed (e.g. in its Expectation). +: very similar results among testers; -: can be super verbose, not clear how to specify states (see below)

---

Bonus questions

If the rule needs to somehow talk about states, how can it be handled?

A convenient way, at least for the widgets, is to use the CSS pseudo-selectors. This has three linked problems: first, other interactions with the page (and scripting) might change the appearance; second even if it's classic interaction, it is not certain that CSS is used for styling, it can be overridden by scripting; third, CSS pseudo-selectors are all good for native HTML elements, but won't work for, say, a <span tabindex="0" onclick="history.push(…)"> that manually implements a link and will never match :visited or :link. This may leave many components out of the test coverage.

Static and Dynamic data

This is an underlying problem we have, not specific to this question. But essentially, we can check static data (DOM + CSS) but not really dynamic one (scripts). As soon as we need to figure out if a script is changing the appearance of the page, we run into undecidable problems and we likely want to avoid that in rules… Thus, stuff like "in any state the page can be" is very dangerous water.

---

Toy examples

A rule checking color contrast of links could be, for the 3 solutions above:

1. Applicability: any link; Expectation: good contrast. The rule doesn't care whether the link is visited, focused, … Testers and tools can have a procedure of a/ put the page in a given state; b/ run the test; c/ repeat in a different state.
2. Applicability: any link that is focused, visited, and not hovered; Expectation: good contrast. The rule care about the state and will only check part of the links. Testing the different states of a given link requires either several rule, or a slightly more complex rule (in which case, the targets are link+state, so the same link will create several targets).
3. Applicability: any link. Expectation: good contrast when focused, visited, and not hovered. The rule cares about the state and explicitly goes through the list of states to test (more states could be separate expectation, or part of the same). Each link is a single test target.

Note that links are easy to reason about because it is easy to figure out which states they can be in. It is not so immediate for other components…

---

Real examples

What do we do in current (and in-progress) rules? Non-exhaustive list, mostly from the top of my head…

• Error message describes invalid form field value is more in line with solution 1. It checks the page in whatever state it is in, and passes if it is not a state showing any error message.
  As far as I understand, <style>input:invalid + span::before { content: "Error!"}</style><input type="url"/><span></span> is passing the rule because there is no error indicator (currently showing) (it would fail the SC as soon as invalid text is entered in the input field, but the rule doesn't care).
  But <style>input:invalid + span::before { content: "Error!"}</style><input type="url" value="aaa"/><span></span> (invalid value provided) would be failing because the error indicator doesn't describe the cause of the error.
• Image filename is accessible name for image is more in line with solution 3. It takes the page, figure out that the image can be in different states due to media queries, and explicitly checks all of these.
• Element in sequential focus order has visible focus is a mix of solutions 3 and 1. The full point of the rule is to compare two states, so we need a bit of solution 3 in; but then the rule doesn't care about other states than focus (for example, whether a link is visited or not).
• (in progress) Inline link has distinguishable style not based on color (hue) alone is more in line of solution 3. The first expectation needs to consider hovered and focused states due to quirks in the SC; the second expectation is explicitly telling that both visited and non-visited links must be considered.
• (in progress) Text inside widget has minimum contrast is firmly into solution 2. The Applicability is a (text inside a) widget + a list of CSS pseudo-states. It is this PR that sparkled the current discussion… Note that if we choose solution 1, it can actually be merged with the existing Text has minimum contrast which is currently leaving widgets out because we wanted to focus first on handling contrast and push the "states" discussion to later (and it's now!)
• (in progress) HTML graphics contain no text is more in line with solution 3, explicitly testing each image source (state depending on the media query). This is currently sparkling discussion also there. I believe that "Image Filename" did not trigger discussions on that point only because it was updated before we became conscious of the question…

---

Further reading

• Minutes of ACT-R CG call on the topic
• Bits of related discussion in "color contrast for widget" rule
• Mention of the problem in "image of text" rule

[tbostic32]

This topic is related to some research we conducted where we were trying to look at possible future capabilities for automated accessibility tools to handle states and state transitions explicitly. Ideally, we wanted a web crawler that could automatically find states of the page and then run accessibility tests on the found states.

The research is all open sourced, please feel free to comment or reach out to me directly if you have any questions.

1. Research Overview
2. Roots in Web Archiving and Web Crawling
3. Github Repo of the prototype

I wanted to include this here just to share some lessons learned and give others some starting points if they wanted to investigate some of the academic research of this area.

[Jym77]

@tbostic32 I feel this can now be closed in favor of #1836 and #2046 which go further in the discussion.
Do you think we should keep this one open too (until we find a solution on the other)?