Screen reader support implementation plan

[BenHenning]

This is replacing #185 since it's going to be describing multiple aspects of both how we're planning to approach screen reader integration, and why certain decisions are made (or the framework of thinking for unknowns/open problems).

Roadmap

Tentative roadmap (will be changed over time):

Phase	Description	Estimated completion date	Status	Code affected
Phase 1	Early prototyping for a fundamental experience to help influence direction.	End of July	Done -- 12 Aug	Core + plugin (not checked in)
Phase 2	Collaborative prototyping and testing to for a viable MVP.	End of October (very tentative)	Ongoing	Primarily plugin
Phase 3	Design long-term solution for screen reader support based on phase 2 requirements and needs.	End of September	Not started	N/A
Phase 4	Implementation phase for long-term screen reader support	TBD	Not started	Primarily core Blockly (v13).
...	... continued testing & iteration ...	...	...

Fundamental approach & overview

Fundamentally, our principles for screen reader support are:

• Screen reader is the complement to keyboard shortcuts: users can use keyboard navigation for moving around the page, but a screen reader provides the context for where the user is and what they're currently doing.
  • More specifically: screen readers should be representing navigation rather than augmenting it.
  • This means that we will not follow WCAG guidelines for recommended keyboard shortcuts for standard ARIA flows unless it specifically makes sense. We would rather emphasize general Blockly keyboard navigation to solve such problems since it's essential to keep the mixed mode scenarios working well (that is, using both mouse and keyboard interactions and being able to move between them without changing settings or reloading the page).
• Where possible, we want to leverage ARIA for providing the best possible context for a particular element or part of Blockly.
  • This means possibly finding a best fit for a given Blockly scenario, such as leveraging trees for Blocks since they both represent hierarchies.
  • This may mean sometimes not leveraging a particular ARIA role because it doesn't read out correctly in the Blockly context and lacks customization.
  • This may mean having to work around ARIA limitations due to completely different modalities. For example: moving a block around is close to ARIA's drag-and-drop support, but that is deprecated with no replacement and has limited support across browsers and screen readers.
• We want to aim to provide the exact context a user needs to understand what's going on. This is a balance between sharing enough context, and minimizing output so as to not overwhelm the user.
• We expect that Blockly users will have a range of screen reader familiarity, from very little to extensive. This will influence the amount and nature of context we share with the end user.
• We want to ensure Blockly works well on all top used screen readers, in particular: NVDA, JAWS, Talkback, VoiceOver, and ChromeVox.
• We want to catalog and eliminate impossibilities, and test to confirm approaches or otherwise change them. In the same way as keyboard navigation, the ultimate goal here is to establish strong opinions backed by user testing that can be used as the basis for deciding future changes to screen reader behavior.

Staging

The current plan is to keep the vast majority of screen reader support in the keyboard navigation plugin, even if that involves monkeypatching, and to stage this support behind a programmatic flag. This specifically allows us to:

• Speed up our ability to iterate on screen reader support during 'phase 2' since we don't need to consider how to change core Blockly safely, and only need to worry about 1 release.
• De-risk core Blockly.
• Provide time to figure out how to design the proper functionality and APIs within core Blockly for the long-term (some of which may significantly change based on screen reader output decisions).
• De-risk regular keyboard navigation (since the flag will be off by default, and we'll aim to ensure all screen reader functionality is tied behind this flag where reasonable).

Note that this is considered 'phase 2' in the roadmap above.

Current status, open problems, and open questions

Current screen reader validation statuses: see RaspberryPiFoundation/blockly#9303.

Current experience (link to latest demo page: https://google.github.io/blockly-keyboard-experimentation/screenreader/ -- semi-automatically updated with changes):

• Most core navigation scenarios now have a screen reader readout.

Open questions that need deciding:

• Terminology: how do each of the following get referenced:
  • Inputs (for both fields and shadow blocks)
  • Outputs
  • Mutators
  • Connections (these are probably contextual based on whether a block is being moved/inserted, or if simply navigating to an empty input connection)
• How should we represent blocks and fields?
  • Blocks can sometimes be represented by their field labels when available, but this is a stopgap since there's no formal way to represent blocks in a reader-friendly way. How should blocks be represented?
  • Fields can be difficult to represent: they have both a name and a 'type' (or editor context). The former is indicated with labels and the latter with visual context in the editor.
• How much location context is needed, and when is it best surfaced?
  • The current demo represents blocks as tree items since they share hierarchical principles with ARIA trees. This means positional context is provided at each stage of navigation. It may be worth considering reducing how much information is shared for more familiar users, though this may also be an option in certain screen readers.
  • Unconstrained movement provides the actual x/y coordinates which is likely confusing.
  • There may be a need for a generic "where am I?" button separately from the above as there's additional program context that could be shared.
• Do we need to introduce type-ahead to quickly jump to a particular position? How should that work?

Main issues found (will be filed as official issues later):

• Sometimes readout is redundant (i.e. reading out the same block twice).
• Sometimes move mode doesn't update its status (e.g. when starting a block insert operation).
• Many blocks have poor readout information due to lack of context (this touches both technical plans and open questions).
• Sometimes the tree item counts or positions are incorrect (especially in the toolbox and flyout).
• Native ARIA support for fields isn't complete yet, so the overall experience for field editing is very poor.
• Comments aren't official supported yet.
• Connections don't always provide the correct readout context (or a context at all).
• Deletion isn't announced (and likely many other keyboard shortcuts).
• There are likely versions of shadow blocks that won't work correctly (if they have more than one field since the current solution assumes each shadow block has exactly one field).
• Blocks with no field labels have no description to output.
• Field output can be confusing, and inconsistent between field editors and shadow blocks.

Technical design considerations

Current considerations that are likely to influence the final design of this project include:

• ARIA will need to be enforced for essentially everything that is focusable. However, this is more complicated than just updating IFocusableNode to mandate an ARIA role: in many cases, tab index, the ARIA role, and even other information needs to be established early. Screen readers rely on neighbors and descendant information when formulating their output, so this cannot be lazy loaded.
  • Switching to a node registration pattern may be an alternative worth considering here despite its own drawbacks and limitations.
• Blocks will need to have a mandated and localizable human-readable description, also accounting for inputs (which probably means it needs to be templatized). It's possible to default a block's label based on its field labels, but this is a very poor substitute for a properly written description and outright fails for blocks that have no field labels.
• Fields will be particularly challenging: existing editors can be updated to follow proper ARIA roles (though past prototyping on this has proved challenging), but field editors can vary widely in capabilities and will require field-specific work (consider the color picker as one example). This will require a combination of technical enforcement and guidance.
• It's very likely (as found during prototyping) that a custom live region will need to be maintained to force announcements to the user so that they can get an updated context (especially for dynamic situations like block movement). This will require specific design and scaffolding to reduce unintentionally overwhelming the user as the code evolves over time.
• Automated testing for screen reader support may be especially challenging and is a large unknown here.
• Ensuring that developers understand well how to implement ARIA support correctly will be critical for long-term code maintenance. This is largely related to the guidance for fields, but there's a broader concept of how to pick and utilize certain ARIA roles, and understanding the limitations of them (beyond existing MDN documentation which tends to be very thorough, but lacks the practical behaviors with a real screen reader).

[kmcnaught]

I've had a very brief read and play. I just wanted to note a few small thoughts now, since I'll miss today's discussion.

Generic "where am I? shortcut"

• Seems like a good idea to me
• It's a bit like NVDA shortcuts for "announce current window title" but for blocks
• I am also very keen that we end up with a simple 'read aloud' shortcut for non SR users, e.g. low vision/literacy challenges - it would be good to piggy back on the same markup requirements for blocks. I don't imagine a non-SR shortcut to expose any context, just the current block - so it's probably 2 different things, but enabled by the same core functionality

Customisable verbosity?

• Should we consider a user preference for SR verbosity?
• Is this something that other apps ever do?
• SRs offer verbosity options for e.g. punctuation, but I'm not sure if similar exists for ARIA? If so we should test and document.

Moving unconstrained

• Let's look at how app window movement works usually with SRs
• On Windows you would usually use Alt+Space then 'm' to enter move mode, and use arrows. NVDA intercepts arrows in this scenario, but you can use NumPad arrow keys instead, but then NVDA gives no feedback at all.
• Worth asking a SR user how they would do this! I have definitely observed when I've carried out user testing (moving windows around so that I could see the visuals while they used the SR)

Arrows in toolbox

• With NVDA there's very mixed behaviour in toolbox, since arrow keys are used for previous/next char (left/right) and previous/next line (up/down). Sometimes the standard arrow nav behaviour works, sometimes you get stuck in NVDA behaviour.

[BenHenning]

Thanks @kmcnaught!

• I am also very keen that we end up with a simple 'read aloud' shortcut for non SR users, e.g. low vision/literacy challenges - it would be good to piggy back on the same markup requirements for blocks. I don't imagine a non-SR shortcut to expose any context, just the current block - so it's probably 2 different things, but enabled by the same core functionality

The latter is interesting--I hadn't heard of the need for that requirement before. What context would you envision would be useful for non-SR users? I'm trying to think if it makes sense to actually distinguish them or to consolidate (in the same way we try to keep keyboard navigation the same for SR and non-SR usage).

I'm suspecting that a general "where am I?" read-out is probably going to be fairly generic in describing the program structure (at least with respect to the other context provided by the screen reader), but I'm not certain on that.

Also for what it's worth: the web API for speech synthesis may limit who can use Blockly applications, but it actually has been supported for a while (it's just worth noting that this may need to be considered as part of compatibility).

Customisable verbosity?

• Should we consider a user preference for SR verbosity?
• Is this something that other apps ever do?
• SRs offer verbosity options for e.g. punctuation, but I'm not sure if similar exists for ARIA? If so we should test and document.

I'm also unsure, and I agree with needing testing to understand customization possibilities with screen readers. I actually principally lean toward keeping the extra specificity until testing shows that it's encumbering users since that aligns better with standard practices.

Moving unconstrained

• Let's look at how app window movement works usually with SRs
• On Windows you would usually use Alt+Space then 'm' to enter move mode, and use arrows. NVDA intercepts arrows in this scenario, but you can use NumPad arrow keys instead, but then NVDA gives no feedback at all.
• Worth asking a SR user how they would do this! I have definitely observed when I've carried out user testing (moving windows around so that I could see the visuals while they used the SR)

Great points, and I agree. I hadn't considered window movement, so that's worth digging into. Separately, asking SR users makes a lot of sense here.

Arrows in toolbox

• With NVDA there's very mixed behaviour in toolbox, since arrow keys are used for previous/next char (left/right) and previous/next line (up/down). Sometimes the standard arrow nav behaviour works, sometimes you get stuck in NVDA behaviour.

This is great to know--I haven't tested with NVDA yet. It sounds like NVDA may be doing some custom keyboard support that's interfering with Blockly's (so far as I can tell, ChromeVox doesn't do that).

[microbit-matt-hillsdon]

To make it usable in NVDA I find you need to press NVDA+Space (Insert+Space by default) to switch NVDA from browse to focus mode. Then arrows work and 't' will move to the toolbox instead of saying something about there being no tables. Not sure if this is just necessary advice or whether there's control. It will automatically switch mode for form elements.

In my quick NVDA testing I found blocks don't read out well. I get "graphic" or "group" all the while whereas Mac VoiceOver is similar to Ben's ChromeVox demo.

[microbit-matt-hillsdon]

Actually, I think blocks read out is a regression in what's deployed rather than an NVDA difference. I'll wait before looking further.

[microbit-matt-hillsdon]

Brain dump of thoughts from an initial play with this in MakeCode, primarily with VoiceOver but some NVDA. Ignoring issues due to setAttribute monkey patch:

• Let's label the toolbox. This is (was?) working for the flyout. When you move between them you should know the new region name.
• Would NVDA and Windows Narrator "just work" if we used role=application rather than needing to switch their modes?
  • What functionality would screen reader users miss if we did this?
  • More generally, what screen reader features do folks use that might be lost/not provided with a strict "we read out what's focused" approach.
• I'm worried about assuming we can add roles to fields (e.g. textbox, checkbox) when we don't support the expected interactions. Blockly fields are not directly editable. You have to press enter before you get something that supports entering text.
• For full block fields, the field roles are on something that isn't selected or even a descendent of what is focused and so aren't of any use for the screen reader output.
• For comprehensibility, I think we probably need to read out field values as well as input values in the simple cases (eliding or perhaps summarising non-trivial value block expression trees). But we also need to make it clear that there are fields/inputs. Navigating could then allow you to get more detail. This is one to start discussing early with folks with relevant screen reader experience and to compare other coding apps with good screen reader support. I'll look at some of this soon.
• Similarly getting input on the use of a tree model seems important. I had it say some very weird things to me (e.g. 25 of 8) so needs more work too. I'll try to find scenarios. VoiceOver calls this a "table" and gives a lot of unhelpful output about moving around that is special shortcuts to move the screen reader cursor not the focus which is kinda unhelpful.
• Icons/images are likely to need different semantics in many cases. From my quick review we have:
  • +/- images used to mutate if statement structure read as "image image"
  • icons used for a debug mode breakpoint that likely wants checkbox semantics
• FlyoutButton hardcoded label to "Button". Can we grab the correct text? Can we distinguish the label/button cases?
• VoiceOver says "selected" a lot for blocks etc. but that doesn't add info as so far as I can tell you can't move to unselected blocks. Is it valuable/necessary? Maybe you can move the screen reader separately from focus?
• Curious about the use of "owns" - I want to follow up on these and understand why the DOM structure doesn't achieve what's needed here.
• Should the plugin (blockly?) manage creation of the aria live region rather than the test app? I've had to copy it into MakeCode.
• Positive confirmation of all actions seems necessary. Would be good to enumerate them all. Some have sounds already and sounds might be an option for some other operations.
• Move mode is confusing when it says “child of” as it means different things for statement input and next connection. Visually in Geras these are very different and not the same short of spatial relation.
• Undo/redo moving focus position should probably be fixed as part of this work. Delete too. It makes it a lot harder to reason about what is focused.
• Unconstrained move readout is intense as it'll just keep going for all the steps. It happens during mouse drag too which has many many such steps. Interesting to get feedback on this. Some users might find the spatial layout more confusing than helpful. Compare Quorum.
• NVDA: completing using a dropdown or widget pointlessly reads out "main landmark, Blockly workspace tree view" presumably because you moved out to widget/dropdown div and back into that region on completing the edit. But it would be better to just read the block as widget/dropdown div being separate is irrelevant to the user. Any chance "owns" can fix this? Or we could look at rearranging DOM structure (!)
• I am very tempted to try a simplified scheme that drops the tree model, introduces role=application ariaRoleDescription=block (Quorum style) and a "fluent" block read out that includes nested value blocks, with a summary of field/input counts at the end for the select block only.
• Do we need to navigate into blocks in the flyout to understand them (keys are awkward here)? Or do they need a different/fuller read out? MakeCode has e.g. an if and and if/else, several "play" blocks with different arguments. So we need to read enough for them to be distinct which is everything, I guess, unless you allow navigation in. Leaning towards a fuller readout (you can always move on).

I'll likely add to this comment with more notes as I continue to explore.