Screen reader integration technicalities

[kmcnaught]

This is a thread to discuss what is planned / pictured for screen reader integration. We are starting to look at fixing keyboard accessibility of various MakeCode divs, and want to be conscious of making design decisions that don't break anything for screen reader access, even though we imagine there will be a later pass to really nail screen reader access across the board.

Currently screen readers are in browse mode in the blockly workspace, which means their shortcuts override our keyboard navigation (this is not a surprise because we're not expecting screen reader support yet). But we'd like to understand more what direction you are planning to go in, so we can be consistent with it.

Are we planning for the blocklyDiv (or some child component of it) to have role="application" to support us in overriding screen reader defaults for right arrow? This would also protect us against issues with single-key shortcuts. But it means there's lots of behaviour you'd usually get 'for free' from a screenreader browsing a well-labelled DOM, that you'll then need to replicate by hand. I imagine if we went this way, we'd leave blocklyWidgetDiv, blocklyDropDownDiv and blocklyTooltipDiv as native elements and expect normal screen reader functionality there, is that right?

Or is there a plan for some kind of shadow-DOM that a screen reader navigates?

Or do we expect a screen reader user to be able to use standard browse paradigms within the blockly workspace? If so, how does this play nicely with the current keyboard nav?

Or something else entirely? If the direction isn't known yet, do we know what options are under consideration?

I don't expect final answers yet, but would be great to understand the general expected direction.

@BenHenning because I think you might be looking at this kind of thing...

Thanks!

[BenHenning]

Thanks for starting this discussion @kmcnaught!

My current experimentation with using Chromevox with a basic version of roving tab index (not quite https://opensource.adobe.com/spectrum-web-components/tools/roving-tab-index/ but close) has mostly "just worked" (with a lot of caveats). At least with Chromevox, I haven't run into issues with screenreader shortcuts overriding keyboard navigation because. Is this something that you've observed? If so, it would be helpful to better understand what you tried so that I can look into it.

For the long-term, I see screen reader support coming from completing two areas of work:

1. Introducing first-class tracking of Blockly node focus (and bidirectionally syncing this with DOM focus). This provides two critical benefits: first, it simplifies keyboard navigation since it allows us to decentralize node highlighting on a per-node basis (plus some other benefits) and second, it (per roving tab index experimentation) provides the fundamental basis for screen reader support (since focus + tab index enables screen readers to read out an element).
2. Improving labeling (or, rather, getting it to a reasonable MVP state) which includes establishing correct ARIA roles and properties. This is where dropdown div especially needs to be reconsidered as, currently, ARIA doesn't seem to provide the correct context to the screen reader with the dropdown div living outside of the main Blockly div (or it's an incompatibility with SVG--we aren't sure yet). It's possible tooltips and the widget div will face other problems, as well. @gonfunko is looking into bringing those three divs into a per-workspace context rather than living outside the main Blockly div.

(1) is currently undergoing design, and (2) will go through some a design phase afterwards. I'm expecting that we will file a number of tasks for both areas of work that, together, should provide an MVP experience for screen reader output.

I do have two WIP branches (one in the plugin and the other in core Blockly) which, together, demonstrate very basic screen reader support. I can provide instructions for how to use those locally if you're interested.

Edit: I also want to add one thing: while the mechanisms for making screen reader support work with keyboard navigation seem straightforward (as does making the nodes read out the correct context), our understanding of this is in a much more basic place than the broader keyboard navigation. In practice, I expect that this means a path to even an MVP state will likely encounter a lot of unexpected problems. Post-MVP, I expect a lot of iteration on the screen reader experience (especially as it comes to readout context and less commonly used Blockly features that may not be covered initially).

[kmcnaught]

Hi @BenHenning, thanks for the details, that's really helpful.

On Windows (I've been testing with NVDA but I believe JAWS is similar) there's a clear distinction between two modes the screen reader operates in.

Browse mode - lots of single-key shortcuts are used for navigation, e.g. "h" to go to the next heading, up/down arrow to go to the next item/line (depending on context) and left/right arrow to read each character out from a line of text
Forms mode - single characters have their usual behaviour, e.g. for typing this sentence.

A lot of the time, if you are using standard UI elements, NVDA will automatically switch you into forms mode and back again and it "just works" without users knowing about the mode-switching. It's also possible to manually switch modes with a shortcut key.

Indeed, if you navigate the current demo you can mode-switch to forms mode once you are in the workspace, and then you get the keyboard navigation working correctly. If you don't do this step, then down arrow take you to the next item in a left-to-right-top-to-bottom direction, and right arrow reads out one character at a time.

It slightly worries me that an emerging screen-reader user (young person) would need to understand and be consciously in control of mode switching, but I'm not aware of any good workarounds (other than making role=application and then having to do an awful lot of work to reimplement the feedback you get from standard web semantics). Would be good to get an outside perspective on whether this is fine / to be expected / something to worry about.

[kmcnaught]

Also worth linking to WCAG - character key shortcuts if we're heading towards shortcuts like "t for toolbox" or "i for insert"

[rachel-fenichel]

I'm tagging @joreneng here to continue discussion based on feedback in #639

In particular, I'd like to explain our decisions about these two issues:

• Shortcut conflicts with screen readers (eg. M and X for JAWS): Screen reader integration technicalities #185
• Alt/Ctrl+Up/Down allows unconstrained movement (Alt conflicts with many other keyboard shortcuts and screen readers): feat: introduce Mover actions + experimental DragMover implementation #352

I discussed the details of browse vs application mode with several accessibility experts and concluded that we need to use application mode for Blockly because of the level of complexity and our desire to have single-key shortcuts.

This does mean that we will have conflicts with JAWS and NVDA shortcuts, and that users need to intentionally enter and exit the Blockly workspace. This is also why we chose to use tab to move between larger regions, but arrow keys inside Blockly.

In particular, Blockly's components map most closely to an ARIA treegrid, but I spoke to the implementer of the treegrid functionality and he agreed that it is not possible to directly map blocks to a treegrid.

We don't remap keys to specifically avoid JAWS or NVDA shortcuts right now because there is no way to check which screen reader is in use (or, in fact, whether a screen reader is in use) from the browser. However, it is possible to remap shortcuts to use different keys, so we could consider separate mappings for different screenreaders and have the user choose their mapping. If that's the case, we still need a good default mapping because not every keyboard user will use a screenreader.