Edit Context: Screen Reader Users Feedback

[aiday-mar]

Pinging @jooyoungseo, @rperez030 and @meganrogge

Recently we have been working on adopting the EditContext API (https://developer.mozilla.org/en-US/docs/Web/API/EditContext_API) within VS Code. The EditContext is a new DOM property that can be set on DOM elements which decouples text input from the textual mutations of the DOM element. Essentially when the user focuses a DOM element on which an EditContext is set and types, the EditContext fires 'textupdate' events, and it is up to the user to mutate the DOM element with the changes from this event.

There are several reasons why we have adopted this API:

• This API has allowed us to greatly simplify the code which handles text input events.
• This API has allowed us to close numerous IME related bugs
• This API can allow us to emit customized typing information from the input events

We have an experimental setting which enables the EditContext API with ID editor.experimentalEditContextEnabled. We would like to ask @jooyoungseo and @rperez030 if when you have time you could try the setting with a screen reader and let us know if you see any issues on typing in the various inputs of VS Code (could be the editor, the panel chat input, the SCM view, the quick input, ie any input that accepts text insertions). We would like to gather feedback from screen reader users before enabling this setting by default.

[rperez030]

Thanks for the heads-up. I'll give it a try and report back here. From: Aiday Marlen Kyzy ***@***.***> Sent: Thursday, September 19, 2024 6:55 AM To: microsoft/vscode ***@***.***> Cc: Mention ***@***.***> Subject: [microsoft/vscode] Edit Context: Screen Reader Users Feedback (Issue #229051) Pinging @jooyoungseo<https://github.com/jooyoungseo>, @rperez030<https://github.com/rperez030> and @meganrogge<https://github.com/meganrogge> Recently we have been working on adopting the EditContext API (https://developer.mozilla.org/en-US/docs/Web/API/EditContext_API) within VS Code. The EditContext is a new DOM property that can be set on DOM elements which decouples text input from the textual mutations of the DOM element. Essentially when the user focuses a DOM element on which an EditContext is set and types, the EditContext fires 'textupdate' events, and it is up to the user to mutate the DOM element with the changes from this event. There are several reasons why we have adopted this API: * This API has allowed us to greatly simplify the code which handles text input events. * This API has allowed us to close numerous IME related bugs * This API can allow us to emit customized typing information from the input events We have an experimental setting which enables the EditContext API with ID editor.experimentalEditContextEnabled. We would like to ask @jooyoungseo<https://github.com/jooyoungseo> and @rperez030<https://github.com/rperez030> if when you have time you could try the setting with a screen reader and let us know if you see any issues on typing in the various inputs of VS Code (could be the editor, the panel chat input, the SCM view, the quick input, ie any input that accepts text insertions). We would like to gather feedback from screen reader users before enabling this setting by default. - Reply to this email directly, view it on GitHub<#229051> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJH5KGXL4TYU3I5ZCMMZUR3ZXKUOXBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLKJTGE3DQMJTGM3DJJDOMFWWLKLIMFZV63DBMJSWZAVFOZQWY5LFVI3TIMRYGM2DKOJVGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44UQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBUGE4DQMJZGAYIFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDKMZVHE4TGMZQGWBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGMYTMOBRGMZTMNECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UNZUGI4DGNBVHE2TJJ3UOJUWOZ3FOKTGG4TFMF2GK>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jooyoungseo]

NVDA does not seem to support this new editor mode; word-by-word navigation with JAWS is not reliable. Sorry for the brevity. This is my initial and quick testing results.

[rperez030]

Hi team,

After enabling the setting, both NVDA and Narrator no longer recognize the editor as a text editor and cannot track the cursor properly. While NVDA performs slightly better, it is still not usable.

Based on my observations, this may be due to Chromium not returning the correct roles and states through the IAccessible API.

Here are the API returns with the setting enabled and disabled:

With the Setting Enabled:

• IAccessible:

  • accName: 'This is a test. • Untitled-1'
  • accRole: ROLE_SYSTEM_GROUPING
  • accState: STATE_SYSTEM_FOCUSED, STATE_SYSTEM_FOCUSABLE, STATE_SYSTEM_VALID (1048580)
  • accDescription: None
  • accValue: 'This is a test.\n\nThis is a line.\n\nThis is a line.'

• IAccessible2:

  • windowHandle: 527658
  • uniqueID: -2569
  • role: IA2_ROLE_SECTION
  • states: IA2_STATE_EDITABLE, IA2_STATE_OPAQUE, IA2_STATE_SELECTABLE_TEXT, IA2_STATE_SINGLE_LINE (13320)

With the Setting Disabled:

• IAccessible:

  • accName: 'This is a test. • Untitled-1'
  • accRole: ROLE_SYSTEM_TEXT
  • accState: STATE_SYSTEM_FOCUSED, STATE_SYSTEM_FOCUSABLE, STATE_SYSTEM_VALID (1048580)
  • accDescription: None
  • accValue: 'This is a test.\n\nThis is a line.\n\nThis is a line.'

• IAccessible2:

  • windowHandle: 527658
  • uniqueID: -12173
  • role: ROLE_SYSTEM_TEXT
  • states: IA2_STATE_EDITABLE, IA2_STATE_MULTI_LINE, IA2_STATE_OPAQUE, IA2_STATE_SELECTABLE_TEXT, IA2_STATE_SUPPORTS_AUTOCOMPLETION (38408)

I think this will require some work in Chromium to be properly supported.

[aiday-mar]

Hi @rperez030 and @jooyoungseo thank you both for your testing. Apologies I should have tested myself first with NVDA to catch this simple bug. I will investigate this issue further.

I was wondering where you got this accessibility information from, is it from the aViewer tool which allows to inspect accessibility information? In case you do use the aViewer tool, is it also very slow for you?

[aiday-mar]

Hi @rperez030 I am using NVDA with the setting editor.experimentalEditContextEnabled. I put the cursor in the editor and am using the arrow keys to read the lines one by one, this seems to work. Reading letter by letter seems to work too.

I tested with Narrator and noticed that the focus box does not follow the editor cursor. I therefore transferred some of the aria labels from the current editor implementation to the edit context, notably: role, aria-required, aria-multiline, aria-autocomplete, aria-roledescription. With these attributes the focus box follows the cursor. The PR is here: https://github.com/microsoft/vscode/pull/229361/files.

I am wondering if perhaps accessibility is fixed or partially improved in NVDA with the PR above?

When you say NVDA/Narrator no longer recognize the editor as a text editor and that it can not track the cursor properly, could you perhaps provide steps on how to reproduce this issue? Thank you.

[rperez030]

Open VS code, and write the following in a new document. This is a test. This is a line. This is another line. Use the arrow keys to navigate, line by line and character by character. Observe how NVDA reports the empty lines between the text lines. Now, enable editor.experimentalEditContextEnabled, and repeat the test. At least in my environment, NVDA no longer tracks the cursor, and the entire document appears as a single line in the Braille display. From: Aiday Marlen Kyzy ***@***.***> Sent: Monday, September 23, 2024 6:26 AM To: microsoft/vscode ***@***.***> Cc: Mention ***@***.***>; Comment ***@***.***> Subject: Re: [microsoft/vscode] Edit Context: Screen Reader Users Feedback (Issue #229051) Hi @rperez030<https://github.com/rperez030> I am using NVDA with the setting editor.experimentalEditContextEnabled. I put the cursor in the editor and am using the arrow keys to read the lines one by one, this seems to work. Reading letter by letter seems to work too. Could I ask you what you mean exactly by NVDA no longer recognizes the editor as a text editor and that it can not track the cursor properly, and potentially provide steps to reproduce this issue? - Reply to this email directly, view it on GitHub<#229051 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJH5KGQUF7SA7C36LEWBL2TZX7UD5BFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEZTCNRYGEZTGNRUURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UNZUGI4DGNBVHE2TJJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOKIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2DCOBYGE4TAMECUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRVGM2TSOJTGMYDLAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTGMJWHAYTGMZWGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKG42DEOBTGQ2TSNJUU52HE2LHM5SXFJTDOJSWC5DF>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[aiday-mar]

Hi @rperez030 thank you for the clarification. Today I am testing the Edit Context with NVDA with the recent aria attributes changes.

After setting the setting to true, reloading the window, and after some (I assume notification) text is read, I place the cursor at the top of the editor and hear the following as I move the cursor down:

This is a test, Blank, This is a line, Blank, This is another line

This seems to correspond to what is read by NVDA without the setting.

The text that is read however does not appear in the NVDA speech viewer, nor do I see it in the Braille Viewer. I tested with the setting turned off, and the text does not appear there either. I am not sure if I am missing some config to see the text that would be sent to a braille device.

Please let me know if I am missing or misunderstanding something.

[rperez030]

In theory, everything that gets sent to the speech subsystem should also be available in the speech viewer. The same applies to braille. does the speech viewer even work for you outside VS Code? Happy to get on a call to troubleshoot if that helps speed things up.

my schedule is quite busy today, but we could connect tomorrow or friday.