[Accessibility] No progress indicator is provided for Copilot review

[jooyoungseo]

Type: Bug

CC @meganrogge

When using GitHub Copilot: Review This (Experimental) feature, there is no verbal or non-verbal progress indicator for screen reader users. SR users do not know when the review process has been completed.

VS Code version: Code - Insiders 1.93.0-insider (4849ca9, 2024-09-04T13:13:15.344Z)
OS version: Windows_NT x64 10.0.22631
Modes:

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz (8 x 2611)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	15.71GB (2.37GB free)
Process Argv	--crash-reporter-id b05b88e5-8894-4031-ae34-fa034ebddea9
Screen Reader	yes
VM	0%

Extensions (125)

Extension	Author (truncated)	Version
vscode-openapi	42C	4.28.1
zotenote	A-W	1.0.1
android-dev-ext	ade	1.4.0
aiprm-lang	AIP	0.0.2
Bookmarks	ale	13.5.0
openscad	Ant	1.2.2
spellright	ban	3.0.136
mermaid-markdown-syntax-highlighting	bpr	1.6.0
external-pdf	cha	1.2.0
doxdocgen	csc	1.4.0
vscode-markdownlint	Dav	0.53.0
vscode-eslint	dba	3.0.10
vscode-quick-select	dba	0.2.9
vscode-deno	den	3.39.0
gitlens	eam	14.6.1
EditorConfig	Edi	0.16.4
prettier-vscode	esb	10.1.0
figma-vscode-extension	fig	0.3.5
vscode-firefox-debug	fir	2.9.10
shell-format	fox	7.2.5
vscode-google-translate	fun	1.4.13
codespaces	Git	1.17.2
copilot	Git	1.229.0
copilot-chat	Git	0.21.2024090602
remotehub	Git	0.64.0
vscode-github-actions	git	0.26.2
vscode-pull-request-github	Git	0.97.2024090514
cloudcode	goo	2.16.7
overleaf-workshop	iam	0.13.2
cslpreview	igo	0.2.2
path-autocomplete	ion	1.25.0
latex-workshop	Jam	10.2.1
lilypond-syntax	jea	0.1.1
scheme	jea	0.2.0
better-cpp-syntax	jef	1.17.2
commitlint	jos	2.6.0
language-julia	jul	1.122.1
google-search	kam	0.0.1
vscode-lua-format	Koi	1.3.8
lilypond-formatter	lhl	0.2.3
lilypond-pdf-preview	lhl	0.2.8
lilypond-snippets	lhl	0.1.1
vslilypond	lhl	1.7.3
language-matlab	Mat	1.2.5
git-graph	mhu	1.30.0
azure-dev	ms-	0.8.3
vscode-azureappservice	ms-	0.25.3
vscode-azurecontainerapps	ms-	0.6.1
vscode-azurefunctions	ms-	1.15.3
vscode-azureresourcegroups	ms-	0.8.3
vscode-azurestaticwebapps	ms-	0.12.2
vscode-azurestorage	ms-	0.16.1
vscode-azurevirtualmachines	ms-	0.6.5
vscode-cosmosdb	ms-	0.22.0
vscode-docker	ms-	1.29.2
vscode-edge-devtools	ms-	2.1.5
black-formatter	ms-	2024.3.12071014
debugpy	ms-	2024.11.2024082901
flake8	ms-	2023.13.12291011
isort	ms-	2023.13.12321012
python	ms-	2024.15.2024090406
vscode-pylance	ms-	2024.8.104
jupyter	ms-	2024.8.2024081201
jupyter-keymap	ms-	1.1.2
jupyter-renderers	ms-	1.0.19
vscode-jupyter-cell-tags	ms-	0.1.8
vscode-jupyter-slideshow	ms-	0.1.5
remote-containers	ms-	0.385.0
remote-ssh	ms-	0.115.2024090522
remote-ssh-edit	ms-	0.86.0
remote-wsl	ms-	0.81.8
vscode-remote-extensionpack	ms-	0.25.0
azure-account	ms-	0.12.0
azure-repos	ms-	0.40.0
cmake-tools	ms-	1.19.51
cpptools	ms-	1.22.2
cpptools-extension-pack	ms-	1.3.0
js-debug-nightly	ms-	2024.9.517
remote-explorer	ms-	0.5.2024011009
remote-repositories	ms-	0.42.0
remote-server	ms-	1.6.2024011109
vscode-github-issue-notebooks	ms-	0.0.130
vscode-node-azure-pack	ms-	1.2.0
vscode-selfhost-test-provider	ms-	0.3.25
vscode-serial-monitor	ms-	0.12.0
vscode-speech	ms-	0.10.0
vscode-speech-language-pack-en-ca	ms-	0.4.0
vscode-speech-language-pack-en-gb	ms-	0.4.0
vscode-speech-language-pack-ko-kr	ms-	0.4.0
vsliveshare	ms-	1.0.5936
windows-ai-studio	ms-	0.5.2024090301
autodocstring	njp	0.6.1
pandocciter	not	0.10.4
typst-lsp	nva	0.13.0
publisher	pos	1.1.6
shiny	Pos	1.1.0
shinyuieditor	pos	0.5.0
quarto	qua	1.114.0
r-debugger	RDe	0.5.5
java	red	1.34.0
vscode-xml	red	0.27.1
vscode-yaml	red	1.14.0
r	REd	2.8.4
multi-command	ryu	1.6.0
AudioQ	Seh	0.0.2
vscode-deepl	soe	1.1.1
abc-music	sof	0.4.0
lua	sum	3.10.5
latex-utilities	tec	0.4.14
cmake	twx	0.0.17
vscode-terminal-here	Tyr	0.2.4
windows-terminal	Tyr	0.7.0
errorlens	use	3.16.0
intellicode-api-usage-examples	Vis	0.2.8
vscodeintellicode	Vis	1.2.30
vscode-conventional-commits	viv	1.26.0
vscode-arduino	vsc	0.7.1
vscode-gradle	vsc	3.16.4
vscode-java-debug	vsc	0.58.0
vscode-java-dependency	vsc	0.24.0
vscode-java-pack	vsc	0.29.0
vscode-java-test	vsc	0.42.0
vscode-maven	vsc	0.44.0
markdown-all-in-one	yzh	3.6.2
grammarly	znc	0.25.0

(1 theme extensions excluded)

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
vsaa593:30376534
py29gd2263:31024238
c4g48928:30535728
a9j8j154:30646983
962ge761:30841072
pythongtdpath:30726887
welcomedialog:30812478
pythonnoceb:30776497
asynctok:30898717
dsvsc014:30777825
dsvsc015:30821418
pythonmypyd1:30859725
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30870582
dsvsc016:30879898
dsvsc017:30880771
dsvsc018:30880772
cppperfnew:30980852
pythonait:30973460
01bff139:31013167
a69g1124:31018687
dvdeprecation:31040973
dwnewjupyter:31046869
impr_priority:31057980
nativerepl1:31104042
refactort:31084545
pythonrstrctxt:31093868
flighttreat:31119334
wkspc-onlycs-t:31132770
nativeloc1:31118317
wkspc-ranged-c:31125598
cf971741:31111988
jh802675:31132134
e80f6927:31120813
ei213698:31121563

[alexr00]

@meganrogge I think this is coming from one of the other copilot review entry points. GHPR does show progress from the "Create" view entry point.

[chrmarti]

@meganrogge We show a progress notification while the review is running. Is there anything we need to do in addition to that?

[meganrogge]

Yes, showing something is not helpful for screen reader users. We should play the progress accessibility signal.

[chrmarti]

I guess that should be done by the progress notification. Not sure if the extension could do that by itself.

[bpasero]

@meganrogge progress in the workbench can be reported at different locations, notifications is just one of them. If we need to add audio cues to indicate progress, then I would suggest to add it to the IProgressService:

vscode/src/vs/workbench/services/progress/browser/progressService.ts

Line 32 in c181660

export class ProgressService extends Disposable implements IProgressService {

[bpasero]

I am happy to try to implement this if you give me an idea how this should work.

[meganrogge]

Thanks @bpasero, I think that sounds good. The AccessibilityProgressSignalScheduler is what you would want to use, added here #210582.

We wait 4 seconds before playing a progress signal for chat and then play the signal every 5 seconds subsequently. I am not sure how long the delay typically is for Copilot review, but maybe we stay consistent for this if we're adding it to the progress service?

One thing to note is that when I considered adding this to the progress service, I realized that very few of our operations actually take this long. The other thing to consider in adding it to the progress service is that it's possible multiple progress indicators will play simultaneously, which could be confusing for screen reader users.

[jooyoungseo]

I think Progress audio cue is not enough for blind people to understand what's going on.

I suggest the following from a blind user's perspective:

1. User runs "GitHub Copilot: Review This"

2. Play "Chat Request Sent" audio cue to signal the Copilot related request has been sent

3. Play "Progress" audio cue

4. Play "Chat Response Received" audio cue to signal the job has been completed

5. Move user focus to the comment thread automatically where they can navigate the Copilot review

This way, blind people can clearly understand what action they have to take.

[bpasero]

@meganrogge it actually turns out that there already is an audio signal for progress bars defined here:

vscode/src/vs/base/browser/ui/progressbar/progressbar.ts

Line 209 in 61f6fbf

this.progressSignal.value = getProgressAcccessibilitySignalScheduler(ProgressBar.PROGRESS_SIGNAL_DEFAULT_DELAY);

Since the notification shows a progress bar, it works (I was able to hear a sound after a few seconds):

Having it on the lowlevel progress bar widget is not a bad idea because then it will work everywhere, including the editor (e.g. inline chat).

I believe this issue is about fine tuning the signals specifically for this case, so moving back to you.

[meganrogge]

Thanks for investigating and makes sense