Should pg up, pg dn, home, and end cause zoomed in focus to follow the hand?

[KatieWoe]

Test device
Dell
Operating System
Win 10
Browser
Chrome
Problem description
For phetsims/qa#582
When the sim is zoomed in, the screen will follow a hand under keyboard nav if you use the arrow keys to move it. If, however, you use pg up, pg dn, home, or end, the hand is not followed. Is it possible/desired for the zoom focus to follow the hand in these cases?

Visuals

Troubleshooting information:

!!!!! DO NOT EDIT !!!!!
Name: ‪Ratio and Proportion‬
URL: https://phet-dev.colorado.edu/html/ratio-and-proportion/1.0.0-dev.77/phet/ratio-and-proportion_all_phet.html
Version: 1.0.0-dev.77 2020-12-08 00:32:10 UTC
Features missing: applicationcache, applicationcache, touch
Flags: pixelRatioScaling
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Language: en-US
Window: 1280x658
Pixel Ratio: 1.5/1
WebGL: WebGL 1.0 (OpenGL ES 2.0 Chromium)
GLSL: WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)
Vendor: WebKit (WebKit WebGL)
Vertex: attribs: 16 varying: 30 uniform: 4096
Texture: size: 16384 imageUnits: 16 (vertex: 16, combined: 32)
Max viewport: 32767x32767
OES_texture_float: true
Dependencies JSON: {}

[brooklynlash]

The "Both hands" also is not followed while zoomed in using keyboard nav.

[zepumph]

This is a general zoom/sliderPDOM behavior question for @jessegreenberg. Have you thought about or discussed this before?

[zepumph]

Is it possible to callback to animated pan zoom listener when certain input changes the position of the node? Or perhaps, more generally, AnimatedPanZoomListener needs to more robustly track the bounds of the focused Node?

[zepumph]

I think this needs to be handled before RC.

[jessegreenberg]

I am not sure what is best here, and I would like to check in with @terracoda and @emily-phet. I see three ways forward.

1. We make our PanZoomListener always pan to the focused Node so that this happens automatically. In some cases, this could cause too much motion. Requires 1-2 hours of development to add this. Here is an example:

2. We never pan the screen to keep the focused Node visible. I noticed that Windows/Chrome for example, does not do work to keep the focused slider on screen when changing its value, example below. This is a simple policy that would require no additional effort from us.

3. Pan to the focused element, but only in certain cases. Requires design on a case-by-case basis. Need a better way to selectively enable/disable this in code, guessing 3-6 hours of development time to add this.

EDIT: This list is also ordered by my preference but curious what others think.

[zepumph]

I think that PhET has spent a lot more time designing focus highlight than general browsers do, from color, shape, layering etc. Specific focus highlight design given our more targeted audience and learning goals makes sense to me, especially given our generally younger user-base. In general though I think that a simpler approach, (2), where Pan/zoom doesn't try to be fancy and follow the user, will result in a better user experience, because it will feel more explicit. Changing the screen around without the user's expressed input feels a bit like an accessibility concern to me in and of itself. That said I don't feel like I am the expert in this discussion like the designers to you tagged for comment. Good luck!

[terracoda]

We do not yet have a lot of user interview data with the Pan & Zoom feature.
I am wondering if different users might want different things.

Maybe an option to follow focus would be a nice user preference. I am thinking that users who choose to have interactive highlights enabled for all interactions may also prefer to have pan & zoom follow focus.

Also, I see following the skateboarder and and following direct user keyboard interactions l(e.g., page up/down and Home/End) as quite different user interactions. They likely require the same solution, but not necessarily as in the second case, the learner is actively interacting using the keyboard whereas in the second case the learner may or may not be actively observing the skateboarder. If they are not observing the skateboarder, they would need to move their focus off the skateboarder. This seems reasonable to me. Unfortunately, I have no user data.

I will check my notes on GFLB. Pan & Zoom was implemented for the Self-Voicing interviews.

[emily-phet]

My two cents:

• I think option 1 is problematic, for the reason @jessegreenberg indicates. For sims with a lot of animated motion it will be crazy!
• I think option 3 is probably ideal, but even in a "simpler" sim like RaP, I can see that it could be difficult to know when panning should happen and when it shouldn't. For example, the "Both Hands" case. Seems like you want to keep both hands visible as much as possible, but perhaps that's trickier than when a single hand has focus?

@terracoda has the most experience with learners with VI using the sims, so I'll defer to her recommendation, but she sees any value to the automatic panning behavior it sounds like 3 is the way to go.

[terracoda]

I am not as familiar with all the possible PhET objects that are focusable, interactable and that also move on their own. I realize there are many. The Balloons in BASE, Skateboarder(s) in ESP, and orbiting planets come to mind. These interactable animating object will likely need the most discretion.

It is important to keep in mind, following focus doesn't mean following every object that animates.

Again, I am going to check my notes.

[emily-phet]

@terracoda I think @jessegreenberg option 1 needs to be ruled out unless it becomes optional to have focus tracked all the time. I think we don't need to think through all the possible cases to determine that, since If in even one case motion is likely to cause a problem, option 1 is a no-go, unless it's opt-in only. I get super motion sick from animation, and the clip @jessegreenberg showed was already too much for me to watch!
The challenge with options, of course, is that few people will find it, and few people will understand it's use.

If we like the idea of always follow focus, but people have to select an option to enable this, it seems like @jessegreenberg is indicating that might be the simplest solution technically.

As you go through your notes, I think the root of the issue is just 1) is following focus valuable in any case, and if so, 2) should it always happen (which would be the opt-in only situation, not automatically) or should we custom design that feature for each sim?

[terracoda]

I agree, following an animating object would need to be an opt-in thing. As you said, many people would not be able to follow the skateboarder.

[terracoda]

3/4 participants used and commented on the Pan & Zoom feature. All comments were positive.
2/4 were very positive stating it was very natural and worked well.
1/4 said the only issue was accidental fires, e.g., sometimes when pinching to zoom in I would activate the speech and I would have to wait for it to stop talking to continue.

Note that GFLB has no interactive objects that move on their own, unless you count the force arrows.

Currently, in GFLB we pan to a focused object when using the Tab key, and we pan with an object (a slider-like object like the mass sphere) when interacting with the arrow keys and shift+ arrow keys. We don't pan with the object when using larger steps (Page Up/Down keys) or jumping with Home and End keys.

I feel like this might have been an oversight, and a useful one for GFLB. The current behaviour provides interesting (and perhaps helpful) ways/options to observe either the sphere (with regular and small steps) or the changing force arrows (with larger steps and jumps). However, for some it may seem a little inconsistent.

GFLB and RaP currently have the same behaviour for slider interactions.
However, the custom Both Hands interaction, though they center nicely based on the focus highlight, the hands just move off-screen when zoomed in. I feel like this has not been fully thought through.

In addition, for the slider interactions, it doesn't make sense to center the invidual hand as it moves further away from the other hand and the other hand disappears from view. It would make more sense to keep both hands centered even when moving an individual slider. That would allow the hands to potentially both be in view when moving an individual hand closer to the other hand and the other hand comes back into view.

I have 2 thoughts:

1. I think we need to go with option 3
2. I think we need to develop some interview questions or survey questions specifically for the Pan & Zoom feature. There are some rich design questions around this feature that could be addressed as we explore potential user preferences more deeply.

[jessegreenberg]

It is sounding like we need to invest in option 3. @terracoda or @emily-phet can you comment on timeline? Does this need to be done prior to RaP publication?

[zepumph]

For Ratio and Proportion, this is what we would like the behavior to be for soon publication:

• The general following of focused elements during tab navigation is nice.
• When highlighting the sliders, home/end/pg up/pg down should follow the slider. (We want this to be the default behavior in common code for all sliders, with an option to turn if off for cases like GFLB as @terracoda described in Should pg up, pg dn, home, and end cause zoomed in focus to follow the hand? #296 (comment) where we may not want the focus to track). @jessegreenberg let's talk about if you would like some help adding this behavior, and what the timeline could be.
• The zoom-to-center behavior when focused on the both hands Node. This doesn't need to be done for publication, so we created Custom focus behavior for both-hands when zoomed/panned #333.

We may need a more general solution to work custom with animation tracking behavior ( and a preferences option), though that doesn't apply to this sim.

@jessegreenberg, let's go over this during our meeting today.

[jessegreenberg]

The zoom-to-center behavior when focused on the both hands Node

I don't really agree with this design request. It isn't just panning, but would also involve zooming out to keep both hands on screen. In the case of RAP, you have to be almost completely zoomed out to see both hands when they are apart. What if the user wants to zoom in on a hand and change ratio? Doesn't this assume too much about what the user is trying to do?

[zepumph]

I would love to discuss that more in #333, because I think I agree with you. But it also is informed by how we handle solution number (3) above, because this could be a custom implementation that we take based on assumptions about the particular sim (RaP).

I'll copy this over to that issue.

[jessegreenberg]

home/end/pg up/pg down should follow the slider

Does this include the numbered hotkeys as well?

[zepumph]

No, that bullet was only for the common code slider implementation.

[zepumph]

@jessegreenberg and I discussed this, and we will pursue turning the sim's AnimatedPanZoomListener in to a singleton in scenery that any code can opt into specific zoomed-focus behavior. This will cover the second bullet in #296 (comment) as well as carving space for a more robust api to customize more sim-specific focus interactions. See phetsims/scenery#1138.

[jessegreenberg]

This should be done, @zepumph can you please test and make sure that it is working as expected (or assign someone else to review)?

[zepumph]

@terracoda for some co-testing.

[terracoda]

I verified all shortcuts on both sliders.

page up page down, home and end and arrow keys and shift arrow still work, too.

Working means focus follows the hand and the hand stays in view.

[terracoda]

Closing.

[terracoda]

Re-opening to test on other browsers. I only tested Safari on Mac OS.

[zepumph]

I tested on chrome/firefox on windows and all looks great. @terracoda tested on chrome on Mac. Closing