Both hands interaction sonification inconsistent during interview

[Matthew-Moore240]

We also noticed that during the Both Hands interaction that a lot of the sounds were very inconsistent. Could you investigate @jbphet again we are not sure if its a bug or a weird recording hiccup. Here is another clip of the issue. Inconsistent sound

[terracoda]

@jbphet, this "interruption" of the sounds also happens for me, so I am not sure if it might be a VO-specific issue. The interviewee was using VO and I also use VO.

[jbphet]

@Matthew-Moore240 and @terracoda - To help me in duplicating this behavior, please let me which platforms (OS, OS version, and hardware manufacturer) you were using when you saw and heard the problems.

[terracoda]

I also posted the system info in issue #229 VoiceOver & Safarii on macOS 10.13.6, on a 2015 macBook Pro, and for the interview the participant they were also using VoicOver & Safari, but on macOS 10.14 and on a macAir. I don't know how new.

[zepumph]

I don't think this is a sound bug. This is in the way the staccato sounds were implemented. They only play when Property says that the both hands is being interacted with. I feel like me logic for that doesn't correctly account for how focus/interaction may change when the screen reader is on. I'll take a closer look.

[emily-phet]

Unassigning @jbphet for now.

[zepumph]

This is what I want to test on a mac, but I don't have one to test. Perhaps I can pair with @jessegreenberg on this to see what is happening here. On windows with or without NVDA, I always have had great success with the sound starting on first both-hands key press, and not stopping until tab navigating away.

Index: js/common/view/BothHandsInteractionListener.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- js/common/view/BothHandsInteractionListener.js	(revision 9bb7762be7b3dcc630df0f13165a9d3d1e657ad7)
+++ js/common/view/BothHandsInteractionListener.js	(date 1606184725412)
@@ -85,7 +85,8 @@
   /**
    * @public
    */
-  blur() {
+  blur( event ) {
+    console.log( 'blur', event );
     this.isBeingInteractedWithProperty.value = false;
   }
 
@@ -103,6 +104,8 @@
       // their behavior during scenery event dispatch
       sceneryEvent.pointer.reserveForKeyboardDrag();
 
+      console.log( 'keydown', sceneryEvent );
+
       this.isBeingInteractedWithProperty.value = true;
 
       if ( event.key === 'ArrowDown' ) {

[zepumph]

@jessegreenberg and I discovered that we could reproduce this bug in safari in https://phet-dev.colorado.edu/html/ratio-and-proportion/1.0.0-dev.71/phet/, but not on master. I'm not sure what has changed, and if it is a tambo thing or a RaP thing.

[zepumph]

I looked around commits in RaP, scenery, and tambo, and nothing stood out as the cause of this fix. It would be nice to know, but I don't know how else to investigate without taking up more time of someone with a mac.

@Matthew-Moore240 can you please confirm that this is fixed for you on master/mac/safari and feel free to close.

[terracoda]

@zepumph, considering your comment in #230 (comment), shall we leave this issue for QA?

I use a mac all the time and I have not experienced this issue.

[zepumph]

Sounds good. I'll just leave open for QA. Great idea!

[zepumph]

QA, when you get here, please take a look at the behavior in the video in the original comment, and then try to reproduce with the "Both hands interaction" (when the keyboard focus highlight is around both hands). I could only see the bug in safari.

[KatieWoe]

I did note that the sound stops when keyboard nav focus leaves. Even when you come back to it. Is this alright @zepumph?
https://drive.google.com/file/d/151Zc6cyhgar9JyNq_w6DInmNuuDEARYf/view?usp=sharing

[zepumph]

Yes, it should not begin again until you focus AND interact with it (move it) again.

[KatieWoe]

I don't think we saw this. We will keep an eye out during future tests.

[zepumph]

I think enough time has been spent on it! Closing. Thanks for all your help.