Making links independently focusable by Talkback

[fabOnReact]

new branch https://github.com/fabriziobertoglio1987/react-native/tree/independent-links-rebased
old branch https://github.com/fabriziobertoglio1987/react-native/tree/independent-links
facebook/react-native@b352e2d

Right now nested Text components are not accessible on Android. This is because we only create a native ReactTextView for the parent component; the styling and touch handling for the child component are handled using spans. In order for TalkBack to announce the link, we need to linkify the text using a ClickableSpan.
This diff adds ReactClickableSpan, which TextLayoutManager uses to linkify a span of text when its corresponding React component has accessibilityRole="link". For example:

  <Text>
    A paragraph with some
    <Text accessible={true} accessibilityRole="link" onPress={onPress} onClick={onClick}>links</Text>
    surrounded by other text.
  </Text>

With this diff, the child Text component will be announced by TalkBack ('links available') and exposed as an option in the context menu. Clicking on the link in the context menu fires the Text component's onClick, which we're explicitly forwarding to onPress in Text.js (for now - ideally this would probably use a separate event, but that would involve wiring it up in the renderer as well).

facebook/react-native#31757

Summary:
This follows up on D23553222 (facebook/react-native@b352e2d), which made links functional by using Talkback's Links menu. We don't often use this as the sole access point for links due to it being more difficult for users to navigate to, and easy for users to miss if they don't listen to the entire description, including the hint text that announces that links are available.

The PR allows TalkBack to move the focus directly on the nested Text link (accessibilityRole="link").
The nested link becomes the next focusable element after the that contains it.
If there are multiple links within a single text element, they will each be focusable in order after the main element.

Related facebook/react-native#30375 https://developer.android.com/reference/android/text/style/ClickableSpan https://developer.android.com/reference/androidx/core/view/ViewCompat#enableAccessibleClickableSpanSupport(android.view.View) https://stackoverflow.com/a/62222068/7295772 facebook/react-native#32004

[fabOnReact]

The nested link becomes the next focusable element after the parent element that contains it.

Talkback moves focus on a <Text> element that has several nested <Text accessibilityRole=“link”> elements

Related facebook/react-native#31757 facebook/react-native@b352e2d

Expected Result:

• Multiple nested links are focusable after the parent.
• Talkback moves the focus to the next element. The focus moves to the next nested link.

CLICK TO OPEN SOURCECODE

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/packages/rn-tester/js/examples/Accessibility/AccessibilityAndroidExample.android.js#L177-L215

CLICK TO OPEN VIDEO TESTS - pr branch

linksOnFabriAndBranch.mp4

CLICK TO OPEN VIDEO TESTS - main branch

linksOnMainDontWork.mp4

[fabOnReact]

User Interacts with links through TalkBack default accessibility menu

• Talkback moves focus on a <Text> element that has a nested <Text accessibilityRole=“link”> element
• The user opens Talkback default accessibility menu

Functionality built with commit facebook/react-native@b352e2d

Expected Result:
The user can interact with the nested link through the TalkBack accessibility menu.

CLICK TO OPEN SOURCECODE

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/packages/rn-tester/js/examples/Accessibility/AccessibilityAndroidExample.android.js#L177-L215

CLICK TO OPEN VIDEO TESTS - pr branch

linksOnFabriAndBranch.mp4

CLICK TO OPEN VIDEO TESTS - main branch - link accessible through menu

2022-02-28.16-14-51.mp4

[fabOnReact]

• make changes to TextAnchorViewManager component and run test
• review diff commit facebook/react-native@b352e2d
• further review Android API used by commit facebook/react-native@b352e2d
• review diff pr https://github.com/facebook/react-native/pull/31757/files
• update this issue summary with more info
• open issue on Github
• verify that you are using fabric

[fabOnReact]

ClickableSpan vs AccessibileClickableSpan

facebook/react-native#30375 (comment)

git diff main..independent-links **/ReactBaseTextShadowNode.java

https://developer.android.com/reference/android/text/style/ClickableSpan
https://stackoverflow.com/a/62332707/7295772

public void onAccessibilityEvent(AccessibilityEvent) {
    AccessibilityNodeInfo nodeInfo = event.getSource();
    if (nodeInfo != null) {
        List<AccessibilityNodeInfo> nodeInfoList = nodeInfo.findAccessibilityNodeInfosByText(
          "This app wants to access your location all the time, even when you're not using the app. Allow in settings.");
        if (nodeInfoList != null && !nodeInfoList.isEmpty()) {
            ClickableSpan[] spans = getClickableSpans(nodeInfoList.get(0).getText());
            if (spans.length > 0) {
                spans[0].onClick(null);
            }
        }
    }
}

getClickableSpan

@NonNull
ClickableSpan[] getClickableSpans(CharSequence text) {
    try {
      if (text instanceof Spanned) {
        Spanned spanned = (Spanned) text;
        return spanned.getSpans(0, text.length(), ClickableSpan.class);
      }
    } catch (Exception e) {
    //log exception
    }
    return new ClickableSpan[0];
}

A hidden class was added called AccessibilityClickableSpan that implements ClickableSpan and you can see that its implementation of onClick does indeed ignore the View parameter, as it would seemingly have to in this case. So while I still don't see any documentation of this, it seems like an implementation detail that can't reasonably change in the future. A ClickableSpan from an AccessibilityNodeInfo should continue to support passing null.

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/text/style/AccessibilityClickableSpan.java#L35-L45

/**
 * Perform the click from an accessibility service. Will not work unless
 * setAccessibilityNodeInfo is called with a properly initialized node.
 *
 * @param unused This argument is required by the superclass but is unused. The real view will
 * be determined by the AccessibilityNodeInfo.
 */
@Override
public void onClick(View unused) {
    Bundle arguments = new Bundle();
    arguments.putParcelable(ACTION_ARGUMENT_ACCESSIBLE_CLICKABLE_SPAN, this);

    if ((mWindowId == UNDEFINED_WINDOW_ID) || (mSourceNodeId == UNDEFINED_NODE_ID)
            || (mConnectionId == UNDEFINED_CONNECTION_ID)) {
        throw new RuntimeException(
                "ClickableSpan for accessibility service not properly initialized");
    }

    AccessibilityInteractionClient client = AccessibilityInteractionClient.getInstance();
    client.performAccessibilityAction(mConnectionId, mWindowId, mSourceNodeId,
            R.id.accessibilityActionClickOnClickableSpan, arguments);
}

[fabOnReact]

enableAccessibleClickableSpanSupport

https://stackoverflow.com/a/62222068/7295772
https://android.googlesource.com/platform/frameworks/support/+/f7528a970ad7f2f099b4dc1397084cc388c9193c/core/src/main/java/androidx/core/view/ViewCompat.java#1316

/**
 * Allow accessibility services to find and activate clickable spans in the application.
 *
 * @param view The view
 * <p>
 * Compatibility:
 * <ul>
 *     <li>API &lt; 19: No-op
 * </ul>
 */
public static void enableAccessibleClickableSpanSupport(View view) {
    if (Build.VERSION.SDK_INT >= 19) {
        getOrCreateAccessibilityDelegateCompat(view);
    }
}

https://github.com/androidx/androidx/blob/150112ca4e1e670fb4eb6756993f3a25df0a5da9/core/core/src/main/java/androidx/core/view/ViewCompat.java#L931-L939

static AccessibilityDelegateCompat getOrCreateAccessibilityDelegateCompat(
        @NonNull View v) {
    AccessibilityDelegateCompat delegateCompat = getAccessibilityDelegate(v);
    if (delegateCompat == null) {
        delegateCompat = new AccessibilityDelegateCompat();
    }
    setAccessibilityDelegate(v, delegateCompat);
    return delegateCompat;
}

[fabOnReact]

• Check diff g d main..independent-links. The functionality works in branch independent-links.
• further investigate Making links independently focusable by Talkback  #9 (comment)

[fabOnReact]

AccessibilityClickableSpan

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/text/style/AccessibilityClickableSpan.java#L36

/**
 * {@link ClickableSpan} cannot be parceled, but accessibility services need to be able to cause
 * their callback handlers to be called. This class serves as a parcelable placeholder for the
 * real spans.
 *
 * This span is also passed back to an app's process when an accessibility service tries to click
 * it. It contains enough information to track down the original clickable span so it can be
 * called.
 *
 * @hide
 */
public class AccessibilityClickableSpan extends ClickableSpan

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/os/Parcel.java#L61

/**
 * Container for a message (data and object references) that can
 * be sent through an IBinder.  A Parcel can contain both flattened data
 * that will be unflattened on the other side of the IPC (using the various
 * methods here for writing specific types, or the general
 * {@link Parcelable} interface), and references to live {@link IBinder}
 * objects that will result in the other side receiving a proxy IBinder
 * connected with the original IBinder in the Parcel. 
  * <h3>Parcelables</h3>
 *
 * <p>The {@link Parcelable} protocol provides an extremely efficient (but
 * low-level) protocol for objects to write and read themselves from Parcels.
 * You can use the direct methods {@link #writeParcelable(Parcelable, int)}
 * and {@link #readParcelable(ClassLoader)} or
 * {@link #writeParcelableArray} and
 * {@link #readParcelableArray(ClassLoader)} to write or read.

https://en.wikipedia.org/wiki/Inter-process_communication

In computer science, inter-process communication or interprocess communication (IPC) refers specifically to the mechanisms an operating system provides to allow the processes to manage shared data. Typically, applications can use IPC, categorized as clients and servers, where the client requests data and the server responds to client requests.[1] Many applications are both clients and servers, as commonly seen in distributed computing.

[fabOnReact]

getVirtualViewAt

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L483-L520

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L32-L48

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L653-L662

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/SimpleMonthView.java#L1050-L1071

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/RadialTimePickerView.java#L1149-L1181

getVirtualViewAt.mp4

onPopulateBoundsForVirtualView

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L534-L549

onPopulateBoundsForVirtualView.mp4

[fabOnReact]

Done

• change values in getVirtualViewAt and verify changes in TalkBack
• Try to implement AccessibilityClickableSpan on Android App and test it with TalkBack
• Implement logic in a separate AccessibilityDelegate attached to TextView
• debug this methods and see what is not working in main branch
• import AccessibilityRole from ReactAccessibilityDelegate
• try to delete logic not added with g d ..blavalla/export-D28691177 **/ReactAccessibilityDelegate.java from ReactTextAccessibilityDelegate
• comment accessibilityAction logic and test if still works in both Text and nested Text
• comment accessibilityState logic => does not announce disabled but is disabled
• check diff with master, comment logic and verify is required for the functionality
• Remove logic for ReactTextAccessibilityDelegate
• check accessibilityAction both on Text, View, Pressable on main and branch
• change the length of the AccessibilityClickableSpan and test TalkBack focus
• verify differences between approach in independent-links and getClickableSpan
• call setFocusable
• Text actions don't work
• Text does not announce disabled
• further test accessibility functionalities in Text, Nested Text, View, Button, TouchableOpacity
• scrolling down with TalkBack sometimes trigger no next link
• ReactAccessibilityDelegate extends ExploreByTouch and ReactTextDelegate extends ReactAccessibilityDelegate
• prepare questions on what you should improve

Low Priority

• improve functionality to span text over one line
• Nested text still does not announce disabled, so you may look into their respective ViewManager and Views
• Move shared logic to a new class
• Try to use default constructor instead of the one with 3 params
• Method setAccessibilityDelegate should be moved to common parent
• Follow same pattern used with ReactSliderAccessibilityDelegate
• review diff between independent-links and main
• save class instance in constructor mSharedClassInstance
• use shared logic mSharedClassInstance.sharedMethod() (composition)
• Use multiple inheritance in ReactTextAccessibility Delegate (extends ExploreByTouchHelper and ReactAccessibilityDelegate)
• Share interface (in ruby a module) between ReactAccessibilityDelegate and ReactTextAccessibilityDelegate

Deleted

• change constructor to take 1 parameter

[fabOnReact]

The TalkBack focus (green rectangle) does not span on multiple lines

CLICK TO OPEN SOURCECODE

<Text
  accessibilityRole="link"
  onPress={() => {
    alert('pressed long link');
  }}>
  link that spans multiple lines because the text is so long.
</Text>

CLICK TO OPEN VIDEO TESTS

linkThatSpansMultipleLines.mp4

[fabOnReact]

isVirtual

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactBaseTextShadowNode.java#L499-L505

https://github.com/fabriziobertoglio1987/react-native-notes/blob/de66d8064361246ddb7ce0b97d4d2d8cfcfcc172/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactShadowNode.java#L24-L54

[fabOnReact]

CustomClickableSpan is clickable without talkback

Related #9 (comment)

CLICK TO OPEN SOURCECODE

MainActivity.java

public class MainActivity extends Activity {
  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    LinearLayout linearLayout =  (LinearLayout) findViewById(R.id.linearlayout);
    TextView textView = new TextView(this);
    textView.setMovementMethod(LinkMovementMethod.getInstance());
    SpannableStringBuilder spannable = new SpannableStringBuilder("Text is spantastic!");
    spannable.setSpan(
        new CustomClickableSpan(Color.RED),
        2, // start
        12, // end
        Spannable.SPAN_EXCLUSIVE_INCLUSIVE
    );
    textView.setText(spannable);
    linearLayout.addView(textView);
  }
}

CustomClickabeSpan.java

public class CustomClickableSpan extends ClickableSpan {
  private int color;
  public CustomClickableSpan(int spanColor) {
    super();
    color = spanColor;
  }
  @Override
  public void onClick(@NonNull View widget) {
    Log.w("TESTING::", "onClick called on view: " + widget);
  }

  @Override
  public void updateDrawState(@NonNull TextPaint ds) {
    super.updateDrawState(ds);
    ds.setColor(color);
  }
}

CLICK TO OPEN VIDEO TESTS

clickableSpan.mp4

[fabOnReact]

Nested Links do not announce disabled or other accessibility properties

Expected Result:

• Announces disabled.
• is disabled

Actual Result:

• Does not announce disabled.
• is disabled

CLICK TO OPEN SOURCECODE

        <Text accessible={true} accessibilityState={{disabled: true}}>
          This is a{' '}
          <Text
            style={{color: 'green'}}
            accessibilityState={{disabled: true}}
            accessibilityRole="link"
            onPress={() => {
              alert('pressed test');
            }}>
            test
          </Text>{' '}
          of{' '}
          <Text
            style={{color: 'green'}}
            accessibilityRole="link"
            onPress={() => {
              alert('pressed Inline Links');
            }}>
            Inline Links
          </Text>
          in React Native. Here's{' '}
          <Text
            accessibilityRole="link"
            onPress={() => {
              alert('pressed another link');
            }}>
            another link
          </Text>
          . Here is a{' '}
          <Text
            accessibilityRole="link"
            accessibilityState={{disabled: true}}
            onPress={() => {
              alert('pressed long link');
            }}>
            link that spans multiple lines because the text is so long.
          </Text>
          I wonder how this works?
        </Text>
        <Text>Normal Text</Text>
        <Button
          accessibilityRole="button"
          title="Testing"
          accessibilityState={{disabled: true}}
        />
      </>

CLICK TO OPEN NOTES

Here the accessibility links are created

https://github.com/fabriziobertoglio1987/react-native-notes/blob/052efec4b4c94dd5cf7a61ef6fc6043a5129884e/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactTextViewManager.java#L77-L82

https://github.com/fabriziobertoglio1987/react-native-notes/blob/052efec4b4c94dd5cf7a61ef6fc6043a5129884e/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactTextAccessibilityDelegate.java#L469-L501

The accessibility node information are set here

https://github.com/fabriziobertoglio1987/react-native-notes/blob/052efec4b4c94dd5cf7a61ef6fc6043a5129884e/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactTextAccessibilityDelegate.java#L366-L372

CLICK TO OPEN VIDEO TESTS

2022-02-21.17-47-24.mp4