Making links independently focusable by Talkback (#33215)

Summary:
This issue fixes [32004][23]. The Pull Request was previously published by [blavalla][10] with [31757][24].
>This is a follow-up on [D23553222 (https://github.com/facebook/react-native/commit/b352e2da8137452f66717cf1cecb2e72abd727d7)][18], which made links functional by using [Talkback's Links menu][1]. We don't often use this as the sole access point for links due to it being more difficult for users to navigate to and easy for users to miss if they don't listen to the full description, including the hint text that announces that links are available.
The Implementation of the functionality consists of:

Retrieving the accessibility links and triggering the TalkBack Focus over the Text
1. nested Text components with accessibilityRole link are saved as [ReactClickableSpan][17] instances in Android native [TextView][20] ([more info][19])
1. If the TextView contains any [ClickableSpans][15] (which are [nested Text][14] components with role link), set a view tag and reset the accessibility delegate.
3. Obtain each link description, start, end, and position relative to the parent Text (id) from the Span as an [AccessibilityLink][16]
4. Use the [AccessibilityLink][16]  to display TalkBack focus over the link with the `getVirtualViewAt` method (more [info][13])

Implementing ExploreByTouchHelper to detect touches over links and to display TalkBack rectangle around them.
1. ReactAccessibilityDelegate inherits from [ExploreByTouchHelper][12]
2. If the [ReactTextView][21] has an accessibility delegate, trigger ExploreByTouchHelper method [dispatchHoverEvent][22]
3.  Implements the methods `getVirtualViewAt` and `onPopulateBoundsForVirtualView`.
     The two methods implements the following functionalities  (more [info][13]):
    * detecting the TalkBack onPress/focus on nested Text with accessibilityRole="link"
    * displaying TalkBack rectangle around nested Text with accessibilityRole="link"

## Changelog

[Android] [Added] - Make links independently focusable by Talkback

Pull Request resolved: #33215

Test Plan:
[1]. User Interacts with links through TalkBack default accessibility menu ([link][1])
[2]. The nested link becomes the next focusable element after the parent element that contains it. ([link][2])
[3]. Testing accessibility examples in pr branch ([link][3])
[4]. Testing accessibility android examples in pr branch ([link][4])
[7]. TalkBack focus moves through links in the correct order from top to bottom (PR Branch with [link.id][25]) ([link to video test][7]) ([discussion][26])
[8]. TalkBack focus does not move through links in the correct order from top to bottom (PR Branch without [link.id][25]) ([link to video test][8]) ([discussion][26])

Test on main branch
[5]. Testing accessibility examples in main branch ([link][5])
[6]. Testing accessibility android examples in main branch ([link][6])

[1]: fabOnReact/react-native-notes#9 (comment)
[2]: fabOnReact/react-native-notes#9 (comment)
[3]: fabOnReact/react-native-notes#9 (comment)
[4]: fabOnReact/react-native-notes#9 (comment)
[5]: fabOnReact/react-native-notes#9 (comment)
[6]: fabOnReact/react-native-notes#9 (comment)
[7]: fabOnReact/react-native-notes#9 (comment)
[8]: fabOnReact/react-native-notes#9 (comment)

[10]: https://github.com/blavalla "blavalla github profile"
[12]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L48 "com/android/internal/widget/ExploreByTouchHelper.java#L48"
[13]: fabOnReact/react-native-notes#9 (comment) "explanation of getVirtualViewAt and onPopulateBoundsForVirtualView"
[14]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/text/Spannable.java#L3 "core/java/android/text/Spannable.java#L3"
[15]: https://github.com/fabriziobertoglio1987/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactTextViewManager.java#L70-L71 "react/views/text/ReactTextViewManager.java#L70-L71"
[16]: https://github.com/fabriziobertoglio1987/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L680-L685 "react/uimanager/ReactAccessibilityDelegate.java#L680-L685"
[17]: https://github.com/facebook/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/views/text/TextLayoutManager.java#L126-L129 "react/views/text/TextLayoutManager.java#L126-L129"
[18]: b352e2d
[19]: #30375 (comment) "explanation on how nested Text are converted to Android Spans"
[20]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/TextView.java#L214-L220 "core/java/android/widget/TextView.java#L214-L220"
[21]: https://github.com/facebook/react-native/blob/485cf6118b0ab0b59e078b96701b69ae64c4dfb7/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactTextView.java#L577 "dispatchHoverEvent in ReactTextView"
[22]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L120-L138 "dispatchHoverEvent in ExploreByTouchHelper"
[23]: #32004
[24]: #31757
[25]: https://github.com/fabriziobertoglio1987/react-native/blob/485cf6118b0ab0b59e078b96701b69ae64c4dfb7/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L648 "setting link.id in the AccessibilityLink constructor"
[26]: https://github.com/facebook/react-native/pull/33215/files/485cf6118b0ab0b59e078b96701b69ae64c4dfb7#r820014411 "comment on role of link.id"

Reviewed By: blavalla

Differential Revision: D34687371

Pulled By: philIip

fbshipit-source-id: 8e63c70e9318ad8d27317bd68497705e595dea0f

Libraries/Text/Text.js

@@ -129,6 +129,7 @@ const Text: React.AbstractComponent<
                 onResponderTerminate(event);
               }
             },
+            onClick: eventHandlers.onClick,
             onResponderTerminationRequest:
               eventHandlers.onResponderTerminationRequest,
             onStartShouldSetResponder: eventHandlers.onStartShouldSetResponder,

Libraries/Text/TextNativeComponent.js

@@ -14,11 +14,13 @@ import {type HostComponent} from '../Renderer/shims/ReactNativeTypes';
 import createReactNativeComponentClass from '../Renderer/shims/createReactNativeComponentClass';
 import {type ProcessedColorValue} from '../StyleSheet/processColor';
 import {type TextProps} from './TextProps';
+import {type PressEvent} from '../Types/CoreEventTypes';
 
 type NativeTextProps = $ReadOnly<{
   ...TextProps,
   isHighlighted?: ?boolean,
   selectionColor?: ?ProcessedColorValue,
+  onClick?: ?(event: PressEvent) => mixed,
   // This is only needed for platforms that optimize text hit testing, e.g.,
   // react-native-windows. It can be used to only hit test virtual text spans
   // that have pressable events attached to them.

ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java

@@ -433,7 +433,8 @@ private static void resetTransformProperty(@NonNull View view) {
   }
 
   private void updateViewAccessibility(@NonNull T view) {
-    ReactAccessibilityDelegate.setDelegate(view);
+    ReactAccessibilityDelegate.setDelegate(
+        view, view.isFocusable(), view.getImportantForAccessibility());
   }
 
   @Override

ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

@@ -8,19 +8,26 @@
 package com.facebook.react.uimanager;
 
 import android.content.Context;
+import android.graphics.Paint;
+import android.graphics.Rect;
 import android.os.Bundle;
 import android.os.Handler;
 import android.os.Message;
-import android.text.SpannableString;
-import android.text.style.URLSpan;
+import android.text.Layout;
+import android.text.Spannable;
+import android.text.Spanned;
+import android.text.style.AbsoluteSizeSpan;
+import android.text.style.ClickableSpan;
 import android.view.View;
 import android.view.accessibility.AccessibilityEvent;
+import android.widget.TextView;
+import androidx.annotation.NonNull;
 import androidx.annotation.Nullable;
-import androidx.core.view.AccessibilityDelegateCompat;
 import androidx.core.view.ViewCompat;
 import androidx.core.view.accessibility.AccessibilityNodeInfoCompat;
 import androidx.core.view.accessibility.AccessibilityNodeInfoCompat.AccessibilityActionCompat;
 import androidx.core.view.accessibility.AccessibilityNodeInfoCompat.RangeInfoCompat;
+import androidx.customview.widget.ExploreByTouchHelper;
 import com.facebook.react.R;
 import com.facebook.react.bridge.Arguments;
 import com.facebook.react.bridge.Dynamic;
@@ -36,13 +43,15 @@
 import com.facebook.react.uimanager.events.Event;
 import com.facebook.react.uimanager.events.EventDispatcher;
 import com.facebook.react.uimanager.util.ReactFindViewUtil;
+import java.util.ArrayList;
 import java.util.HashMap;
+import java.util.List;
 
 /**
  * Utility class that handles the addition of a "role" for accessibility to either a View or
  * AccessibilityNodeInfo.
  */
-public class ReactAccessibilityDelegate extends AccessibilityDelegateCompat {
+public class ReactAccessibilityDelegate extends ExploreByTouchHelper {
 
   private static final String TAG = "ReactAccessibilityDelegate";
   public static final String TOP_ACCESSIBILITY_ACTION_EVENT = "topAccessibilityAction";
@@ -59,6 +68,9 @@ public class ReactAccessibilityDelegate extends AccessibilityDelegateCompat {
     sActionIdMap.put("decrement", AccessibilityActionCompat.ACTION_SCROLL_BACKWARD.getId());
   }
 
+  private final View mView;
+  private final AccessibilityLinks mAccessibilityLinks;
+
   private Handler mHandler;
 
   /**
@@ -179,8 +191,10 @@ public static AccessibilityRole fromValue(@Nullable String value) {
   private static final String STATE_SELECTED = "selected";
   private static final String STATE_CHECKED = "checked";
 
-  public ReactAccessibilityDelegate() {
-    super();
+  public ReactAccessibilityDelegate(
+      final View view, boolean originalFocus, int originalImportantForAccessibility) {
+    super(view);
+    mView = view;
     mAccessibilityActionsMap = new HashMap<Integer, String>();
     mHandler =
         new Handler() {
@@ -190,6 +204,14 @@ public void handleMessage(Message msg) {
             host.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_SELECTED);
           }
         };
+
+    // We need to reset these two properties, as ExploreByTouchHelper sets focusable to "true" and
+    // importantForAccessibility to "Yes" (if it is Auto). If we don't reset these it would force
+    // every element that has this delegate attached to be focusable, and not allow for
+    // announcement coalescing.
+    mView.setFocusable(originalFocus);
+    ViewCompat.setImportantForAccessibility(mView, originalImportantForAccessibility);
+    mAccessibilityLinks = (AccessibilityLinks) mView.getTag(R.id.accessibility_links);
   }
 
   @Nullable View mAccessibilityLabelledBy;
@@ -388,18 +410,6 @@ public static void setRole(
     nodeInfo.setClassName(AccessibilityRole.getValue(role));
     if (role.equals(AccessibilityRole.LINK)) {
       nodeInfo.setRoleDescription(context.getString(R.string.link_description));
-
-      if (nodeInfo.getContentDescription() != null) {
-        SpannableString spannable = new SpannableString(nodeInfo.getContentDescription());
-        spannable.setSpan(new URLSpan(""), 0, spannable.length(), 0);
-        nodeInfo.setContentDescription(spannable);
-      }
-
-      if (nodeInfo.getText() != null) {
-        SpannableString spannable = new SpannableString(nodeInfo.getText());
-        spannable.setSpan(new URLSpan(""), 0, spannable.length(), 0);
-        nodeInfo.setText(spannable);
-      }
     } else if (role.equals(AccessibilityRole.IMAGE)) {
       nodeInfo.setRoleDescription(context.getString(R.string.image_description));
     } else if (role.equals(AccessibilityRole.IMAGEBUTTON)) {
@@ -445,16 +455,233 @@ public static void setRole(
     }
   }
 
-  public static void setDelegate(final View view) {
+  public static void setDelegate(
+      final View view, boolean originalFocus, int originalImportantForAccessibility) {
     // if a view already has an accessibility delegate, replacing it could cause
     // problems,
     // so leave it alone.
     if (!ViewCompat.hasAccessibilityDelegate(view)
         && (view.getTag(R.id.accessibility_role) != null
             || view.getTag(R.id.accessibility_state) != null
             || view.getTag(R.id.accessibility_actions) != null
-            || view.getTag(R.id.react_test_id) != null)) {
-      ViewCompat.setAccessibilityDelegate(view, new ReactAccessibilityDelegate());
+            || view.getTag(R.id.react_test_id) != null
+            || view.getTag(R.id.accessibility_links) != null)) {
+      ViewCompat.setAccessibilityDelegate(
+          view,
+          new ReactAccessibilityDelegate(view, originalFocus, originalImportantForAccessibility));
+    }
+  }
+
+  // Explicitly re-set the delegate, even if one has already been set.
+  public static void resetDelegate(
+      final View view, boolean originalFocus, int originalImportantForAccessibility) {
+    ViewCompat.setAccessibilityDelegate(
+        view,
+        new ReactAccessibilityDelegate(view, originalFocus, originalImportantForAccessibility));
+  }
+
+  @Override
+  protected int getVirtualViewAt(float x, float y) {
+    if (mAccessibilityLinks == null
+        || mAccessibilityLinks.size() == 0
+        || !(mView instanceof TextView)) {
+      return INVALID_ID;
+    }
+
+    TextView textView = (TextView) mView;
+    if (!(textView.getText() instanceof Spanned)) {
+      return INVALID_ID;
+    }
+
+    Layout layout = textView.getLayout();
+    if (layout == null) {
+      return INVALID_ID;
+    }
+
+    x -= textView.getTotalPaddingLeft();
+    y -= textView.getTotalPaddingTop();
+    x += textView.getScrollX();
+    y += textView.getScrollY();
+
+    int line = layout.getLineForVertical((int) y);
+    int charOffset = layout.getOffsetForHorizontal(line, x);
+
+    ClickableSpan clickableSpan = getFirstSpan(charOffset, charOffset, ClickableSpan.class);
+    if (clickableSpan == null) {
+      return INVALID_ID;
+    }
+
+    Spanned spanned = (Spanned) textView.getText();
+    int start = spanned.getSpanStart(clickableSpan);
+    int end = spanned.getSpanEnd(clickableSpan);
+
+    final AccessibilityLinks.AccessibleLink link = mAccessibilityLinks.getLinkBySpanPos(start, end);
+    return link != null ? link.id : INVALID_ID;
+  }
+
+  @Override
+  protected void getVisibleVirtualViews(List<Integer> virtualViewIds) {
+    if (mAccessibilityLinks == null) {
+      return;
+    }
+
+    for (int i = 0; i < mAccessibilityLinks.size(); i++) {
+      virtualViewIds.add(i);
+    }
+  }
+
+  @Override
+  protected void onPopulateNodeForVirtualView(
+      int virtualViewId, @NonNull AccessibilityNodeInfoCompat node) {
+    // If we get an invalid virtualViewId for some reason (which is known to happen in API 19 and
+    // below), return an "empty" node to prevent from crashing. This will never be presented to
+    // the user, as Talkback filters out nodes with no content to announce.
+    if (mAccessibilityLinks == null) {
+      node.setContentDescription("");
+      node.setBoundsInParent(new Rect(0, 0, 1, 1));
+      return;
+    }
+
+    final AccessibilityLinks.AccessibleLink accessibleTextSpan =
+        mAccessibilityLinks.getLinkById(virtualViewId);
+    if (accessibleTextSpan == null) {
+      node.setContentDescription("");
+      node.setBoundsInParent(new Rect(0, 0, 1, 1));
+      return;
+    }
+
+    node.setContentDescription(accessibleTextSpan.description);
+    node.addAction(AccessibilityNodeInfoCompat.ACTION_CLICK);
+    node.setBoundsInParent(getBoundsInParent(accessibleTextSpan));
+    node.setRoleDescription(mView.getResources().getString(R.string.link_description));
+    node.setClassName(AccessibilityRole.getValue(AccessibilityRole.BUTTON));
+  }
+
+  private Rect getBoundsInParent(AccessibilityLinks.AccessibleLink accessibleLink) {
+    // This view is not a text view, so return the entire views bounds.
+    if (!(mView instanceof TextView)) {
+      return new Rect(0, 0, mView.getWidth(), mView.getHeight());
+    }
+
+    TextView textView = (TextView) mView;
+    Layout textViewLayout = textView.getLayout();
+    if (textViewLayout == null) {
+      return new Rect(0, 0, textView.getWidth(), textView.getHeight());
+    }
+
+    Rect rootRect = new Rect();
+
+    double startOffset = accessibleLink.start;
+    double endOffset = accessibleLink.end;
+    double startXCoordinates = textViewLayout.getPrimaryHorizontal((int) startOffset);
+
+    final Paint paint = new Paint();
+    AbsoluteSizeSpan sizeSpan =
+        getFirstSpan(accessibleLink.start, accessibleLink.end, AbsoluteSizeSpan.class);
+    float textSize = sizeSpan != null ? sizeSpan.getSize() : textView.getTextSize();
+    paint.setTextSize(textSize);
+    int textWidth = (int) Math.ceil(paint.measureText(accessibleLink.description));
+
+    int startOffsetLineNumber = textViewLayout.getLineForOffset((int) startOffset);
+    int endOffsetLineNumber = textViewLayout.getLineForOffset((int) endOffset);
+    boolean isMultiline = startOffsetLineNumber != endOffsetLineNumber;
+    textViewLayout.getLineBounds(startOffsetLineNumber, rootRect);
+
+    int verticalOffset = textView.getScrollY() + textView.getTotalPaddingTop();
+    rootRect.top += verticalOffset;
+    rootRect.bottom += verticalOffset;
+    rootRect.left += startXCoordinates + textView.getTotalPaddingLeft() - textView.getScrollX();
+
+    // The bounds for multi-line strings should *only* include the first line. This is because for
+    // API 25 and below, Talkback's click is triggered at the center point of these bounds, and if
+    // that center point is outside the spannable, it will click on something else. There is no
+    // harm in not outlining the wrapped part of the string, as the text for the whole string will
+    // be read regardless of the bounding box.
+    if (isMultiline) {
+      return new Rect(rootRect.left, rootRect.top, rootRect.right, rootRect.bottom);
+    }
+
+    return new Rect(rootRect.left, rootRect.top, rootRect.left + textWidth, rootRect.bottom);
+  }
+
+  @Override
+  protected boolean onPerformActionForVirtualView(
+      int virtualViewId, int action, @Nullable Bundle arguments) {
+    return false;
+  }
+
+  protected @Nullable <T> T getFirstSpan(int start, int end, Class<T> classType) {
+    if (!(mView instanceof TextView) || !(((TextView) mView).getText() instanceof Spanned)) {
+      return null;
+    }
+
+    Spanned spanned = (Spanned) ((TextView) mView).getText();
+    T[] spans = spanned.getSpans(start, end, classType);
+    return spans.length > 0 ? spans[0] : null;
+  }
+
+  public static class AccessibilityLinks {
+    private final List<AccessibleLink> mLinks;
+
+    public AccessibilityLinks(ClickableSpan[] spans, Spannable text) {
+      ArrayList<AccessibleLink> links = new ArrayList<>();
+      for (int i = 0; i < spans.length; i++) {
+        ClickableSpan span = spans[i];
+        int start = text.getSpanStart(span);
+        int end = text.getSpanEnd(span);
+        // zero length spans, and out of range spans should not be included.
+        if (start == end || start < 0 || end < 0 || start > text.length() || end > text.length()) {
+          continue;
+        }
+
+        final AccessibleLink link = new AccessibleLink();
+        link.description = text.subSequence(start, end).toString();
+        link.start = start;
+        link.end = end;
+
+        // ID is the reverse of what is expected, since the ClickableSpans are returned in reverse
+        // order due to being added in reverse order. If we don't do this, focus will move to the
+        // last link first and move backwards.
+        //
+        // If this approach becomes unreliable, we should instead look at their start position and
+        // order them manually.
+        link.id = spans.length - 1 - i;
+        links.add(link);
+      }
+      mLinks = links;
+    }
+
+    @Nullable
+    public AccessibleLink getLinkById(int id) {
+      for (AccessibleLink link : mLinks) {
+        if (link.id == id) {
+          return link;
+        }
+      }
+
+      return null;
+    }
+
+    @Nullable
+    public AccessibleLink getLinkBySpanPos(int start, int end) {
+      for (AccessibleLink link : mLinks) {
+        if (link.start == start && link.end == end) {
+          return link;
+        }
+      }
+
+      return null;
+    }
+
+    public int size() {
+      return mLinks.size();
+    }
+
+    private static class AccessibleLink {
+      public String description;
+      public int start;
+      public int end;
+      public int id;
     }
   }
 }