[Android] Using AutomationId prevents TalkBack screenreader accessibility

[tomgilder]

Description

When using the AutomationId property on Android, apps end up almost completely inaccessible to users who use Android's TalkBack screenreader.

Xamarin.Forms "borrows" the ContentDescription property on Android for Automation IDs. These IDs polute Android's TalkBack accessibility tree, making apps almost impossible to navigate.

This means you can support test automation or accessibility, not both. Our app needs to support both 😃

I can't find any easy way of working around this restriction, short of walking the Android view tree and setting ContentDescription to null. Being unable to update AutomationId once set also adds to the difficulty of working around this issue.

Potential solutions

1. Use another system for setting AutomationIDs

Probably a non-starter for backwards compatibility?

2. Monitor AccessibilityManager to see if accessibility is enabled

Assuming there’s no alternative property to use for setting IDs, one solution would be for Xamarin.Forms to use Android's AccessibilityManager and check the IsTouchExplorationEnabled property.

This is true when the TalkBack screen-reader is being used, and Forms could ignore AutomationId and not set ContentDescription. Forms should then also use an ITouchExplorationStateChangeListener to monitor changes to this state.

Steps to Reproduce

1. Create any element with an AutomationId set
2. Go to that element on an Android device with TalkBack enabled

Expected Behavior

Element is ignored, or its actual content is read.

Actual Behavior

Element's AutomationId is read to the user.

Basic Information

• Version with issue: 2.5.0.121934
• Last known good version: none. Always happened since AutomationId supported.

[nexxuno]

Hello,
I'm having this problem too, this is how I workaround it while waiting for an improved xamarin version.

In xaml production code:
<Button AutomationProperties.Name="{international:BMapTranslate goLabel}" MinimumWidthRequest="150" Text="{international:BMapTranslate goLabel}" ...

In Xamarin Coded UI testing:
app.Tap(c => c.Button(tr.GetTranslation("goLabel")));

In other words I'm using the same translator class and resource files I use in production as helpers in the UItests. I know it's not optimal but it works for now...

[codingL3gend]

@tomgilder

i was able to find a work around to this issue by creating a customrenderer and respective custom component to allow for overriding the native android method(s) that get triggered when accessibility events are fired. you will need to create some bindable properties on your custom component that you can access in the custom renderer to allow for setting the content description value to what you want but that's simple enough.

this method gets triggered in the control/custom renderer whenever an accessibility event is fired

public override bool OnRequestSendAccessibilityEvent(Android.Views.View child, AccessibilityEvent e)
  {
         if (AccessibilityHandler.IsAccessibilityEnabled(_context) && child != null)
         {
                if (!string.IsNullOrEmpty(_automationId) && _automationId.Equals(child.ContentDescription))
                {
                    child.ContentDescription = $"{_automationName} {_helpText}";
                }
         }

         return base.OnRequestSendAccessibilityEvent(child, e);
}

then you can set the contentDescription value of the control/custom renderer back to what the automationId value was originally when the control/custom renderer is detached from the view.

protected override void OnDetachedFromWindow()
  {
        base.OnDetachedFromWindow();

        if (!string.IsNullOrEmpty(_automationId))
        {
            Control.ContentDescription = _automationId;
        }
  }

helper class

public static class AccessibilityHandler
    {
        public static bool IsAccessibilityEnabled(Context context)
        {
            var accessibility = (AccessibilityManager)context.GetSystemService(MainActivity.AccessibilityService);

            return accessibility?.GetEnabledAccessibilityServiceList(Android.AccessibilityServices.FeedbackFlags.Spoken)?.Count > 0;
        }
    }

[hminaya]

@jimbobbennett have you run into this before? any thoughts?

[claudiosanchez]

@brminnick and @davidortinau are aware of this.

[claudiosanchez]

@codingL3gend How many renderers did you have to create? Have you had any additional issues with the approach you suggested?

[codingL3gend]

@claudiosanchez i think your question may have been meant to be directed toward me. We already had renderers created for other reasons outside of needed ada and automation support; so I just leveraged those implementations. However for any controls where we need this support that we don't have renderers for already, we will create new ones for those. So far we haven't had any issues with this approach thus far and it is working as expected with the controls we have tested it against.

[claudiosanchez]

@codingL3gend You are correct. I fixed the mention :). We implemented your workaround and so far is working like a charm for a Button Renderer, and your answer is on point. I wanted to know what had been the experience with this workaround so far. We are going to propose this solution to the development team 💯. cc: @brminnick

[codingL3gend]

@claudiosanchez glad to hear that I could be of assistance with the workaround. currently our application is still in the development phase and have only gone through local testing so we don't know if any major issues will arise from it if any. We've also ran it against some automated testing and its performing as expected. Hopefully the implementation will help others in the future and perhaps help with an overall platform solution soon enough.

[claudiosanchez]

@codingL3gend Happy to report we have implemented a button solution based on your guidance. The hope is to submit back to the code base at some point. What other renderers did you have implement to support both Accessibility and UI Test Automation?

[codingL3gend]

@claudiosanchez that's great news to hear; happy I could help in some capacity with providing a solution. the other renderers implemented were around the default xamarin forms controls built on top of the native controls such as Label, Switch, DateEntry, LinkButton, and Entry. these are the more common controls that are being used; although custom controls have been created for ListView, ScrollView, and TableView. those renderers haven't needed any specific Accessibility or UI Test Automation configuration.

[codingL3gend]

@claudiosanchez also want to add a quick touch point. we noticed that on android when adding automationId to elements on the xamarin forms side (not sure if this is a result of our custom renderers or just how the controls would get created in general) the android resulting control would be wrapped in an additional ViewGroup. meaning that instead of say a button being wrapped in a single ViewGroup parent container and then the resulting Button element as its child, it is instead 2 Parent ViewGroup containers and then the resulting Button element as the inner most child. as a result the direct Parent ViewGroup of the Button then gets its ContentDescription value assigned by default to the child's or element's (i.e Button) AutomationId property plus _Container such as Button_Container. this in turn causes these container elements to be read when the top most parent element is clicked on a View when the Accessibility tree is triggered. even using the FastRenderers (though in beta) does not flatten the elements enough to not include the direct ViewGroup parent of the Button element. this is the next challenge that will be tackled from a workaround standpoint to get either those elements to not read the ContentDescription of Button_Container or ideally to not have the additional ViewGroup to be created at all. if there is a known workaround for this, i'd appreciate the direction.

hopefully that all made sense. I can provide a screen shot if needed.

[geeetarguy]

@codingL3gend @claudiosanchez Did you ever find a fix for this? I'm experiencing the same thing with a ListView reported here: #6758

[codingL3gend]

@geeetarguy we did find a fix for this based on the implementation I posted above. however when we updated the version of our Android plugins the implementation began to run into issues. Updating to 4.0 to take advantage of Fast Renderers may help your situation. Also we had to fall back to an implementation that I had as a worst case scenario which is to have two different builds where one would have automation turned on and one would not using compiler flags. hope this helps.

[jfversluis]

I’m wondering it this would be enough to fix it: ba65be1

Simply skipping over the assigning of the AutomationId whenever the AccessibilityManager.IsTouchExplorationEnabled is true. This means you cannot have accessibility on when doing automated tests, but I guess that is the implication of the implementation.

@samhouts thought? Also in light of #5237

[codingL3gend]

@jfversluis that's not a bad idea because we have had to separate our implementations out to have a build for automation vs a build for accessibility. however for situations where accessibility has to tested through automation as well this will not suffice.

[ryl]

I've has some luck using an AccessibilityDelegate to provide text to TalkBack without disrupting ContentDescription.

[jfversluis]

@ryl, sorry for the late reply. Could you define "some luck"? Can we translate it into a fully usable solution perhaps?

[ryl]

@jfversluis Take a look at the UI test I created in this pull request (https://github.com/xamarin/Xamarin.Forms/pull/5237/files#diff-420d1f5f1ff2e21615eb786f50ef8328). Using the AccessibilityDelegate, I am able to find a variety of elements using their AutomationId, but their Name + HelpText is still read out correctly by TalkBack. My original bug report was only focused on buttons, so I'm not sure how well it applies across the entire board of elements. It might need some (or a lot of) refinement. It seems like it might be a good starting point, though.

The nice thing about the approach: ContentDescription and AutomationId continue to work like they do now, but the addition of an AccessibiltiyDelegate allows TalkBack to read from a source other than ContentDescription.

[jfversluis]

It looks like AccessibilityDelegate is available on the View object, so I guess it would be available on all visual elements and we can use this to create a solution for this while retaining both the AutomationId functionality as well as gaining accessibility?

[joshardt]

Any news on this? We do have UI tests in our project but also want to support accessibility. What should we do?

[rachelkang]

Hi everyone! Apologies for the late update - in case you missed it, this was fixed earlier this summer by #14089. All the awesome fixes @PureWeen made in this PR are set up behind an Accessibility_Experimental flag, which is available in XF 5.0.0.2083 and later versions.

If you aren't familiar with experimental flags, you can read more about them here: https://docs.microsoft.com/en-us/xamarin/xamarin-forms/internals/experimental-flags. But in short, after updating XF, enable the flag by adding the following to your code-behind, after the InitializeComponent and before setting the MainPage:

Xamarin.Forms.Device.SetFlags(new List<string> { "Accessibility_Experimental"});

Furthermore, be sure to check out the new SemanticEffects properties in XCT which also circumvents the AutomationID issue. These can also be applied to your XF projects, and reflect the properties that will be brought into .NET MAUI as well.

Hope this helps, and let us know if you have more questions! :)