Description

This issue lists improvements needed to gain reliable and comprehensive coverage on Mobile platforms.

Goals

The purpose of the following improvements is to improve experience with Mono/Mobile testing. Specifically, minimize failures occurring on runtime-extra-platforms CI pipeline so that its status can be monitored using Build Analysis and be used regularly by contributors. Additionally, increase lacking test coverage over some aspects of .NET mobile platforms.

Tasks

The list will be updated as the progress continues...

• Apple mobile test coverage improvements: [tests] Apple mobile test coverage improvements #84254
• Enable x64/arm64 fullAOT-llvm pipelines: Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs #90427
• XHarness improvements
  • Unify console and device output logs for mobile platforms: [mobile] [infra] Unify reporting of mobile CI testing into a single console log file xharness#1188
  • XHarness failed to determine exit code: fail: Application has finished but XHarness failed to determine its exit code! xharness#1189
• Triage failing tests to get knowledge about issues related to: infrastructure (e.g., device not found, time-out), flakiness (tests are nor reliable), runtime (real failures caused by Runtime), etc.
  • Disabling failing tests [Mono][infra] Disable failing tests/pipelines on runtime-extra-platforms #100227
  • Disabling Libraries tests runs for Linux Bionic [mono][android] Disable Linux Bionic [x64/arm64] libraries test runs due to failures #100980
• Triage disabled tests, determine ownership and priority to fix/re-enable
  • Investigate and fix Linux Bionic jobs [mono][android] Investigate failures on Linux Bionic [x64/arm64] jobs #100281
• Introduce tests verifying correct retrieval of locals from devices set in different languages (e.g. Chinese Traditional). This requires changing device/simulator's language and location.
• Remove Mobile CI from the .NET 6 branch and official builds because it's out of support
• Re-visit running runtime tests on Android physical devices Re-enable runtime tests running on Android arm64, when the installation timeout gets fixed #60128