Analysis scope
Period: 2026-04-10 to 2026-04-24 (14 days)
Analysis run: 2026-04-24 09:39 UTC
Repository: dotnet/android
PRs analyzed: 57 merged PRs
Data sources
| Resource |
Details |
| GitHub REST API |
Merged PRs via gh pr list --state merged --limit 100 --json number,title,mergedAt,headRefName,isCrossRepository |
| Azure DevOps pipeline |
Xamarin.Android-PR (definition ID 12278) on devdiv.visualstudio.com, project DevDiv |
| AZDO test runs API |
GET /DevDiv/_apis/test/runs?buildUri=vstfs:///Build/Build/{buildId}&api-version=7.1 |
| AZDO test results API |
GET /DevDiv/_apis/test/runs/{runId}/results?outcomes=Failed&$top=200&api-version=7.1 |
The dotnet-android pipeline on dnceng-public (dev.azure.com) was not used as the primary data source. Since #11153 (merged 2026-04-17), that pipeline runs build-only for direct team-member PRs — device test results for those PRs exist exclusively in the Xamarin.Android-PR devdiv pipeline.
Of the 57 PRs in the window:
For each merged PR, the script used the highest-numbered build ID for that PR in Xamarin.Android-PR (the last/final build, representing the merge commit). Build-to-PR correlation used the sourceBranch field (refs/pull/{number}/merge).
How AZDO auto-retry creates the flakiness signal
When a test fails in Xamarin.Android-PR, the pipeline creates a separate auto-retry run identified by (Auto-Retry) in the test run name (e.g. MSBuildDeviceIntegration On Device - macOS-3 (Auto-Retry)). Only originally-failed tests are re-executed in this run.
Critical data artifact: after an auto-retry run completes, the failedTests counter on the original test run is reset to 0 regardless of the retry outcome. As a result, querying failedTests on standard runs almost always returns 0, even when tests genuinely failed. All test failure evidence lives exclusively in the (Auto-Retry) named runs.
This analysis targets (Auto-Retry) runs and uses the result outcome of each test within those runs as the signal.
A PR is classified as "merged with red CI" if the devdiv Xamarin.Android-PR build's result field was "failed" (evaluated at analysis time). This includes builds where the failure was in a non-test step (setup, artifact publishing, infra) — those will have a failed build result but zero auto-retry test records.
Signal categories
| Signal |
Symbol |
Meaning |
red_auto_retry_passed |
🔴 |
Test appeared in auto-retry run AND passed on retry; the overall build result was failed — the PR was merged past this test failure |
green_auto_retry_passed |
🟢 |
Test appeared in auto-retry run AND passed on retry; the overall build was ultimately succeeded |
auto_retry_failed |
⚠️ |
Test appeared in auto-retry run but still failed on retry — the build remained red due to this test |
The 🔴 column is the highest-confidence flakiness signal: the team observed the failure, CI retried, it passed, and the PR was merged — the team explicitly decided the failure was not a blocker.
Test run environments
All auto-retry runs executed on macOS agents. The test suite is sharded into parallel slots (macOS-1 through macOS-12). Two suites produced auto-retry data in this window:
MSBuildDeviceIntegration On Device — primary device integration test suiteWearOS On Device — WearOS-specific tests (runs on macOS agents with an Android emulator)
A test appearing in "Both" suites means it was observed failing independently in both suite's auto-retry runs across different PRs.
Summary statistics
| Metric |
Count |
| Merged PRs analyzed |
57 |
| Direct (team-member) PRs |
55 |
| Fork PRs |
2 |
PRs merged with failed Xamarin.Android-PR build |
38 |
| Unique test names observed in auto-retry runs |
51 |
| Tests with ≥1 🔴 (passed on retry in a red build) |
33 |
| Tests with ≥1 🟢 (passed on retry in a green build only) |
14 |
| Tests that appear in both 🔴 and 🟢 |
~10 |
Full flaky test table (51 tests)
Sorted by 🔴 count descending, then total PRs affected descending.
Legend: Suite — MSDI = MSBuildDeviceIntegration, WearOS = WearOS On Device
| # |
Test |
Suite |
🔴 RedRetryPass |
🟢 GreenRetryPass |
⚠️ RetryFail |
PRs |
| 1 |
ApplicationRunsWithDebuggerAndBreaks(True,null,"apk",True,MonoVM) |
WearOS |
8 |
1 |
0 |
9 |
| 2 |
Build_XAML_Change(False) |
MSDI |
5 |
0 |
3 |
8 |
| 3 |
DotNetRunWaitForExit |
MSDI |
5 |
0 |
0 |
5 |
| 4 |
ApplicationRunsWithDebuggerAndBreaks(False,null,"aab",True,MonoVM) |
Both |
4 |
1 |
1 |
6 |
| 5 |
ApplicationRunsWithDebuggerAndBreaks(False,"guest1","aab",True,MonoVM) |
Both |
4 |
1 |
1 |
6 |
| 6 |
ApplicationRunsWithDebuggerAndBreaks(False,null,"apk",True,MonoVM) |
WearOS |
4 |
0 |
1 |
5 |
| 7 |
ApplicationRunsWithDebuggerAndBreaks(True,null,"apk",False,MonoVM) |
WearOS |
4 |
0 |
0 |
4 |
| 8 |
ApplicationRunsWithDebuggerAndBreaks(False,"guest1","apk",True,MonoVM) |
Both |
3 |
1 |
2 |
6 |
| 9 |
DesignTimeBuild_CSharp_From_Clean |
MSDI |
3 |
0 |
1 |
4 |
| 10 |
EnsureUncaughtExceptionWorks(MonoVM) |
MSDI |
3 |
0 |
0 |
3 |
| 11 |
ApplicationRunsWithDebuggerAndBreaks(True,null,"aab",True,MonoVM) |
WearOS |
2 |
2 |
0 |
4 |
| 12 |
Install_CSharp_Change |
MSDI |
2 |
0 |
5 |
7 |
| 13 |
Build_AndroidManifest_Change |
MSDI |
2 |
0 |
4 |
6 |
| 14 |
BuildBasicApplicationAndAotProfileIt |
MSDI |
2 |
1 |
1 |
4 |
| 15 |
Build_AndroidResource_Change |
MSDI |
2 |
1 |
1 |
4 |
| 16 |
Install_CSharp_FromClean |
MSDI |
2 |
0 |
0 |
2 |
| 17 |
SupportDesugaringStaticInterfaceMethods(MonoVM) |
MSDI |
2 |
0 |
0 |
2 |
| 18 |
DotNetRunWithDeviceParameter |
MSDI |
2 |
0 |
0 |
2 |
| 19 |
MonoAndroidExportReferencedAppStarts(False,False,MonoVM) |
MSDI |
2 |
0 |
0 |
2 |
| 20 |
ApplicationRunsWithDebuggerAndBreaks(True,"guest1","apk",True,MonoVM) |
WearOS |
1 |
1 |
1 |
3 |
| 21 |
ApplicationRunsWithDebuggerAndBreaks(True,"guest1","aab",True,MonoVM) |
WearOS |
1 |
1 |
0 |
2 |
| 22 |
SupportDesugaringStaticInterfaceMethods(CoreCLR) |
MSDI |
1 |
1 |
0 |
2 |
| 23 |
Build_CSharp_Change |
MSDI |
1 |
0 |
3 |
4 |
| 24 |
Build_AndroidAsset_Change |
MSDI |
1 |
0 |
2 |
3 |
| 25 |
DotNetInstallAndRunPreviousSdk(True,MonoVM) |
WearOS |
1 |
0 |
1 |
2 |
| 26 |
MonoAndroidExportReferencedAppStarts(True,False,CoreCLR) |
MSDI |
1 |
0 |
0 |
1 |
| 27 |
MonoAndroidExportReferencedAppStarts(True,True,CoreCLR) |
MSDI |
1 |
0 |
0 |
1 |
| 28 |
ApplicationRunsWithoutDebugger(True,True,True,MonoVM) |
MSDI |
1 |
0 |
0 |
1 |
| 29 |
ApplicationRunsWithoutDebugger(True,False,True,MonoVM) |
MSDI |
1 |
0 |
0 |
1 |
| 30 |
SubscribeToAppDomainUnhandledException(MonoVM) |
MSDI |
1 |
0 |
0 |
1 |
| 31 |
ExportedMembersSurviveGarbageCollection(True,CoreCLR) |
MSDI |
1 |
0 |
0 |
1 |
| 32 |
JsonDeserializationCreatesJavaHandle(True,CoreCLR) |
MSDI |
1 |
0 |
0 |
1 |
| 33 |
DotNetInstallAndRunPreviousSdk(False,MonoVM) |
WearOS |
1 |
0 |
0 |
1 |
| 34 |
TypeAndMemberRemapping(False,MonoVM) |
MSDI |
0 |
1 |
0 |
1 |
| 35 |
SmokeTestBuildAndRunWithSpecialCharacters("随机生成器",CoreCLR) |
MSDI |
0 |
1 |
0 |
1 |
| 36 |
ApkSet |
MSDI |
0 |
1 |
0 |
1 |
| 37 |
GradleFBProj(True,CoreCLR) |
MSDI |
0 |
1 |
0 |
1 |
| 38 |
DotNetNewAndroidTest(MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 39 |
DotNetInstallAndRunMinorAPILevels(True,"net10.0-android36.1",MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 40 |
UnhandledExceptionFromButtonClick(CoreCLR) |
MSDI |
0 |
0 |
1 |
1 |
| 41 |
InstallWithoutSharedRuntime(CoreCLR) |
MSDI |
0 |
0 |
1 |
1 |
| 42 |
SkiaSharpCanvasBasedAppRuns(True,True,MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 43 |
FixLegacyResourceDesignerStep(False,CoreCLR) |
MSDI |
0 |
0 |
1 |
1 |
| 44 |
InstantRunFastDevDexes(False) |
MSDI |
0 |
0 |
1 |
1 |
| 45 |
CustomLinkDescriptionPreserve(SdkOnly,MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 46 |
GradleFBProj(False,CoreCLR) |
MSDI |
0 |
0 |
1 |
1 |
| 47 |
DotNetRun(False,"llvm-ir",MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 48 |
TestAndroidStoreKey(False,False,"apk","True","file:android","-keystore test.keystore",True) |
MSDI |
0 |
0 |
1 |
1 |
| 49 |
SingleProject_ApplicationId(True,MonoVM) |
MSDI |
0 |
0 |
1 |
1 |
| 50 |
Build_No_Changes |
MSDI |
0 |
0 |
1 |
1 |
| 51 |
Build_XAML_Change(True) |
MSDI |
0 |
0 |
1 |
1 |
Suite totals: MSBuildDeviceIntegration — 39 tests; WearOS — 10 tests; Both — 2 tests (appearing in both suites across different PRs)
Tests with elevated ⚠️ retry-fail counts
Several tests have a notably high ⚠️ count relative to 🔴, meaning they frequently fail and do not recover on retry. These may represent persistent failures in specific configurations or environments, not just flakiness:
| Test |
🔴 |
⚠️ |
PRs |
Install_CSharp_Change |
2 |
5 |
7 |
Build_AndroidManifest_Change |
2 |
4 |
6 |
Build_XAML_Change(False) |
5 |
3 |
8 |
Build_CSharp_Change |
1 |
3 |
4 |
Build_AndroidAsset_Change |
1 |
2 |
3 |
ApplicationRunsWithDebuggerAndBreaks(False,"guest1","apk",True,MonoVM) |
3 |
2 |
6 |
PRs merged with red CI (38)
These PRs had a failed result on their last Xamarin.Android-PR build at the time of merge.
| PR |
Title |
| #11070 |
Guard Mono-specific AOT targets for CoreCLR runtime, add XA1042 warning |
| #11082 |
Remove duplicate @ prefix from issueAuthor in GitOps |
| #11084 |
Run FixLegacyResourceDesigner before trimming |
| #11109 |
Remove broken 'Windows > Tests > Debugging' CI lane |
| #11110 |
Use Assert.Inconclusive for emulator acquisition failures |
| #11112 |
[Mono.Android] fix global ref leak in TypeManager.Activate |
| #11117 |
LEGO: Pull request from juno/hb_6dddf33b-c6da-43d8-ac04-14d2c339cb00_20260415103450130 to main |
| #11121 |
Localized file check-in by OneLocBuild Task: Build definition ID 17928: Build ID 13854166 |
| #11122 |
[TrimmableTypeMap] Implement alias support in codegen and runtime |
| #11124 |
Remove unused HashSet allocations in HashJavaNames |
| #11125 |
Bump external/Java.Interop from 85919bb to 7b018fe |
| #11126 |
Port TypeMapObjectsXmlFile.Import to XmlReader streaming |
| #11127 |
Avoid O(n²) array growth in GenerateTypeMappings |
| #11129 |
Document AndroidInstrumentation and EnableMSTestRunner build properties |
| #11132 |
Compute Java name hashes on demand instead of pre-computing both |
| #11133 |
Use stackalloc in TypeMapHelper to reduce allocations |
| #11141 |
Add investigation & debugging practices to copilot-instructions |
| #11142 |
[TrimmableTypeMap] Package CoreCLR preserve list in SDK pack |
| #11143 |
[TrimmableTypeMap] Manifest generator fixes |
| #11144 |
[TrimmableTypeMap] Fix IL1034 by excluding app assembly from trimmer roots |
| #11149 |
[copilot] Add /review agentic workflow and update android-reviewer skill |
| #11150 |
[copilot] Use Claude Opus 4.6 for android-reviewer workflow |
| #11152 |
Set min-integrity: none on /review workflow |
| #11153 |
[ci] Only run dnceng-public pipeline for fork PRs |
| #11155 |
Localized file check-in by OneLocBuild Task: Build definition ID 17928: Build ID 13878536 |
| #11159 |
Bump com.android.tools.build:manifest-merger from 32.1.0 to 32.1.1 |
| #11160 |
[main] Update dependencies from dotnet/dotnet |
| #11162 |
[tests] Improve NUnit runner reporting and dry-run auditing |
| #11163 |
LEGO: Pull request from juno/hb_6dddf33b-c6da-43d8-ac04-14d2c339cb00_20260420103228482 to main |
| #11164 |
Add network allowlist to android-reviewer workflow |
| #11168 |
[TrimmableTypeMap] Fix UCO boolean return type mismatch causing n_* callback trimming |
| #11169 |
Make CoreCLR the default runtime for Debug builds |
| #11171 |
Bump external/Java.Interop from 7b018fe to 69c9daa |
| #11173 |
Add roles restriction to /review slash command |
| #11175 |
Localized file check-in by OneLocBuild Task: Build definition ID 17928: Build ID 13905231 |
| #11178 |
[TrimmableTypeMap] Add exception handling to UCO constructor callbacks (nctor_*_uco) |
| #11181 |
[TrimmableTypeMap] Per-assembly typemap universes with startup hook initialization |
| #11195 |
[Xamarin.Android.Build.Tasks] Retry RemoveDirFixed on ERROR_DIR_NOT_EMPTY |
Caveats and limitations
- Counter reset artifact: the
failedTests count on original AZDO test runs is reset to 0 after auto-retry. All failure evidence exists exclusively in (Auto-Retry) named runs. Tests that fail without triggering an auto-retry are not captured by this analysis.
- Single sample per PR: only the last build for each PR is analyzed. If a PR was rebuilt multiple times, only the final build's test records are included.
- No error message content: this analysis captures test names and pass/fail outcomes only. Error messages and stack traces were not collected.
- Infrastructure failures: some of the 38 red-CI PRs had failures in non-test stages (setup, artifact publishing). Those PRs have no associated test records in this dataset. The exact count of infra-vs-test failures was not determined.
- Time window: the 14-day window captures the most recent flaky tests but does not reflect longer-term trends. Tests with ⚠️ retry-fail counts of 1 (rows 38–51) may represent one-off events.
- No per-shard attribution: when a test fails on multiple macOS shards for the same PR, each shard failure is counted as a separate ⚠️ or 🔴 occurrence. The PR count (
total_prs) deduplicated this; the signal counts did not.
Analysis scope
Period: 2026-04-10 to 2026-04-24 (14 days)
Analysis run: 2026-04-24 09:39 UTC
Repository: dotnet/android
PRs analyzed: 57 merged PRs
Data sources
gh pr list --state merged --limit 100 --json number,title,mergedAt,headRefName,isCrossRepositoryXamarin.Android-PR(definition ID 12278) ondevdiv.visualstudio.com, projectDevDivGET /DevDiv/_apis/test/runs?buildUri=vstfs:///Build/Build/{buildId}&api-version=7.1GET /DevDiv/_apis/test/runs/{runId}/results?outcomes=Failed&$top=200&api-version=7.1The
dotnet-androidpipeline ondnceng-public(dev.azure.com) was not used as the primary data source. Since #11153 (merged 2026-04-17), that pipeline runs build-only for direct team-member PRs — device test results for those PRs exist exclusively in theXamarin.Android-PRdevdiv pipeline.Of the 57 PRs in the window:
dotnet/android) → test data inXamarin.Android-PRdnceng-publicFor each merged PR, the script used the highest-numbered build ID for that PR in
Xamarin.Android-PR(the last/final build, representing the merge commit). Build-to-PR correlation used thesourceBranchfield (refs/pull/{number}/merge).How AZDO auto-retry creates the flakiness signal
When a test fails in
Xamarin.Android-PR, the pipeline creates a separate auto-retry run identified by(Auto-Retry)in the test run name (e.g.MSBuildDeviceIntegration On Device - macOS-3 (Auto-Retry)). Only originally-failed tests are re-executed in this run.Critical data artifact: after an auto-retry run completes, the
failedTestscounter on the original test run is reset to 0 regardless of the retry outcome. As a result, queryingfailedTestson standard runs almost always returns 0, even when tests genuinely failed. All test failure evidence lives exclusively in the(Auto-Retry)named runs.This analysis targets
(Auto-Retry)runs and uses the result outcome of each test within those runs as the signal.A PR is classified as "merged with red CI" if the devdiv
Xamarin.Android-PRbuild'sresultfield was"failed"(evaluated at analysis time). This includes builds where the failure was in a non-test step (setup, artifact publishing, infra) — those will have a failed build result but zero auto-retry test records.Signal categories
red_auto_retry_passedfailed— the PR was merged past this test failuregreen_auto_retry_passedsucceededauto_retry_failedThe 🔴 column is the highest-confidence flakiness signal: the team observed the failure, CI retried, it passed, and the PR was merged — the team explicitly decided the failure was not a blocker.
Test run environments
All auto-retry runs executed on macOS agents. The test suite is sharded into parallel slots (
macOS-1throughmacOS-12). Two suites produced auto-retry data in this window:MSBuildDeviceIntegration On Device— primary device integration test suiteWearOS On Device— WearOS-specific tests (runs on macOS agents with an Android emulator)A test appearing in "Both" suites means it was observed failing independently in both suite's auto-retry runs across different PRs.
Summary statistics
Xamarin.Android-PRbuildFull flaky test table (51 tests)
Sorted by 🔴 count descending, then total PRs affected descending.
Legend: Suite —
MSDI= MSBuildDeviceIntegration,WearOS= WearOS On DeviceApplicationRunsWithDebuggerAndBreaks(True,null,"apk",True,MonoVM)Build_XAML_Change(False)DotNetRunWaitForExitApplicationRunsWithDebuggerAndBreaks(False,null,"aab",True,MonoVM)ApplicationRunsWithDebuggerAndBreaks(False,"guest1","aab",True,MonoVM)ApplicationRunsWithDebuggerAndBreaks(False,null,"apk",True,MonoVM)ApplicationRunsWithDebuggerAndBreaks(True,null,"apk",False,MonoVM)ApplicationRunsWithDebuggerAndBreaks(False,"guest1","apk",True,MonoVM)DesignTimeBuild_CSharp_From_CleanEnsureUncaughtExceptionWorks(MonoVM)ApplicationRunsWithDebuggerAndBreaks(True,null,"aab",True,MonoVM)Install_CSharp_ChangeBuild_AndroidManifest_ChangeBuildBasicApplicationAndAotProfileItBuild_AndroidResource_ChangeInstall_CSharp_FromCleanSupportDesugaringStaticInterfaceMethods(MonoVM)DotNetRunWithDeviceParameterMonoAndroidExportReferencedAppStarts(False,False,MonoVM)ApplicationRunsWithDebuggerAndBreaks(True,"guest1","apk",True,MonoVM)ApplicationRunsWithDebuggerAndBreaks(True,"guest1","aab",True,MonoVM)SupportDesugaringStaticInterfaceMethods(CoreCLR)Build_CSharp_ChangeBuild_AndroidAsset_ChangeDotNetInstallAndRunPreviousSdk(True,MonoVM)MonoAndroidExportReferencedAppStarts(True,False,CoreCLR)MonoAndroidExportReferencedAppStarts(True,True,CoreCLR)ApplicationRunsWithoutDebugger(True,True,True,MonoVM)ApplicationRunsWithoutDebugger(True,False,True,MonoVM)SubscribeToAppDomainUnhandledException(MonoVM)ExportedMembersSurviveGarbageCollection(True,CoreCLR)JsonDeserializationCreatesJavaHandle(True,CoreCLR)DotNetInstallAndRunPreviousSdk(False,MonoVM)TypeAndMemberRemapping(False,MonoVM)SmokeTestBuildAndRunWithSpecialCharacters("随机生成器",CoreCLR)ApkSetGradleFBProj(True,CoreCLR)DotNetNewAndroidTest(MonoVM)DotNetInstallAndRunMinorAPILevels(True,"net10.0-android36.1",MonoVM)UnhandledExceptionFromButtonClick(CoreCLR)InstallWithoutSharedRuntime(CoreCLR)SkiaSharpCanvasBasedAppRuns(True,True,MonoVM)FixLegacyResourceDesignerStep(False,CoreCLR)InstantRunFastDevDexes(False)CustomLinkDescriptionPreserve(SdkOnly,MonoVM)GradleFBProj(False,CoreCLR)DotNetRun(False,"llvm-ir",MonoVM)TestAndroidStoreKey(False,False,"apk","True","file:android","-keystore test.keystore",True)SingleProject_ApplicationId(True,MonoVM)Build_No_ChangesBuild_XAML_Change(True)Suite totals: MSBuildDeviceIntegration — 39 tests; WearOS — 10 tests; Both — 2 tests (appearing in both suites across different PRs)
Tests with elevated⚠️ retry-fail counts
Several tests have a notably high⚠️ count relative to 🔴, meaning they frequently fail and do not recover on retry. These may represent persistent failures in specific configurations or environments, not just flakiness:
Install_CSharp_ChangeBuild_AndroidManifest_ChangeBuild_XAML_Change(False)Build_CSharp_ChangeBuild_AndroidAsset_ChangeApplicationRunsWithDebuggerAndBreaks(False,"guest1","apk",True,MonoVM)PRs merged with red CI (38)
These PRs had a
failedresult on their lastXamarin.Android-PRbuild at the time of merge.85919bbto7b018femin-integrity: noneon /review workflow7b018feto69c9daa/reviewslash commandCaveats and limitations
failedTestscount on original AZDO test runs is reset to 0 after auto-retry. All failure evidence exists exclusively in(Auto-Retry)named runs. Tests that fail without triggering an auto-retry are not captured by this analysis.total_prs) deduplicated this; the signal counts did not.