Android emulator not booting completely on Helix queue

[akoeplinger]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=472093

Build leg reported

android-x86 Release AllSubsets_Mono

Pull Request

dotnet/runtime#93220

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "Did not detect boot completion variable on device",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=472093
Error message validated: Did not detect boot completion variable on device
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/20/2023 10:43:09 AM UTC

Report

Build	Definition	Test	Pull Request
1027363	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1027038	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1026668	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1025934	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1025626	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution
1024193	dotnet/runtime	System.Net.NameResolution.Pal.Tests.WorkItemExecution
1023955	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1023237	dotnet/runtime	Microsoft.Extensions.Configuration.Functional.Tests.WorkItemExecution
1023185	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	dotnet/runtime#114148
1022138	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1021506	dotnet/xharness	System.Buffers.Tests-x64.WorkItemExecution	dotnet/xharness#1401
1021410	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution
1021153	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1021008	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1018968	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114757
1017254	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
1017227	dotnet/runtime	System.Xml.Schema.Extensions.Tests.WorkItemExecution
1015761	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	dotnet/runtime#114633
1016458	dotnet/runtime	System.Transactions.Local.Tests.WorkItemExecution
1016396	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution
1016082	dotnet/runtime	System.Console.Manual.Tests.WorkItemExecution
1015074	dotnet/arcade	System.Buffers.Tests-x64.WorkItemExecution	dotnet/arcade#15735
1015003	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114602
1014675	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114598
1014531	dotnet/runtime	System.Composition.Hosting.Tests.WorkItemExecution
1013746	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
1012838	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#113977
1012245	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114003
1012108	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114448
1011539	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114410
1011252	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	dotnet/runtime#113313
1011261	dotnet/runtime	System.Composition.Tests.WorkItemExecution
1010491	dotnet/runtime	System.Runtime.Loader.Tests.WorkItemExecution
1008937	dotnet/runtime	System.Runtime.ReflectionInvokeInterpreted.Tests.WorkItemExecution
1008364	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1008163	dotnet/runtime	Common.Tests.WorkItemExecution
1007936	dotnet/runtime	System.Tests.TimeZoneInfoTests.NoAlternateUTCIdsInvariant	dotnet/runtime#114148
1006965	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1006905	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114304
1006782	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1006646	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1006575	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114300
1006102	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1005541	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114263
1005296	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
1005186	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	dotnet/runtime#114148
1004581	dotnet/runtime	System.ServiceModel.Syndication.Tests.WorkItemExecution
1004551	dotnet/runtime	Android.Device_Emulator.PInvoke.Test.WorkItemExecution
1004326	dotnet/runtime	System.Net.Http.Json.Unit.Tests.WorkItemExecution
1004300	dotnet/runtime	System.Transactions.Local.Tests.WorkItemExecution
1003241	dotnet/runtime	Android.Device_Emulator.Aot.Test.WorkItemExecution
1003328	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114170
1003296	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#113286
1002979	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	dotnet/runtime#113313
1002851	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1002815	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1001409	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution	dotnet/runtime#113313
1000655	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution	dotnet/runtime#113313
999459	dotnet/runtime	Android.Device_Emulator.JIT.Test.WorkItemExecution	dotnet/runtime#114067
999130	dotnet/runtime	Android.Device_Emulator.PInvoke.Test.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	9	60

[premun]

fyi @dougbu this seems to be catching cases when Android emulators are not booted properly

[akoeplinger]

Not sure why the result validation doesn't match, do we need to set up something special to monitor the runtime-extra-platforms pipeline?

[dougbu]

This feels very similar to #1383 and #1415. The general theme is the emulator isn't starting as quickly as expected (there's a 5 minute loop checking for boot_completed in the XHarness case) or just isn't started. We haven't made much progress on either issue, partially because only @premun knows much about the emulators and he's busy elsewhere.

the ubuntu.2204.amd64.android.29.open queue is one of many we've had problems with when deploying in our staging environment.

I can see how dotnet/xharness#1106 could help here and suggest we keep an eye on this issue for additional hits.

[akoeplinger]

#1383 should be different since that is about Android devices i.e. there's no emulator to start so if they report not booted the device is usually hosed.

If the emulator issue is really about not starting fast enough I think I'd be happy if you add a sleep 5min into the VM provisioning as a quick workaround.

[AlitzelMendez]

Not sure why the result validation doesn't match, do we need to set up something special to monitor the runtime-extra-platforms pipeline?

for this particular question the problem is not the runtime-extra-platforms (we are analyzing it), it is a problem on our side when there are helix work items internal retries, we are not analyzing the logs of all the attempts, created an issue for this: #1467

[dougbu]

I'd be happy if you add a sleep 5min into the VM provisioning as a quick workaround.

there's a 5 minute loop just prior to the failing sys.boot_completed search in the function starting at https://github.com/dotnet/xharness/blob/38841f0f33ca713ca5d6388c681bdd911425b488/src/Microsoft.DotNet.XHarness.Android/AdbRunner.cs#L191

personally, I'm nervous about adding Thread.Sleep(...) in that code b/c @premun seemed confident my similar actions for #1415 (where I extended a loop searching for a different readiness signal) were unhelpful. we found that "fix" only reduced the likelihood of our validation failures; a build soon after my fix went in failed again and we (temporarily❔) gave up

if someone understands dotnet/xharness better, please chime in❕

[dougbu]

@akoeplinger how are things going w/ your fix attempt(s)❔

[akoeplinger]

I just came back from vacation, will take another stab at this early next week :)