[android] Run tests for libdispatch in x64 emulator (wip) #78762

weliveindetail · 2025-01-20T17:48:19Z

Early preview for a ctest-based library that raises some questions:

If we need additional scripts, where should we put them? For example ctest_mock.sh (we might be able to avoid this one).
On my local machine I have to grant adb network access once. How to do that in CI?
How to track the emulator process and make sure: (1) it booted, (2) it shut down

andrurogerz · 2025-01-20T18:11:33Z

Is the goal to make it easy to run these tests locally or to run them as part of CI? I think this change makes sense for local testing, but for CI we tend to put the bulk of the logic into a GitHub workflow. This way, you can leverage many well-tested, supported GitHub actions like nttld/setup-ndk and reactivecircus/android-emulator-runner to install NDK and run tests on Android emulators.

The ds2 github workflow is an example for running non-Swift tests on an Android emulator: https://github.com/compnerd/ds2/blob/main/.github/workflows/build.yml#L460

weliveindetail · 2025-01-21T07:58:46Z

Thanks for the feedback and links @andrurogerz. I am sure a GitHub workflow would be a good solution, but I think we want to run this in Swift CI. Thus, we need to implement it in the build script.

weliveindetail · 2025-01-21T11:09:06Z

If we need additional scripts, where should we put them?

libc++ has utils/ci for such scripts in upstream LLVM: https://github.com/llvm/llvm-project/tree/main/libcxx/utils/ci/vendor/android

How to track the emulator process and make sure: (1) it booted, (2) it shut down

We likely find an answer in the above scripts

On my local machine I have to grant adb network access once. How to do that in CI?

For the moment this question remains as it's Windows-specific. We will see with trial and error.

weliveindetail · 2025-01-21T16:30:10Z

We should consider to cross-compile ctest for the targets we need and use it as a test driver instead of the script

finagolfin · 2025-01-21T19:36:57Z

Interesting, I wanted to test this in the emulator once Android is added to the official CI, so we can probably reuse this script on a linux host then. You're probably aware that the compiler validation suite currently supports running its executable tests in the emulator through adb. You obviously don't use build-script on Windows, but maybe you can just use that through a Python script or it may simply give you ideas. I can't answer any of your questions about running the emulator on Windows, because I don't use Windows, but Andrew is right that you're better off using the existing github actions for that, if those are available in your Windows CI.

weliveindetail · 2025-01-22T10:51:11Z

Thanks for your notes @finagolfin!

This is meant as a first test balloon to run something in the emulator at all in the Swift CI for Windows. I considered LIT-based tests more complicated and wanted to try something simple first. Happy to collaborate if there is overlap with Linux!

You're probably aware that the compiler validation suite currently supports running its executable tests in the emulator through adb.

Yes, I literally found them 2 days ago :) That will help a lot with tests for Swift itself and out-of-tree libs like xctest!

Andrew is right that you're better off using the existing github actions for that, if those are available in your Windows CI

I don't see how we would run these on ci.swift.org, is that possible?

weliveindetail · 2025-01-22T10:53:58Z

@swift-ci Please test Windows

weliveindetail · 2025-01-22T12:44:19Z

Pushed a number of fixes and try to save some CI resources temporarily (only build Windows x64 + Android x86_64, only test dispatch).

@swift-ci Please test Windows

finagolfin · 2025-01-22T13:52:47Z

I don't see how we would run these on ci.swift.org, is that possible?

I don't know how the official Windows CI works. If it doesn't allow running github action workflows, then no, it's not possible.

I think Andrew's point is that you'd be better off running these tests in your existing github-based CI first, before proposing them for non-github-based CI that requires more configuration work.

weliveindetail · 2025-01-22T16:38:47Z

In order to test Android in Swift CI, we need the emulator and adb from Android cmdline-tools. We can download and install them pretty quickly.

However, my above attempt failed, because the machine's Java Runtime version is outdated (55 is from 2018). @shahmishal Is there a chance that the build machines get an update? (61 is from 2021)

T:\android-cmdline-tools.zip not found. Downloading ...
Extracting 'android-cmdline-tools.zip' ...
Error: LinkageError occurred while loading main class com.android.sdklib.tool.sdkmanager.SdkManagerCli
	java.lang.UnsupportedClassVersionError: com/android/sdklib/tool/sdkmanager/SdkManagerCli has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
Error: Error: sdkmanager.bat exited with code 1.
Invocation:
  T:\android-ndk-r26b\.temp\cmdline-tools\bin\sdkmanager.bat --sdk_root=T:\android-ndk-r26b cmdline-tools;latest --channel=3

finagolfin · 2025-01-23T11:30:13Z

If you TBC devs don't control the Windows environment in that CI, it may be using one of the Windows Docker files, in which case you can submit a Java update change there yourself, @weliveindetail.

weliveindetail · 2025-01-23T14:32:45Z

Thanks @finagolfin! Yes, I'd like to consider that once I double-checked it's the right image source.

weliveindetail · 2025-01-24T14:30:04Z

Unfortunately, the Windows builders haven't been ported to docker. In order to unblock progress we're installing Java on the fly now. This is a terrible workaround though. It's a waste of time and resources.

weliveindetail · 2025-01-24T14:30:25Z

@swift-ci Please test Windows

weliveindetail · 2025-01-27T07:54:46Z

The installation failed with:

T:\android-ndk-r26b-windows.zip not found. Downloading ...
Extracting 'android-ndk-r26b-windows.zip' ...
T:\android-cmdline-tools.zip not found. Downloading ...
T:\microsoft-jdk.zip not found. Downloading ...
Extracting 'microsoft-jdk.zip' ...
'android-cmdline-tools.zip' is already extracted and up to date.
Error: The term 'T:\android-ndk-r26b\.temp\cmdline-tools\bin\sdkmanager.bat' is not recognized

Must be an issue with Extract-ZipFile

weliveindetail · 2025-01-27T10:48:20Z

@swift-ci Please test Windows

weliveindetail · 2025-01-27T12:58:23Z

AVD failed to create a virtual device:

Error: Package path is not valid. Valid system image paths are:
null
Error: Error: avdmanager.bat exited with code 1.
Invocation:
  T:\android-ndk-r26b\cmdline-tools\latest\bin\avdmanager.bat create avd --name swift-test-device --package system-images;android-29;default;x86_64

weliveindetail · 2025-01-27T14:38:57Z

@swift-ci Please test Windows

weliveindetail · 2025-01-28T14:50:12Z

The build failed before any of the changes in $WebClient.DownloadFile($URL, $Destination):

T:\android-ndk-r26b-windows.zip not found. Downloading ...
Error: Exception calling "DownloadFile" with "2" argument(s): "The operation has timed out."
    at DownloadAndVerify, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 645
    at Fetch-Dependencies, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 795

Let's try that again.
@swift-ci Please test Windows

weliveindetail · 2025-01-28T15:26:38Z

We reach that step now that creates the virtual device and need to figure out how to avoid the interactive config:

Auto-selecting single ABI x86_64
Do you wish to create a custom hardware profile? [no] offset 0, count -1, length 256
Error: Error: avdmanager.bat exited with code 1.
Invocation:
  T:\android-sdk\cmdline-tools\latest\bin\avdmanager.bat create avd --name "swift-test-device" --package "system-images;android-29;default;x86_64"

weliveindetail · 2025-01-28T16:04:17Z

@swift-ci Please test Windows

weliveindetail · 2025-01-29T14:20:52Z

@swift-ci please test Windows

weliveindetail · 2025-01-29T16:12:49Z

The build failed due to an unrelated issue in LLDB:

llvm-project\lldb\source\Plugins\LanguageRuntime\Swift\ReflectionContext.cpp(405): error C2039: 'ResumeAsyncContext': is not a member of 'swift::reflection::ReflectionContext<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8>>>>::AsyncTaskInfo'

…-device exists

weliveindetail · 2025-01-30T13:01:09Z

swiftlang/swift-corelibs-libdispatch#855
@swift-ci please test Windows

weliveindetail · 2025-01-30T13:39:43Z

Looks like we need the Java runtime for the emulator as well and not only for for cmdline-tools. This worked locally, because the Java Runtime on my host isn't outdated like the one on the CI machines. I will retry once the current run reached the timeout.

weliveindetail · 2025-01-30T14:53:28Z

swiftlang/swift-corelibs-libdispatch#855
@swift-ci please test Windows

weliveindetail · 2025-01-30T15:30:44Z

Still:

error: no devices/emulators found

…empts

weliveindetail · 2025-02-03T13:24:05Z

I can reproduce this locally by deleting the virtual device before running the build script. It seems that we cannot run a virtual device if we just created it, in particular, in the same process tree. The emulator process crashes without further notice. We can even wait and attempt it again and it crashes the same way. My latest patch demonstrates that. Running the build script again, however, works perfectly. So, this isn't an issue with permissions or execution environment.

Instead, I assume that the avdmanager keeps a file handle open and the emulator fails to lock it. Once we close the build script process, the handle is released and next time the emulator can lock it, because we don't need to recreate the virtual devices. I will double-check and see how to close them.

Anyway, let's run through the current state one more time.

swiftlang/swift-corelibs-libdispatch#855
@swift-ci please test Windows

weliveindetail · 2025-02-03T14:19:46Z

Build bot failed as expected:

ANDROID_SDK_HOME = T:\android-sdk
T:\android-sdk\cmdline-tools\latest\bin\avdmanager.bat list avd reports:
Available Android Virtual Devices:
Create Android virtual device for arch x86_64
T:\android-sdk\cmdline-tools\latest\bin\avdmanager.bat list avd reports:
Available Android Virtual Devices:
    Name: swift-test-device-x86_64
    Path: T:\android-sdk\.android\avd\swift-test-device-x86_64.avd
  Target: Default Android System Image
          Based on: Android 10.0 ("Q") Tag/ABI: default/x86_64
  Sdcard: 512 MB
T:\android-sdk\emulator\emulator.exe -list-avds reports:
swift-test-device-x86_64
Start Android emulator for arch x86_64
Waiting for process 4584 to start
Process 4584 failed to start, trying again...
Waiting for process 9360 to start
Process 9360 failed to start, trying again...
Waiting for process 9668 to start
Process 9668 failed to start, trying again...
Waiting for process 5356 to start
Process 5356 failed to start, trying again...
Waiting for process 7624 to start
Error: Android emulator process 7624 failed to start.

weliveindetail · 2025-02-04T11:36:00Z

Trial and error found a fix locally. Let's see if CI reproduces that 🤞
@swift-ci please test Windows

weliveindetail

We got past the emulator startup 🙌

weliveindetail · 2025-02-04T12:49:05Z

utils/build.ps1

+
+      Write-Host "Waiting while Android emulator boots in process $AndroidEmulatorPid"
+      $adb = "$BinaryCache\android-sdk\platform-tools\adb.exe"
+      Invoke-Program $adb "wait-for-device"


Now the build bot hangs here in adb wait-for-device

weliveindetail · 2025-02-04T14:07:36Z

@swift-ci please test Windows

weliveindetail · 2025-02-04T14:46:53Z

No luck with that

T:\android-sdk\emulator\emulator.exe -list-avds reports:
swift-test-device-x86_64
Start Android emulator for arch x86_64
Waiting for process 6632 to start
Waiting while Android emulator boots in process 6632
adb wait-for-device running in process 7648
* daemon not running; starting now at tcp:5037
* daemon started successfully
adb failed to connect. Shutting it down.
adb wait-for-device running in process 3928
adb failed to connect. Shutting it down.
adb wait-for-device running in process 6132
adb failed to connect. Shutting it down.
adb wait-for-device running in process 10728
adb failed to connect. Shutting it down.
adb wait-for-device running in process 6324
Error: Android Debug Bridge process 6324 failed to connect.
    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 1047
    at Isolate-EnvVars, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 610
    at AndroidEmulator-Run, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 993
    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\utils\build.ps1: line 3214
  From System.Management.Automation.RuntimeException: Android Debug Bridge process 6324 failed to connect.
error: no emulator detected

finagolfin · 2025-02-04T14:53:13Z

Are you familiar with the popular ReactiveCircus github workflow to start an Android emulator on macOS/linux? They have it create some kind of polling loop in Typescript that waits to make sure the emulator starts up. Maybe you could look at their OSS code and find something useful for your Windows emulator startup here.

weliveindetail · 2025-02-04T15:18:43Z

No I wasn't aware of it. I am afraid the issues here are very Windows specific, but it's still a good source for common things to try. Like emulator flags -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none for example. Thanks

weliveindetail · 2025-02-20T18:50:04Z

@swift-ci please test Windows

weliveindetail · 2025-02-21T16:28:12Z

@swift-ci please test Windows

weliveindetail · 2025-02-21T17:36:01Z

@swift-ci please test Windows

weliveindetail · 2025-02-21T18:24:29Z

Android emulator requires Hyper-V support in Windows, but the CI machines don't seem to have it.

T:\android-sdk\emulator\emulator.exe -accel-check reports:
accel:
3
Android Emulator requires an Intel/AMD processor with virtualization extension support.  (Virtualization extension is not supported)
accel

weliveindetail requested a review from compnerd January 20, 2025 17:49

weliveindetail force-pushed the android-ctest-libdispatch branch 3 times, most recently from 1b32852 to e01333a Compare January 22, 2025 12:39

weliveindetail added 3 commits January 29, 2025 17:25

[android] Run tests for libdispatch in x64 emulator (wip)

e3d3571

fixup! [android] Run tests for libdispatch in x64 emulator (wip)

00a6f90

[dev] Temporarily enable Android x86_64 in CI and test only dispatch

d025ab2

[dev] Dump virtual devices with avdmanager to confirm that swift-test…

cc59b11

…-device exists

Set Java environment again when starting the emulator

d328dcb

weliveindetail mentioned this pull request Jan 31, 2025

[windows][toolchain] Run Swift Runtime execution tests in x86_64 Android emulator #79081

Closed

[dev] Run and teardown the emulator with debug output and multipe att…

26cfa08

…empts

Running avdmanager via Start-Process fixes the emulator crashes

c2d9b34

weliveindetail commented Feb 4, 2025

View reviewed changes

Let's grant adb a few more attempts

f933288

weliveindetail added 2 commits February 20, 2025 19:35

Merge branch 'main' into android-ctest-libdispatch

d68524a

Try to grant network access permission to adb

984688d

[dev] More emulator params and dump verbose outputs

a3595d7

weliveindetail added 2 commits February 21, 2025 17:39

[dev] Run -accel-check and dump the output

f12cf68

Dump logs in erro case as well

c63f016

weliveindetail mentioned this pull request Feb 24, 2025

[android] build and run LIT-based tests on x64 thebrowsercompany/swift-build#229

Open

weliveindetail mentioned this pull request Mar 14, 2025

utils: Boot and tear-down Android Emulator in build.ps1 #80022

Open

weliveindetail closed this Mar 15, 2025

[android] Run tests for libdispatch in x64 emulator (wip) #78762

[android] Run tests for libdispatch in x64 emulator (wip) #78762

Uh oh!

Conversation

weliveindetail commented Jan 20, 2025

Uh oh!

andrurogerz commented Jan 20, 2025

Uh oh!

weliveindetail commented Jan 21, 2025

Uh oh!

weliveindetail commented Jan 21, 2025

Uh oh!

weliveindetail commented Jan 21, 2025

Uh oh!

finagolfin commented Jan 21, 2025

Uh oh!

weliveindetail commented Jan 22, 2025

Uh oh!

weliveindetail commented Jan 22, 2025

Uh oh!

weliveindetail commented Jan 22, 2025

Uh oh!

finagolfin commented Jan 22, 2025

Uh oh!

weliveindetail commented Jan 22, 2025

Uh oh!

finagolfin commented Jan 23, 2025

Uh oh!

weliveindetail commented Jan 23, 2025

Uh oh!

weliveindetail commented Jan 24, 2025

Uh oh!

weliveindetail commented Jan 24, 2025

Uh oh!

weliveindetail commented Jan 27, 2025

Uh oh!

weliveindetail commented Jan 27, 2025

Uh oh!

weliveindetail commented Jan 27, 2025

Uh oh!

weliveindetail commented Jan 27, 2025

Uh oh!

weliveindetail commented Jan 28, 2025

Uh oh!

weliveindetail commented Jan 28, 2025

Uh oh!

weliveindetail commented Jan 28, 2025

Uh oh!

weliveindetail commented Jan 29, 2025

Uh oh!

weliveindetail commented Jan 29, 2025

Uh oh!

weliveindetail commented Jan 30, 2025

Uh oh!

weliveindetail commented Jan 30, 2025

Uh oh!

weliveindetail commented Jan 30, 2025

Uh oh!

weliveindetail commented Jan 30, 2025

Uh oh!

weliveindetail commented Feb 3, 2025

Uh oh!

weliveindetail commented Feb 3, 2025

Uh oh!

weliveindetail commented Feb 4, 2025

Uh oh!

weliveindetail left a comment

Choose a reason for hiding this comment

Uh oh!

weliveindetail Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weliveindetail commented Feb 4, 2025

Uh oh!

weliveindetail commented Feb 4, 2025

Uh oh!

finagolfin commented Feb 4, 2025

Uh oh!

weliveindetail commented Feb 4, 2025

weliveindetail Feb 4, 2025 •

edited

Loading

weliveindetail commented Feb 21, 2025 •

edited

Loading