Skip to content

[Android] Run CoreCLR functional tests on Android #112283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 12, 2025
Merged

[Android] Run CoreCLR functional tests on Android #112283

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d1a6018
Setup a new job
kotlarmilos Feb 7, 2025
0ff724c
Remove -cross argument on linux
kotlarmilos Feb 7, 2025
6d38b25
Add android job to the runtime pipeline
kotlarmilos Feb 7, 2025
956e00d
Use android test exclusions
kotlarmilos Feb 10, 2025
5a8042e
Remove included projects
kotlarmilos Feb 10, 2025
63cd36e
Remove unused attribute
kotlarmilos Feb 10, 2025
388c626
Update eng/pipelines/extra-platforms/runtime-extra-platforms-androide…
kotlarmilos Feb 11, 2025
2cc3984
Update build command to include host+packs
kotlarmilos Feb 11, 2025
13dbb57
Don't include test assemblies in coreclr andorid jobs
kotlarmilos Feb 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions eng/pipelines/extra-platforms/runtime-extra-platforms-androidemulator.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,35 @@ jobs:
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Android emulators
# Build the whole product using CoreCLR and run libraries tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these library tests? I thought that functional tests are usually under the runtime test suit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functional tests are under libraries. There might be smoke tests under the runtime tests.

Copy link
Member

@matouskozak matouskozak Feb 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I thought it was the reverse. The https://github.com/dotnet/runtime/blob/main/src/tests/FunctionalTests/Android/Device_Emulator/JIT/Android.Device_Emulator.JIT.Test.csproj is placed under src/tests so I got mistaken that it is part of runtime tests.

In case of smoke tests, I think these run functional tests and System.Runtime.Tests which comes from libraries.

#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
buildConfig: Release
runtimeFlavor: coreclr
platforms:
- android_x64
- android_arm64
Comment on lines +133 to +134
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have arm64 emulators setup? I thought that by default these will go to devices, see

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Lines 96 to 97 in a4f9ec2

- ${{ if in(parameters.platform, 'android_arm', 'android_arm64', 'linux_bionic_arm', 'linux_bionic_arm64') }}:
- Windows.11.Amd64.Android.Open

Comparing the logs between x64 and am64, I would say that the arm64 are run on devices https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-112283-merge-8bd807b9687245838c/Android.Device_Emulator.JIT.Test/1/console.36c43a9c.log?helixlogtype=result.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it seems that arm64 run on devices so we can move it to extra-platforms-android.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also wondering, if we have the same runs in runtime.yml (which runs always), why we need them again in extra-platforms?

I think we can have just runtime.yml for now and when we include full libraries run we should move that to extra-platforms.

variables:
# map dependencies variables to local variables
- name: librariesContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: coreclrContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_coreclr.containsChange'] ]
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_CoreCLR
isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }}
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs+libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true /p:TestAssemblies=false
timeoutInMinutes: 180
# extra steps, run tests
postBuildSteps:
- template: /eng/pipelines/libraries/helix.yml
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
41 changes: 41 additions & 0 deletions eng/pipelines/runtime.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -964,6 +964,47 @@ extends:
eq(variables['monoContainsChange'], true),
eq(variables['isRollingBuild'], true))

#
# Android devices
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devices or emulators?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be x64 emulator and arm64 device.

# Build the whole product using CoreCLR and run libraries tests
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
buildConfig: Release
runtimeFlavor: coreclr
platforms:
- android_x64
- android_arm64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these indeed run on devices (see my other comment), we need to make sure we have enough devices so that we can run them on PRs by default and not block other Android jobs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can we test the load?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should monitor the Windows.11.Amd64.Android.Open queue in Grafana to see that the waiting times don't grow so that we don't slow down runtime pipeline. I'm not aware of any stress we have for this.

variables:
# map dependencies variables to local variables
- name: librariesContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: coreclrContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_coreclr.containsChange'] ]
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_CoreCLR
buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs++libs.tests+host+packs -c $(_BuildConfig) /p:ArchiveTests=true /p:TestAssemblies=false
timeoutInMinutes: 480
condition: >-
or(
eq(stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
eq(stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true),
eq(variables['isRollingBuild'], true))
# extra steps, run tests
postBuildSteps:
- template: /eng/pipelines/libraries/helix.yml
parameters:
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)
condition: >-
or(
eq(variables['librariesContainsChange'], true),
eq(variables['coreclrContainsChange'], true),
eq(variables['isRollingBuild'], true))

#
# iOS/tvOS devices - Full AOT + AggressiveTrimming to reduce size
# Build the whole product using Mono and run libraries tests
Expand Down
7 changes: 6 additions & 1 deletion src/libraries/tests.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,7 @@
BuildInParallel="false" />
</ItemGroup>

<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(RunGrpcTestsOnly)' != 'true' and '$(TargetOS)' == 'android'">
<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(RunGrpcTestsOnly)' != 'true' and '$(TargetOS)' == 'android' and '$(RuntimeFlavor)' == 'Mono'">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing this condition?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are adding the condition to separate mono and coreclr ProjectReference.

<ProjectReference Include="$(MonoProjectRoot)sample\Android\AndroidSampleApp.csproj"
BuildInParallel="false" />
<ProjectReference Include="$(RepoRoot)\src\tests\FunctionalTests\Android\**\*.Test.csproj"
Expand Down Expand Up @@ -759,6 +759,11 @@
AdditionalProperties="%(AdditionalProperties);_IsPublishing=true;Configuration=Release;PublishDir=$(XUnitLogCheckerLibrariesOutDir);ROOTFS_DIR=$(ROOTFS_DIR);RuntimeIdentifier=$(OutputRID)" />
</ItemGroup>

<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(TargetOS)' == 'android' and '$(RuntimeFlavor)' == 'CoreCLR'">
<ProjectReference Include="$(RepoRoot)\src\tests\FunctionalTests\Android\Device_Emulator\JIT\Android.Device_Emulator.JIT.Test.csproj"
BuildInParallel="false" />
</ItemGroup>

<Target Name="GenerateMergedCoverageReport"
AfterTargets="Test"
DependsOnTargets="GenerateCoverageReport"
Expand Down
Loading