Add test leg to the PR build to run libraries tests on iOS x64 Simulators #37476

premun · 2020-06-05T08:55:36Z

Forked from #37057

Adds the iOS test leg that will execute selected BCL tests on an iOS x64 Simulator (13.5) using the OSX.1015.Amd64.Open queue which is our on-prem Mac Mini infrastructure (60 machines).

To prepare the Helix job, we are using the Arcade Helix SDK, more specifically the XHarness Helix SDK which uses the XHarness tool to control the test run.

ghost · 2020-06-05T08:55:42Z

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

azure-pipelines · 2020-06-05T08:57:19Z

Commenter does not have sufficient privileges for PR 37476 in repo dotnet/runtime

src/libraries/sendtohelix.proj

…to prvysoky/use-helix-sdk

None of these fixes are wrong per se (maybe the ApkBuilder changes?) but they complicate the PR by mixing unrelated changes

This reverts commit d1a4226.

This reverts commit 64da851.

…"" This reverts commit 506afd2.

eng/testing/AppleRunnerTemplate.sh

akoeplinger

LGTM!

directhex · 2020-09-04T14:35:14Z

https://helix.dot.net/api/2019-06-17/jobs/ba701e37-fa40-4a78-92d9-729043b36d9c/workitems/Microsoft.Extensions.Configuration.Json.Tests/console

Timed out installing the test app?

safern · 2020-09-04T19:57:04Z

eng/testing/AppleRunnerTemplate.sh


-dotnet xharness ios test --app="$APP_BUNDLE" \
-    --targets=$TARGET \
+dotnet xharness ios test  \


Will we support filtering tests or any xunit parameters? Like -notrait Category=Outerloop? If so, this is wrong and should use RunScriptCommand in the MSBuild side to set this command.

Ideally yes, but we first need a way to flow arbitrary command line arguments to the iOS/Android test apps and that will need some changes in xharness and the iOS/Android test runners, I don't think we need to block the PR on that.

Sounds good. We just need to be conscious that PR validation is running outerloop tests for iOS as well, which might be causing the slow runs.

ah no, we added handling for that already with

runtime/eng/testing/tests.mobile.targets

Line 76 in 3c1b48e

<WriteLinesToFile File="$(PublishDir)xunit-excludes.txt" Lines="$(_withoutCategories.Replace(';', '%0dcategory='))" />

which is read by the testrunner so Outerloop and failing tests should already be excluded.

Ok, sounds good. Maybe we should do it how we do it in other places passing them as an argument but that is in the future, no need to block this PR.

src/libraries/sendtohelixhelp.proj

premun · 2020-09-07T10:15:20Z

https://helix.dot.net/api/2019-06-17/jobs/ba701e37-fa40-4a78-92d9-729043b36d9c/workitems/Microsoft.Extensions.Configuration.Json.Tests/console

Timed out installing the test app?

Looks like it happened again but on the very same machine (dci-macpro-06). I will investigate what's up

premun · 2020-09-08T14:28:22Z

I took the problematic machine offline for investigations so let's see whether the rest goes green now.
I will have time to investigate next week.

premun · 2020-09-08T17:29:47Z

Alright, looks everything is green. So I wonder if we want to check this in or not. The reason is that sometimes the iOS build times out when it is on one of the slow agents (or whatever the reason is), so this will bring flakey-ness. On the other hand, I am not sure whether this is not happening already then?

safern · 2020-09-08T17:56:16Z

I think if we don't check this in we won't find those flaky issues and fix them. However, the question I have is, do we have the capacity yet to enable this on all PRs and CI?

premun · 2020-09-09T07:52:47Z

The capacity for the iOS tests itself is OK. The tests only take 6 minutes right now. We have 59 machines for that and there's 155 of work items so the test itself is usually super short. The capacity for building the iOS tests is coming from AzDO and I am not sure how are we doing there. I will try to find out.

safern · 2020-09-09T16:07:25Z

Building the product and tests took 36 mins in last build which is not bad.

directhex · 2020-09-09T16:46:47Z

Practically speaking, what blockers remain on this PR?

premun · 2020-09-09T21:24:04Z

@directhex Practically speaking, what blockers remain on this PR?

So, from my point of view (and please consider that this is my first and only interaction with dotnet/runtime so far and I don't have experience in how the overall PR builds behave):

Building the iOS apps times out every now and then when a "slow agent" is processing the build. A re-run is needed. I am not sure whether this causes problems with other legs already since it doesn't seem to be tied with this particular leg but..
I don't know how often it happens on regular basis that you need to re-run a leg in a PR build so I don't know what is the "regular flakey-ness".
This week one machine somehow went bad and installation of apps started to hang. I took it offline for investigations but I am not sure about the root cause still and I am not sure this won't happen again to bring more flakey-ness. (but as @safern noted, maybe the only way to find out is to check it in and observe).
I am not sure how to approach the question of capacity. I personally think we are fine for iOS but again - maybe the only way to see is to check it in.

So.. I don't know how flakey the PR builds in dotnet/runtime are already and whether we won't bring a lot of pain by checking this in, but I guess this is up to you guys. I don't know whether we shouldn't just flag the leg as not required and observe first or what is the regular procedure when adding new platforms, but I just don't have experience with this repository to decide this.

Other than that, the iOS tests part seems quite stable and fast, so..

safern · 2020-09-09T21:46:09Z

Building the iOS apps times out every now and then when a "slow agent" is processing the build. A re-run is needed. I am not sure whether this causes problems with other legs already since it doesn't seem to be tied with this particular leg but..

We're already seeing this in other OSX, iOS legs and it seems to be an AzDo bug which seems to be under investigation so I wouldn't be concerned by this and also seems like it has somewhat being mitigated.

This week one machine somehow went bad and installation of apps started to hang. I took it offline for investigations but I am not sure about the root cause still and I am not sure this won't happen again to bring more flakey-ness. (but as @safern noted, maybe the only way to find out is to check it in and observe).

I still stand my point the best way to mitigate issues is to check-in and investigate issues that come show up.

I am not sure how to approach the question of capacity. I personally think we are fine for iOS but again - maybe the only way to see is to check it in.

Given the number of machines you shared above and the time it is taking to process work items it seems fine. We won't know this with certainty until we check it in, and as we enable more test projects and run on more PRs if we hit an issue we can always disable the leg as we did for a couple of days for WASM until we got more capacity.

Overall, this LGTM and we should try checking it in, specially since master is already 6.0.

steveisok · 2020-09-10T12:30:15Z

@premun Please give your approval on the PR and we can merge.

ViktorHofer · 2020-09-10T14:11:28Z

eng/testing/AppleRunnerTemplate.sh

-TARGET_ARCH=$2
-TARGET=
-SCHEME_SDK=
+[[RunCommands]]


I'm trying to get rid of the runcommands templating mechanism as having individual runner scripts for different targets (ios/android/wasm) seems to be sufficient. Doing that in #39923. That will basically undo this change here. Any objections?

No, I support that change as we should be able to define it via MSBuild.

No objections from me.

@directhex - any objections?

Dotnet-GitSync-Bot added the area-Infrastructure label Jun 5, 2020

Add XHarness Helix SDK support

be7b233

premun force-pushed the prvysoky/use-helix-sdk branch from 71b4ee2 to be7b233 Compare June 5, 2020 09:40

premun added 2 commits June 5, 2020 11:51

Try to build mono libraries

066bafa

Try to build mono libraries

ebbf08b

premun commented Jun 5, 2020

View reviewed changes

src/libraries/sendtohelix.proj Outdated Show resolved Hide resolved

premun and others added 15 commits June 5, 2020 12:16

Address some of Santiago's feedback from Jo's PR

83d7cab

Merge runtime.yml from master

1379eef

Build iOSx64 libraries

98377f1

Try to build mono product

495c602

Try to build mono product

c571262

Merge master

636bc66

Do not build AllSubsets

9d9da07

Remove dead code

21d5ebc

Revert: Do not built AllSubsets

df64e06

Disable test archivation

553147d

Improvements for local test running

9e7d3f2

Merge branch 'prvysoky/use-helix-sdk' of github.com:premun/runtime in…

138706f

…to prvysoky/use-helix-sdk

Return dead code

2a2243d

Reintroduce whitespace errors

030511c

None of these fixes are wrong per se (maybe the ApkBuilder changes?) but they complicate the PR by mixing unrelated changes

Bump Helix SDK and keep trailing slashes

d1a4226

premun mentioned this pull request Jun 8, 2020

Add test leg to the PR build to run libraries tests on Android emulators #37585

Merged

premun and others added 6 commits June 8, 2020 12:58

Revert "Bump Helix SDK and keep trailing slashes"

d3366ae

This reverts commit d1a4226.

Try building 2 app bundles

16168da

Try a cleaner .app find in msbuild, from @akoeplinger

64da851

Revert "Try a cleaner .app find in msbuild, from @akoeplinger"

506afd2

This reverts commit 64da851.

Do not archiveTests on iOS, it causes tests to be included twice

6778369

Revert "Revert "Try a cleaner .app find in msbuild, from @akoeplinger…

f3252fd

…"" This reverts commit 506afd2.

premun added 2 commits September 4, 2020 09:45

Fix comments

ca4a175

Merge remote-tracking branch 'dotnet/master' into prvysoky/use-helix-sdk

fff75e7

akoeplinger reviewed Sep 4, 2020

View reviewed changes

eng/testing/AppleRunnerTemplate.sh Outdated Show resolved Hide resolved

Print the output path instead of listing the files

51f1849

akoeplinger reviewed Sep 4, 2020

View reviewed changes

akoeplinger approved these changes Sep 4, 2020

View reviewed changes

Merge branch 'master' into prvysoky/use-helix-sdk

9360c81

safern reviewed Sep 4, 2020

View reviewed changes

src/libraries/sendtohelixhelp.proj Outdated Show resolved Hide resolved

safern reviewed Sep 4, 2020

View reviewed changes

src/libraries/sendtohelixhelp.proj Outdated Show resolved Hide resolved

premun added 3 commits September 7, 2020 12:30

Merge remote-tracking branch 'dotnet/master' into prvysoky/use-helix-sdk

a266f68

Reset timeouts and only change it for iOS/tvOS

f097772

Merge remote-tracking branch 'dotnet/master' into prvysoky/use-helix-sdk

8a1ed9c

Merge remote-tracking branch 'dotnet/master' into prvysoky/use-helix-sdk

af9244b

safern approved these changes Sep 9, 2020

View reviewed changes

premun merged commit bb921d6 into dotnet:master Sep 10, 2020

ViktorHofer reviewed Sep 10, 2020

View reviewed changes

ghost locked as resolved and limited conversation to collaborators Dec 8, 2020

premun deleted the prvysoky/use-helix-sdk branch January 26, 2021 13:03

Add test leg to the PR build to run libraries tests on iOS x64 Simulators #37476

Add test leg to the PR build to run libraries tests on iOS x64 Simulators #37476

Uh oh!

Conversation

premun commented Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 5, 2020

Uh oh!

azure-pipelines bot commented Jun 5, 2020

Uh oh!

Uh oh!

Uh oh!

akoeplinger left a comment

Choose a reason for hiding this comment

Uh oh!

directhex commented Sep 4, 2020

Uh oh!

safern Sep 4, 2020

Choose a reason for hiding this comment

Uh oh!

akoeplinger Sep 4, 2020

Choose a reason for hiding this comment

Uh oh!

safern Sep 4, 2020

Choose a reason for hiding this comment

Uh oh!

akoeplinger Sep 4, 2020

Choose a reason for hiding this comment

Uh oh!

safern Sep 4, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

premun commented Sep 7, 2020

Uh oh!

premun commented Sep 8, 2020

Uh oh!

premun commented Sep 8, 2020

Uh oh!

safern commented Sep 8, 2020

Uh oh!

premun commented Sep 9, 2020

Uh oh!

safern commented Sep 9, 2020

Uh oh!

directhex commented Sep 9, 2020

Uh oh!

premun commented Sep 9, 2020

Uh oh!

safern commented Sep 9, 2020

Uh oh!

steveisok commented Sep 10, 2020

Uh oh!

ViktorHofer Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

safern Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

steveisok Sep 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

premun commented Jun 5, 2020 •

edited

Loading

steveisok Sep 10, 2020 •

edited

Loading