Making Runtime Tests independent of CoreCLR; building a mono version of CoreRun

[naricc]

Currently, in order to execute the runtime tests using mono, we first need to build CoreCLR to produce corerun, then build mono, then patch corerun. Corerun is a minimal dotnet host. (this process is defined in mono.proj here:

runtime/src/mono/mono.proj

Line 903 in feddac7

<Target Name="RunCoreClrTests" DependsOnTargets="ValidateLocalDotnet;PatchCoreClrCoreRoot">

)

This is undesirable because it lengthens the build, but also limits us to running the tests only on those platforms coreclr builds on.

I think the solution is to produce an equivalent to corerun that can just be configured to use one runtime or the other at build time (using runtime packs maybe)?

However, I don't know if that is the right archtectural decision or exactly how to build it. I'm creating this issue to start figuring that out.

@trylek @jkotas @SamMonoRT @marek-safar @steveisok @janvorli

[jkotas]

The corerun sources are in https://github.com/dotnet/runtime/tree/master/src/coreclr/src/hosts/unixcorerun and https://github.com/dotnet/runtime/tree/master/src/coreclr/src/hosts/unixcoreruncommon . If you are looking for something quick, you should be able to take it, convert it to C, and build it as part of Mono. It is not much code so it would not be end of the world to duplicate it, but it is not ideal.

We have increasing need to be able to share parts between runtimes that are not managed libraries. It came up e.g. with event pipe, IL Verify, and I expect that it is only going to come up more frequently as we are working closer together. My proposal would be to start a new runtime/src/shared as a place for the parts shared between coreclr and mono runtimes.

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[marek-safar]

We had corerun working with netcore profile on mono/mono (https://github.com/mono/mono/tree/master/netcore/corerun) the sources are almost identical to what is in dotnet/runtime.

My proposal would be to start a new runtime/src/shared as a place for the parts shared between coreclr and mono runtimes.

I like that. Who could help to set this up?

[ViktorHofer]

I think the solution is to produce an equivalent to corerun that can just be configured to use one runtime or the other at build time (using runtime packs maybe)?

Probably unpopular opinion, why do we still need CoreRun? We have several tracking issues to get off CoreRun in coreclr for tests and use dotnet test (VSTest) instead.

[marek-safar]

I think we'll need both. Corerun to help with bootstrapping on new platforms and dotnet test for easier scaling to multiple platforms on Helix.

[janvorli]

@ViktorHofer I have always assumed that making coreclr tests run using dotnet host meant just running the xunit using that, not the actual tests. The way the coreclr test execution works is that xunit executes test wrappers that further execute test specific .cmd/.sh files and those then execute the actual tests using corerun. The .cmd / .sh need to be preserved so that we can do custom stuff specific for some tests or for specific testing mode. Also, the tests must be run in separate processes from the xunit so that we can set things like GCStress only for the test execution.

[ViktorHofer]

Only arguing about the invocation of CoreRun in the cmd/sh script. Can't we just use the host for that (which is what VSTest does by invoking dotnet exec)?

VSTest currently loads the test assembly from the testhost.dll/exe but with the user defined switches specified in the test app's runtimeconfig.json file. It actually does a dotnet exec testhost.dll x.dll - -runtimeconfig x.runtimeconfig.json - -depsfile x.deps.json. A testhost is invoked per assembly and environment variables can be set via the CLI or in a runsettings file.

Another thought, can/should we use RemoteExecutor for some/most of these tests?

[trylek]

@ViktorHofer - one additional point to consider is that CoreRun is by far the easiest for debugging runtime issues when running the tests exactly because it's just a thin front-end for the runtime. If we think about removing CoreRun from the picture altogether, we should try to make sure that any alternate approach (like RemoteExecutor, VSTest, dotnet or whatnot) provides reasonable diagnostic experience for developers.