August infra rollout: continued cleanup of runtime tests

[trylek]

Follow-up of:

#37440

After the runtime test source code and project scripts have been moved under src/tests, I have the following proposals for further test cleanup:

1. Mop-up changes - moving the rest of src/coreclr/tests/src and ideally the entire src/coreclr/tests folder under src/tests;
2. Move build-test.cmd / build-test.sh from src/coreclr to src/tests; it might make sense to rename them to just build.cmd / build.sh in the process as the "test" part is implied by the folder and becomes superfluous in the script name;
3. Modify the logic for Crossgen1/Crossgen2 compilation of framework assemblies to r2rtest or, if that doesn't work, to MSBuild;
4. Start moving most of functional logic in the build-test scripts to the build.proj script with the ultimate goal of either keeping the two scripts as just thin command-line front-ends or deleting them altogether and triggering test build directly from the root build script. I don't think this is achievable for the August infra rollout in its entirety but it would be good to agree on the general process and start executing on it.

/cc: @dotnet/runtime-infrastructure

[trylek]

/cc: @jkotas @marek-safar @SamMonoRT @naricc @ViktorHofer

[trylek]

@ViktorHofer - As I want to move the build-test scripts under src/tests, I'm wondering where to put the script src/coreclr/setup_vs_tools.cmd that is called both from the CoreCLR build scripts and from the test build script; do you think it would be reasonable to move it under eng with the rationale that it's more or less generally usable?

[jkotas]

Yes, it would be nice to switch other places where we have this logic duplicated while you are on it:
https://github.com/dotnet/runtime/blob/master/src/libraries/Native/build-native.cmd#L43
https://github.com/dotnet/runtime/blob/master/src/installer/corehost/build.cmd#L48

[ViktorHofer]

Ideally we would use what comes with Arcade. I believe they are doing something similar.

[jkotas]

I believe they are doing something similar.

I do not see anything close enough in Arcade.

[trylek]

Thanks @jkotas and @ViktorHofer for your quick feedback. It seems to me that somewhat similar logic resides in the script

runtime/eng/common/tools.ps1

Line 287 in 6998f4c

function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements = $null) {

I can look into switching over the existing scripts to use this logic instead of using several open-coded versions; I have not yet verified if it's a full replacement of the existing VS detection logic bits though.

[jkotas]

I have not yet verified if it's a full replacement of the existing VS detection

It is not. eng/common/tools.ps1 locates msbuild. We need to setup environment for C++ compilers that is quite different.

Also, eng/common/tools.ps1 has non-desirable reliability characteristics. It downloads vswhere from the network, instead of just using the one one the machine. I do not think we want yet another place where the build can fail due to network glitch.

[trylek]

Thanks @jkotas for clarifying. Adding @dotnet/runtime-infrastructure to chime in as I believe JanK's responses imply that:

1. We cannot use the existing Arcade scripting logic to locate and set up VS build environment.

2. Should we perhaps create a new future-facing issue to work with Arcade on adding such logic?

3. As a corollary of (1), for now we need to keep some homebrew scripting logic for VS environment setup.

4. Is src/coreclr/setup_vs_tools.cmd the best implementation? Should we somehow consolidate it with the other versions of this detection logic that JanK mentioned above?

5. We want to unify several places in the runtime repo scripts doing VS detection and setup. Is it reasonable to move the consolidated VS setup script under the eng folder?

[naricc]

So, in this new directory structure, where should any runtime test related utilities go? Undr /src/tests? Specifically I am thinking of the wasm/android test runner I am creating.

[trylek]

@naricc - Right now the various scripts and utilities reside under src/tests/Common inherited from src/coreclr/tests/src/Common; I don't think it's ideal but it was the option causing the least amount of churn. I'm thinking about adding a new folder src/tests/eng where I'd move the various remaining scripts from src/coreclr/tests (perhaps under a coreclr folder) and use for all test infrastructure stuff going forward; ultimately we should probably move the Common folder there too but at this point it feels like expendable busywork to me. This is also what I'd suggest as the root for mono test harness scripts - in your case something like src/tests/eng/wasm/android. I'm open to any suggestions regarding organization of the newly emerging src/tests subtree.