[Quality Week] Improve quality of Crossgen2 testing, most notably in context of library tests

[trylek]

Over the course of 2 years since we shipped the first preview of Crossgen2 we have identified several testing gaps that bit us in various phases of development. We would like to at least create an action plan how to address them and ideally implement at least part of the work in context of the Core Runtime team Quality Weeks.

The problem

• Library testing in the runtime repo doesn't target Crossgen2 at all, neither for compiling the framework nor the individual test apps.

• While CoreCLR does implement Crossgen2 testing, it does so using a raw freshly built Crossgen2, not the NativeAOT-compiled package we ship to customers. Ideally we should test what customers are using both in CoreCLR and in library testing.

• Implementing support for Crossgen2 in the library tests would let us expand testing in directions that weren't previously possible - e.g. testing of the new feature of "speculative" cross-version inlining - that would have very limited coverage in many of the tiny runtime tests.

Overall approach

To address these issues, we're broadly thinking about the following:

• Somehow restructuring the runtime build so that both CoreCLR and library testing can leverage the "final" Crossgen2 produced as part of packs.product.

• Modifying src\tests\build.proj to publish the customer Crossgen2 to CORE_ROOT for the purpose of CoreCLR testing.

• Modifying library build scripts to provide more flexibility regarding framework and / or test compilation using Crossgen2 and passing additional flags to Crossgen2.

Open issues

• The proposed build restructuring goes largely against the current build pipeline (build clr + libs, then the packs). We need to figure out something that doesn't place additional burden on runtime developers (too many new switches or steps) and doesn't substantially regress local innerloop (runtime build perf)

• At this point it's unclear to me whether library testing with Crossgenned framework should also just grab the pack-produced framework package or basically duplicate the step (perhaps by just calling into the already existing SDK target).

[trylek]

/cc @dotnet/crossgen-contrib, @davidwrighton, @AntonLapounov, @jkotas, @jkoritzinsky, @ViktorHofer, @danmoseley, @stephentoub, @agocke. Just for visibility I'm also /cc-ing @SamMonoRT even though at this point I'm not aware of any particular Mono impact of this work.

[trylek]

Related to (precursor issue): #50127

[MichalStrehovsky]

Cc @LakshanF since moving where we AOT compile crossgen2 is similar to AOT compiling ILC that Lakshan has been looking at.

[davidwrighton]

I think we may be able to have a relatively minor variation to our test build and setup.

In particular, I think the following approaches may be viable:

1. We can probably use the Crossgen2 outerloop job to run these new test variations instead of modifying the normal test legs.
2. If we build the product.pack build, we'll have both the full Runtime+libraries compiled via Crossgen2 available as well as the final Crossgen2 product that we ship to customers.
3. When constructing the Libraries helix bits, it produces a "faux" product release to run the tests. We should be able to simply drop the final product instead into the Libraries tests.
4. Similarly, we should be able to drop a real crossgen2.exe into the CoreCLR tests, and convince our tests to use that.

I do not yet have a strategy for how to run crossgen2 on the actual libraries tests. I think we may just want to run crossgen2 directly before sending to helix.
5.

[trylek]

Thanks David, that is an interesting direction - while using the packs-produced Crossgen2 (and framework) for local CoreCLR / libraries testing may be challenging due to the build ordering issues, for lab testing it's a no-concern as we have all the dependencies readily available.

On the other hand, generally speaking we should strive to keep most lab chores reproducible locally for the purpose of bug investigation i.o.w. once we start running crossgenned library tests in the lab, we should have at least a non-trivial recipe for doing the same thing locally.

By non-trivial I mean that e.g. developers will newly need to build the packs along with clr+libs prior to running tests to make this work, otherwise in 99% of cases running the "raw" Crossgen2 should yield the same result as using the "nuget"-one and for normal developer innerloop that should be typically sufficient.

[jkotas]

I agree that one needs to be able to replicate the lab testing locally. It does not need to be as efficient as dev inner loop, it is fine to require building the whole repo for that.

In general, I think we should be making stronger distinction between dev inner loop where a dev iterates on a specific component or a specific test, and lab testing where we want to run all tests as efficiently as possible in number of configuration, including configurations that are identical to installed product.

[davidwrighton]

Yes. For instance, we could easily have another flag to the build script for the src\tests\build.cmd which could use the crossgen2 from the products pack. something like src\tests\build.cmd generatelayoutonly use_final_product_crossgen2 and that would simply pull the crossgen2 from the actual product build. Or possibly we could just have an environment variable which does the redirection. So the developer would need to know where the crossgen2 was produced, and then set something like set crossgen2_exec_command=c:\git\runtime\artifacts\path to crossgen2.exe

[trylek]

Adding a new option to src\tests\build.cmd/sh for using the "nuget" Crossgen2 sounds great to me and should be pretty easy to implement including some warning diagnostics in case the pack is not available ("You need to build packs.product prior to using the nuget-crossgen2 option"). The environment variable will ultimately report the same via a "file not found" exception but direct support in the test build script can presumably provide better developer-facing diagnostics.

[davidwrighton]

The bigger question is about running the libraries tests with official bits, and crossgening the libraries tests themselves. @danmoseley, do you have someone who we could talk to about various options around the libraries tests?

[trylek]

While I made the speculative syntax nuget-crossgen2 up, it's actually an interesting aspect as we could use the same or similar syntax to make CoreCLR test build grab the "nuget" version of the framework (per-assembly or composite). While this may sound nitpicky, CoreCLR test build uses the very non-standard internal app R2RTest to compile the framework (if it sounds like I'm trying to troll R2RTest author let me remind you it is myself) that may differ in many aspects from the way we're compiling the nuget framework. Naturally all of this holds for runtime testing that could be used as a first testbed for exercising the nuget Crossgen2 and framework, for library testing we'll need to massage the scripts much more.

[trylek]

In a way it's an interesting question whether, if we decide to add support for using the nuget framework for runtime testing, it makes any sense to keep the pre-existing logic of custom crossgenning of the framework using R2RTest - my gut feeling is that two different versions of R2R framework would do more harm than good and we should just use the one we ultimately ship.

[trylek]

And David's question for Dan made me realize that we can already start actually building up an ordering of tasks:

1. Add options to src/tests/build.cmd/sh/proj to use nuget Crossgen2 / nuget framework instead of using raw Crossgen2 and manually compiled framework. Switch over all Crossgen2 pipelines to use this new logic. This seems well understood, I would estimate it at 1/2-1 workweek and in my opinion @ivdiazsa would be a great person to implement that based on his past experience but I certainly don't insist.

2. Add options to library tests to use the nuget framework instead of the raw one (per-assembly or composite). Switch [at least some] library test runs to use this option. This is less understood in terms of actual script changes but still relatively well defined. I would expect this to take 1/2-1 workweek for someone familiar with library test build.

3. Add options to library tests to use the nuget Crossgen2 to compile the test apps themselves. Switch [at least some] library test runs to use this option. This is hand-wavy but hopefully doable in something like a workweek by someone familiar with library test build assisted by a Core Runtime team member familiar with Crossgen2.