Currently we run ML.NET and Roslyn benchmarks in this repo. They use pinned versions of those products and test against live .NET Core bits. This is a great defense mechanism for ensuring future .NET Core products don't have regressions. It does not help us track the performance of ML.NET and Roslyn as they change, however. We'd like to do that as well.

The challenge is that in a given perf run we'd like there to be a single variable. Moving both the underlying .NET Core version as well as the library version makes it very hard to reason about what caused a regression. This is complicated by the fact that it is reasonable to assume that future versions of libraries and tools will take dependencies on new .NET Core features in prerelased versions. We can't expect a given Roslyn build to just work on the previously RTMed .NET Core.

We need a scheme that allows for:

• Rolling forward the non-platform bits frequently
• Pinning a .NET Core platform version as long as possible
• Has a mechanism for moving the pinned version with fixed versions of the other libraries

There may be other considerations but those are the critical ones to me.