Redesign the subset feature

[ViktorHofer]

• Combine the -subset and -subsetcategory switches into just -subset. This probably involves making subset names distinct which IMO is a good idea anyway as we should remove the sub repositories (another case of that is removing the helper scripts for coreclr, libraries, mono, installer). Specifying -subset libraries should be sufficient to build all default subsets for libraries. No need for the extra -subsetcategory verb.
• Allow adding to the default set of subsets without the need to specify all of them, ie. -subset libs+libs.tests.
• Allow removing from the default set of subsets without needing to specify all of them...

Moved proposal up

Proposal

You can specify multiple subsets. clr, mono, libs and installer imply the child subsets.

clr
 clr.runtime
 linuxdac
 clr.corelib
 clr.corelibnative
 clr.tools
 clr.packages
mono
 mono.runtime
 mono.corelib
 mono.packages
libs
 libs.native
 libs.ref
 libs.src
 libs.pretest
 libs.tests
 libs.packages
installer
 corehost
 installer.managed
 installer.depproj
 installer.packages
 bundles
 installers
 installer.tests

Examples

1. Build libraries and coreclr
   build.cmd/sh libs+clr

2. Build libraries tests:
   build.cmd/sh libs.tests

3. Build mono corelib and libraries
   build.cmd/sh mono.corelib+libs

4. Build libraries tests and installer tests
   build.cmd/sh installer.tests+libs.tests

5. Build coreclr, libraries and libraries tests
   build.cmd/sh clr libs+libs.tests

cc @dotnet/runtime-infrastructure @jkotas @danmosemsft @stephentoub

Area Owners

@ViktorHofer, @jeffschwMSFT, @dleeapho

[jkotas]

It would be nice to drop the need for the -subset prefix. I would like to be able to just type build libraries.

[safern]

Something to consider while designing this is how are we going to identify in between different subsets when they clash. For example, if at some point in mono we could build only corelib, how will I be able to build mono corelib if I just say, build -subset corelib, that would be ambiguous if both mono and coreclr have a corelib subset, so we might need to append mono or coreclr to it. Something like, -subset coreclrruntime or coreclrcorelib, and if we want to build all coreclr, you would just say, subset coreclr.

[ViktorHofer]

Something to consider while designing this is how are we going to identify in between different subsets when they clash.

Having unique subset ids make most sense IMO. We shouldn't need to append the sub repo for all of them, ie only installers build installers (lol). What I'm doing in #33553 is naming the libraries test subset, which we haven't had before, libtests.

It would be nice to drop the need for the -subset prefix. I would like to be able to just type build libraries.

For the subset I think that makes sense, for other arguments I would like to keep the current syntax which is -<option-name> <option-value> primarily to stay consistent with other repositories. One thing to keep in mind here is that we already overload the "non-option" value (which can occur at any position) to build individual libraries: https://github.com/dotnet/runtime/blob/master/eng/build.ps1#L125-L138. We could change this to require -projects option which already is an Arcade standardized one. As not many libraries devs us that feature I would be ok with changing that. Of course there are other ways to solve this (ie positional, hardcoded checks, ...).

Based on #33821, the earliest we can get this in is start of May.

[safern]

Having unique subset ids make most sense IMO.

I agree, and that is why I brought that up. So how would we do it for the scenario I posted above, could you provide an example of what you're envisioning?

For the subset I think that makes sense, for other arguments I would like to keep the current syntax which is - primarily to stay consistent with other repositories.

https://github.com/dotnet/runtime/blob/master/eng/build.ps1#L125-L138. We could change this to require -projects option which already is an Arcade standardized one.

+1 on those two statements.

[BruceForstall]

Related: #33167

[BruceForstall]

So we have top-level project subsets (coreclr, libraries, installer, mono) and each has a set of their own subsets (sub-subsets?); for the latter the names only need to be unique within their subset, right? So, e.g., "native" in "coreclr" and "native" in "libraries" mean something different, and don't conflict.

Keeping something similar to what we have today, maybe you should be able to:

build coreclr:runtime+corelib+nativecorelib libraries

where you specify "coreclr" subsets, but build whatever the "libraries" default is.

[danmoseley]

I liked coreclr build.cmd where I could type eg build -windowmscorlib -linuxmscorlib

Rather than worrying about subsets, why not just have a list of workloads? It could be a long list, if it's logically organized. Then just list off the ones you want eg build coreclr coreclrcorelib or build monoruntime monocorelib libraries ?

[ViktorHofer]

Rather than worrying about subsets, why not just have a list of workloads? It could be a long list, if it's logically organized. Then just list off the ones you want eg build coreclr coreclrcorelib or build monoruntime monocorelib libraries ?

Workloads are the same thing as subsets. What you are proposing is the same that I was thinking of.

Keeping something similar to what we have today, maybe you should be able to:

I would like to get rid of the sub-repository boundaries in the subsets/workloads.

That's a draft proposal:

Moved proposal into top post

[trylek]

Looks great to me overall, I'm just wondering whether some of the longer combined words (like coreclrnativecorelib) aren't a bit hard to read/write (even though I guess you're well trained for that with your German language background ;-)) - any chance you'd be willing to consider using some word separator like coreclr-native-corelib, coreclr.native.corelib or coreclr/native/corelib?

[ViktorHofer]

That's a great idea, let me update the proposal.

[ViktorHofer]

Added an example above of how we would do an exclusion with the - sign.

[safern]

coreclr-corelib-native

Maybe that one should just be coreclr-corelibnative?