Design for multi-band support in source-build

[mmitche]

No description provided.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[mmitche]

I don't know if this is the final home for this doc, but it seems as good a place as any to review it.

[mmitche]

Major changes made:

• Split --with-packages into two switches, defining "CSB" and "PSB" artifacts.
• Fixed up the distro maintainer workflows.

[tkapin]

I understand why this uncertainty exists, but I'm curious why we don't have more strict rules on dependencies of SDK components on SDK features that would make this behavior predictable. What would be the implications of requesting the product to be always buildable with the previous SDK bands?

[mmitche]

The SDK bands are driven by VS development, which moves forward between .NET releases. Those components of the .NET SDK need to introduce new functionality, which may require support from the SDK.

The implication of requiring the SDK bands to be always buildable with earlier bands is that you couldn't introduce some new features. In addition, it wouldn't improve distro maintainer workflow by much. You're still building the 1xx and Nxx bands. Whether you depend on the Nxx band to build the Nxx band or not is only a tiny difference.

[tkapin]

The implication of requiring the SDK bands to be always buildable with earlier bands is that you couldn't introduce some new features.

I'm still not comfortable with this statement. In my thinking, we are mixing together two things - introducing an SDK feature (that is later used by the corresponding VS version) and using this feature to build the SDK itself (as can happen when a project shipping in the given SDK band takes dependency on the latest features introduced in that given SDK band). It's clear the compiler and other tooling want to dogfood the latest features they introduce aggressively, but I still don't see a case where it wouldn't be possible to introduce a set of new features solely through the features available in existing version of the SDK.

[ayakael]

I agree with @tkapin. In an ideal world, every feature band should be buildable using 1xx. But at minimum, it should be buildable using the previous band. Anything else breaks N+1.

An ideal packaging approach for me would be to have a package of dotnet that builds 1xx, and a package of dotnet that builds the latest feature band, using the artifacts of 1xx. Considering that this isn't on the table, I can imagine then packaging 2xx sdk to build 3xx sdk once that's released.

[mmitche]

I don't think we can make that guarantee. I think it may work in most cases, but in others, a bootstrap of the new band will be required. Even if we get an agreement from VS that they would not break this workflow, new functionality is being introduced in those bands, some of which may require SDK support, and so bugs are possible.

In conclusion,

We can always say (barring catastrophic problems), that 200 GA will build 201, will build 202, etc. This is the same garuantee we make with the 1xx band+runtime today. It just may be that a 200 bootstrap will be required to build the initial 200.

[mmitche]

I clarified the logic here. Can you take a look once I push the latest update?

[MichaelSimons]

@dotnet/distro-maintainers - please take a look and provide feedback.

[tmds]

Do we need an additional argument that allows passing the prebuilts packages tarball?

My understanding it contains packages needed for a successive build to work, which weren't yet buildable from source. So it we should be able to pass it to a build?

[mmitche]

That's info is currently encoded into the prep script, and so I don't think a new workflow or additional switch would be required.

By the time we have multiple bands (after RTM), we should also never have prebuilts

[tmds]

That's info is currently encoded into the prep script

I have some CI jobs that build a packages dir from a previous build's Artifacts and Prebuilts and then pass that as --with-packages to a new build.

It seems I need some way to pass the in Prebuilts when --with-packages gets removed?

I assume I can use one of these properties to pass the Artifacts, and the other to pass Prebuilts, and that should work?

[mmitche]

I have some CI jobs that build a packages dir from a previous build's Artifacts and Prebuilts and then pass that as --with-packages to a new build.

--with-previous-artifacts will be functionally the same as --with-packages. So you could use that same workflow. --with-current-artifacts would only be used for building newer bands, passing the artifacts built in the 1xx band.

[tmds]

--with-previous-artifacts will be functionally the same as --with-packages. So you could use that same workflow. --with-current-artifacts would only be used for building newer bands, passing the artifacts built in the 1xx band.

If these arguments would accept tarballs (or even tarball urls) it would be handy to not have to extract both tarballs into a single directory to pass it to --with-previous-artifacts.

Question: will a build in some way functionally distinguish between packages from --with-previous-artifacts vs --with-current-artifacts, or will both be 'simply' treated a source of packages (maybe with a preference for --with-current-artifacts over --with-previous-artifacts)?

[mmitche]

The individual component builds do not distinguish. Just another source. When deciding what versions of a various package to use, current takes precedence over previous. This is the same as today. What just got built (e.g. when building runtime), takes precedence over previously source built.

[mmitche]

Another way to view this whole effort is that there already is such a concept of "current-artifacts" within the infrastructure today. A shared directory where the outputs of each repo is placed as it is built.

flowchart TD
  previouslysourcebuilt[("previousy source built (--with-packages/prep.sh)")]
  currentsourcebuilt[("current live source built (built in current build invocation)")]
  startbuild["start build"]-.->arcade["Arcade build"]-.->aspnetcore["aspnetcore build"]-.->sdk["sdk build"]-.->preparefinaloutputs["gather source built artifacts"]-.->finishbuid["finish build"]
  previouslysourcebuilt-->arcade
  previouslysourcebuilt-->aspnetcore
  previouslysourcebuilt-->sdk
  currentsourcebuilt-->aspnetcore
  currentsourcebuilt-->sdk
  arcade-->currentsourcebuilt
  aspnetcore-->currentsourcebuilt
  sdk-->currentsourcebuilt
  currentsourcebuilt-->preparefinaloutputs

  subgraph versionselection["version selection"]
    current["current source built version"]-->|overrides|previous["previously source built version"]-->|overrides|repoCheckedInDefault["repo checked in eng/Versions.props"]
  end

Loading

What this proposal is doing is extending this concept of another set of "current" artifacts that come from another build invocation.

flowchart TD
  previouslysourcebuilt[("previousy source built (--with-packages/prep.sh)")]
  currentlivesourcebuilt[("current source built (built in current build invocation)")]
  currentsourcebuilt[("current source built (--with-current-packages)")]
  startbuild["start build"]-.->roslyn["roslyn build"]-.->sdk["sdk build"]-.->preparefinaloutputs["gather source built artifacts"]-.->finishbuid["finish build"]
  previouslysourcebuilt-->roslyn
  previouslysourcebuilt-->sdk
  currentsourcebuilt-->roslyn
  currentlivesourcebuilt-->sdk
  currentsourcebuilt-->sdk
  roslyn-->currentlivesourcebuilt
  sdk-->currentlivesourcebuilt
  currentlivesourcebuilt-->preparefinaloutputs

  subgraph versionselection["version selection"]
    currentlive["current live source built version"]-->|overrides|current["current source built version"]-->|overrides|previous["previously source built version"]-->|overrides|repoCheckedInDefault["repo checked in eng/Versions.props"]
  end

Loading

[mmitche]

Above, the dotted lines represent the execution flow, while solid lines represent the data flow. Hopefully that makes it a little clearer.

[mmitche]

I added this as an index.