Experiment with adding source-build infrastructure to a repo with Arcade

[dagood]

This issue tracks doing some exploratory work on moving the low level infra into Arcade, in a way that feeds into our goal of validating source-build and producing a tarball during the Microsoft product build and repo PR validation.

The main topics I want to investigate:

• Adding source-build functionality to the Arcade SDK.
  • Can we jump to the point where instead of cloning source-build, clone the target repo (Core-SDK) at the desired commit and do some "build from source" or "create a tarball for me" gesture?
• Taking inputs in a way that's naturally compatible with Darc/Maestro++/BAR.
• Packaging outputs in a way that can be cached and used in downstream source-builds.
• Maintaining a potential way to have per-repo patches.
  • Ideally the Microsoft build will respin when source-build doesn't work. However, can we still add/maintain patches if we're unable to do this? Is there an analogous flow for emergencies?

---

Notes:

• Having source-build's darc clone manually manipulate .git/modules to store Git data is probably ok to ensure git clean -xdf doesn't make you have to redo a ton of cloning time, however I don't think it's reasonable to roll that out to users that don't normally interact with source-build due to large size and cleanup difficulty.
• Similar to signing, build, etc., Arcade can implement the basics, but repos need to add eng/SourceBuild.props/targets for the custom stuff we're handling in repos/*.proj today.
• Patching could be represented by a separate branch in the repo. We can force the commit hash to match the official build, if need be with something like git checkout --detach; git reset --soft <official>. This is effectively the same as a patch, but raises awareness and is more intuitive to work with as a repo contributor.
• darc clone doesn't quite do what we need here. Right now, source-build has a dependency for every single repo (recursively) necessary to source-build, and relies on auto-updates to "fix up" the info based on coherent parent dependencies. We can copy these dependencies to the target repo, however that's a significant amount of unnecessary info. darc clone should ideally be able to figure it out instead.
• The "bootstrap Arcade" is the version that the repo depends on. While source-building the repo, you can download a source-built Arcade from intermediates, and that works fine for the upstream repos, however for the current repo it's too late to switch the version of the tooling (MSBuild, SDK, Arcade) being used at that point because we're already running the process. We need some workaround: recursive build?
• We may want to consider doing some housekeeping on the source-build side to make it easier to push the way source-build orchestrates things into Core-SDK/Arcade. In my prototype, there are some common themes: stuff is running without the Arcade SDK, so some properties aren't available; artifacts are mostly in bin, not artifacts, so it's not trivial to redirect; there is a lot of old code; and some info is set up in the build scripts rather than in MSBuild logic. The work would also likely be trivial to apply to 3.1 servicing to help out there. (There are also less trivial things like incrementality and isolation that could be improved.)
• global.json resolution throws a wrench in some things. For example, when using Arcade's Build.proj, that's located in the package cache. I think that therefore, the SDK looks for the global.json in the hierarchy of the package cache, so it matters where it's located. What I observed was that two versions of Arcade are used at once in this scenario, which causes problems.
  • I added a target that copies the global.json into the package cache dir before each subrepo build for now, and we can probably use the source-build SDK resolver to forcefully resolve this situation if necessary.

[dagood]

I hit a decent milestone and took a snapshot branch:
dotnet/installer@v3.1.102...dagood:sb-proto-sb-in-eng

This command builds upstreams using the mostly ordinary source-build infra I copied into eng/source-build, then builds Core-SDK using those assets (including source-built Arcade):

./build.sh -c Release /p:FromSource=true /p:BuildUpstreamRepos=true /bl:artifacts/msbuild.binlog /nr:false /p:KeepUpstreamGitDataInDotGit=true

Source-built outputs end up in artifacts/source-build/repo/packages/Release/Shipping.

The Core-SDK-side source-build is the most interesting bit for this prototype, and I have the targets somewhat separated into different eng/SourceBuild*.[props/targets] files. All the pre-build infra is driven through eng/Build.props. I think this approach could be migrated to Arcade cleanly.

To get the source-built SDK loaded, it uses the recursive build approach I have a discussion thread about at dotnet/arcade#5116. It's similar to (1) at https://github.com/dagood/source-build/blob/sb-in-official-doc/Documentation/planning/arcade-powered-source-build/in-arcade.md#supporting-source-built-sdks but doesn't require recloning the entire Core-SDK repo (the current checked out sources are used).

A scenario that might still call for a fresh clone is if the repo doesn't properly put its intermediates in artifacts/ (pollutes the source dir) and a dev expects source-build to work from an unclean repo state.

---

The next big rock I'm looking at to let us actually reasonably merge this stuff into Core-SDK is teaching darc clone how to clone the dependency graph with forced coherency. (#1495.) The prototype copies the hand-crafted expanded graph from source-build into core-sdk, which would not be good to actually use for 5.0.

The last big thing looming that hasn't been prototyped yet is the intermediate nupkg structure and infra. ("Packaging outputs in a way that can be cached and used in downstream source-builds" above.) This is key for reducing build time so source-build can reasonably happen during validation builds.

[dagood]

I'm going to close this issue as the Core-SDK prototype is completed, and track remaining work (some I'm working on now) in new issues:

• teaching darc clone how to clone the dependency graph with forced coherency. (Add forced coherency ability to "darc clone" rather than maintaining a dependency for every repo #1495.)
• the intermediate nupkg structure and infra: Create proof of concept: the source-build intermediate nupkg format #1543