Add ci channel for per-commit Build Cache Service framework resolution

docs/build_cache_requirements.md

@@ -0,0 +1,129 @@
+# Build Cache Service — Requirements for Crank Integration
+
+## Context
+
+Crank (the .NET benchmarking tool) has been updated to support a new `ci` channel that downloads pre-built binaries from the Build Cache Service (BCS) instead of resolving versions from VMR/NuGet feeds. On this channel crank overrides **both** the base runtime (`Microsoft.NETCore.App`, from dotnet/runtime) and the ASP.NET Core shared framework (`Microsoft.AspNetCore.App`, from dotnet/aspnetcore), each resolved independently by commit (the commit is passed via the `runtimeVersion` / `aspNetCoreVersion` arguments). This gives per-commit granularity for performance testing and regression bisection.
+
+The crank-side changes are complete. This document describes what's needed on the BCS/dotnet-performance-infra side to make the integration work end-to-end.
+
+---
+
+## Requirement 1: Public Blob Access
+
+**Status:** Already in progress (per prior discussion).
+
+Crank's BCS client uses unauthenticated HTTP GET requests to download artifacts. The blobs in the `pvscmdupload` storage account's `$web` container need to be publicly readable.
+
+**URLs crank will hit:**
+
+```
+GET https://pvscmdupload.z22.web.core.windows.net/builds/{repoName}/latest/{branch}/latestBuilds.json
+GET https://pvscmdupload.z22.web.core.windows.net/builds/{repoName}/buildArtifacts/{commitSha}/{configKey}/{artifactFile}
+```
+
+Where:
+- `repoName` = `runtime` and `aspnetcore` (both are resolved on every `ci`-channel job; each hit uses its own repo segment)
+- `branch` = e.g., `main`, `release/10.0`
+- `configKey` = e.g., `coreclr_x64_linux`, `coreclr_arm64_windows` (runtime); `aspnetcore_x64_linux`, `aspnetcore_arm64_windows` (aspnetcore)
+- `artifactFile` = e.g., `BuildArtifacts_linux_x64_Release_coreclr.tar.gz` (runtime); `BuildArtifacts_linux_x64_Release_aspnetcore.nupkg` (aspnetcore — the verbatim runtime-pack nupkg)
+
+The aspnetcore `latestBuilds.json` lives at `builds/aspnetcore/latest/{branch}/latestBuilds.json` and contains only the 5 `aspnetcore_*` config keys plus an `all` entry and `branch_name`; it does **not** carry the runtime `coreclr_*` keys. Crank's parser enumerates keys dynamically, so it tolerates either repo's file.
+
+---
+
+## Requirement 2: Commit Index File (Not Required)
+
+~~Originally proposed as a per-branch `commitIndex.json` mapping commits to timestamps.~~
+
+**Decision:** Not needed. For the default case, `latestBuilds.json` provides the latest commit. For specific-commit runs (e.g., bisection), users will already know the SHAs — either from git history, GitHub, or a local tool that queries the GitHub API for the commit list. A separate index in BCS would be redundant.
+
+If automated bisection tooling is built in the future, it can query GitHub directly for ordered commit SHAs and then check BCS blob existence per-commit.
+
+---
+
+## Requirement 3: latestBuilds.json Compatibility
+
+**Resolved.** The actual `latestBuilds.json` uses PascalCase (`CommitSha`, `CommitTime`), not snake_case. Crank's parser has been updated to accept both casings for forward compatibility.
+
+---
+
+## Requirement 4: Artifact Layout Stability
+
+Crank extracts **runtime** artifacts using this path convention inside the archive:
+
+```
+microsoft.netcore.app.runtime.{rid}/Release/runtimes/{rid}/lib/net{X}.0/  → managed DLLs
+microsoft.netcore.app.runtime.{rid}/Release/runtimes/{rid}/native/        → native libs
+{rid}.Release/corehost/                                                    → host binaries (dotnet, libhostfxr, libhostpolicy)
+```
+
+For **aspnetcore** artifacts the stored blob is the **verbatim runtime-pack nupkg** (a zip,
+extension `.nupkg`), so the layout is the nupkg's own — `runtimes/{rid}` sits at the archive root,
+with no `microsoft.aspnetcore.app.runtime.{rid}/Release` wrapper. Crucially, the verbatim nupkg
+carries the host-resolvable framework metadata next to the managed assemblies:
+
+```
+runtimes/{rid}/lib/net{X}.0/Microsoft.AspNetCore.*.dll                  → managed assemblies
+runtimes/{rid}/lib/net{X}.0/Microsoft.AspNetCore.App.deps.json          → host-resolvable metadata (REQUIRED)
+runtimes/{rid}/lib/net{X}.0/Microsoft.AspNetCore.App.runtimeconfig.json → host-resolvable metadata (REQUIRED)
+runtimes/{rid}/native/                                                  → native libs (optional)
+```
+
+Because the nupkg is a complete framework, crank **places `Microsoft.AspNetCore.App` directly** from
+the pack into the per-job dotnet home (the whole managed set incl `deps.json`/`runtimeconfig.json`,
+no feed contribution) and **fails the job** (`BuildCacheIncompleteException`) if the pack is missing
+managed assemblies, `deps.json`, or `runtimeconfig.json` — for perf runs, erroring is preferable to
+silently running mixed/feed bits. The base runtime + host stay feed-resolved (the aspnetcore pack
+ships neither). Self-contained (SCD) publishes are the exception: the framework is co-mingled with
+the app under the app's own `.deps.json`, so for SCD crank overlays only the managed `*.dll`
+(+ native), not the framework metadata.
+
+This differs from **runtime**, whose archive is raw build output (no shared-framework
+`deps.json`/`runtimeconfig.json`), so the runtime flavour overlays BCS binaries onto a feed-installed
+runtime (reusing the feed's metadata) rather than placing directly.
+
+Where `{rid}` = `linux-x64`, `linux-arm64`, `win-x64`, `win-arm64`, `win-x86`. (aspnetcore v1 has no
+musl/osx/arm32 configs.)
+
+The runtime layout was confirmed by inspecting `BuildArtifacts_linux_x64_Release_coreclr.tar.gz` (no
+framework metadata in the pack lib). The aspnetcore contract — a verbatim runtime-pack nupkg carrying
+managed + `deps.json` + `runtimeconfig.json`, uploaded as `BuildArtifacts_{os}_{arch}_Release_aspnetcore.nupkg`
+— was confirmed against a locally-built `Microsoft.AspNetCore.App.Runtime.win-x64.nupkg` and
+dotnet/performance#5243's `stage-bcs-nupkg-aspnetcore.ps1`.
+**If either layout changes in future builds, the crank extraction will break.** Consider treating it
+as a stable contract or documenting it.
+
+---
+
+## Nice-to-Have: Artifact Manifest
+
+A `manifest.json` per commit+config that describes the archive contents would make extraction more robust:
+
+```
+builds/{repoName}/buildArtifacts/{commitSha}/{configKey}/manifest.json
+```
+
+```json
+{
+  "runtimeVersion": "10.0.0-preview.4.26120.3",
+  "commitSha": "abc123...",
+  "rid": "linux-arm64",
+  "managedPath": "microsoft.netcore.app.runtime.linux-arm64/Release/runtimes/linux-arm64/lib/net10.0",
+  "nativePath": "microsoft.netcore.app.runtime.linux-arm64/Release/runtimes/linux-arm64/native",
+  "corehostPath": "linux-arm64.Release/corehost"
+}
+```
+
+This isn't blocking — crank currently discovers paths by convention — but it would decouple crank from the internal archive layout and make future changes safe.
+
+---
+
+## Summary
+
+| # | Requirement | Priority | Blocking? |
+|---|-------------|----------|-----------|
+| 1 | Public blob access | High | Yes — crank can't download without it |
+| 2 | ~~Commit index~~ | N/A | Dropped — users provide SHAs directly or use GitHub |
+| 3 | `latestBuilds.json` field names | N/A | Resolved — crank parser updated to handle PascalCase |
+| 4 | Artifact layout stability | Medium | Not now, but breaking changes would break crank |
+| 5 | Artifact manifest.json | Low | Nice-to-have for robustness |

docs/dotnet_versions.md

@@ -55,9 +55,12 @@ When a TFM is configured, the agent will download the corresponding .NET SDK ver
 - `current`: only latest public versions, this is the default
 - `latest`: latest versions used by ASP.NET 
 - `edge`: latest nightly builds available
+- `ci`: base runtime + ASP.NET Core from the Build Cache Service, resolved per-commit (see below)
 
 The difference between `latest` and `edge` is that `latest` will pick runtimes and SDKs that are deemed compatible together. For instance a very recent .NET core runtime might be compatible with a less recent ASP.NET runtime. The `edge` is used to pick the absolute latest build for the select TFM.
 
+The `ci` channel uses the Build Cache Service (BCS) from `dotnet-performance-infra` to resolve framework versions by individual commit SHA rather than from VMR feeds. This provides much finer-grained control — every cached commit is available, whereas VMR feeds may have multi-day gaps between ingested commits. On this channel crank overrides **both** the base .NET runtime (`Microsoft.NETCore.App`, from dotnet/runtime) **and** the ASP.NET Core shared framework (`Microsoft.AspNetCore.App`, from dotnet/aspnetcore); each defaults to the latest cached build and can be pinned independently by passing a commit SHA in `runtimeVersion` / `aspNetCoreVersion`. SDK and desktop versions are resolved from `latest`.
+
 In order to benchmark and ASP.NET application using very recent runtimes of .NET 5, the `latest` channel is recommended:
 
 ```
@@ -115,4 +118,63 @@ The following command uses the `edge` channel but ASP.NET is fixed so it doesn't
 
 ```
 > crank --config /crank/samples/hello/hello.benchmarks.yml --scenario hello --profile local --application.framework netcoreapp5.0 --application.channel edge --application.aspnetCoreVersion 5.0.0-preview.6.20279.12
-```
+```
+
+## Using the ci channel
+
+The `ci` channel resolves pre-built binaries for individual commits from the Build Cache Service (BCS). This is useful for performance regression bisection where VMR feed gaps make it hard to pinpoint which commit caused a regression.
+
+On this channel crank always overrides **both** frameworks, each resolved from its own repository:
+
+- **Base runtime** (`Microsoft.NETCore.App`) is **overlaid** with BCS bits built from a [dotnet/runtime](https://github.com/dotnet/runtime) commit. The runtime archive is raw build output (no shared-framework metadata), so BCS binaries are overlaid onto a feed-installed runtime.
+- **ASP.NET Core shared framework** (`Microsoft.AspNetCore.App`) is **placed directly** from a [dotnet/aspnetcore](https://github.com/dotnet/aspnetcore) commit's BCS build. The aspnetcore archive is the runtime-pack nupkg stored verbatim (carrying `deps.json` + `runtimeconfig.json`), so the framework folder is built entirely from BCS and the job **fails** if the pack is incomplete.
+
+Each repository resolves **independently**: by default both use the latest cached build on `main`. On the `ci` channel the existing `runtimeVersion` and `aspNetCoreVersion` arguments carry a **commit SHA** rather than a feed version — supply a SHA to pin/bisect one repo while the other stays latest, or leave it empty to use the latest cached build. The *latest* lookup always targets `main` (the pipeline only builds `main`).
+
+> **Note:** On the `ci` channel, `runtimeVersion` / `aspNetCoreVersion` accept **only** a commit SHA (8–40 hex characters) or an empty value (= latest cached build). A feed version string is **rejected with an error** — version-string pinning is not supported on this channel. Conversely, on the other channels (`current` / `latest` / `edge`) those same arguments accept a version string and do **not** accept a commit SHA. The reported `runtimeVersion` / `aspNetCoreVersion` for a `ci` run are stamped with the resolved commit as `{feedVersion}+ci.{shortSha}`.
+
+### Basic usage (latest cached build of both frameworks on main)
+
+```
+> crank --config benchmarks.yml --scenario json --profile aspnet-perf-lin --application.channel ci
+```
+
+### Bisecting ASP.NET Core (pin aspnetcore, runtime stays latest)
+
+```
+> crank --config benchmarks.yml --scenario json --profile aspnet-perf-lin --application.channel ci --application.aspNetCoreVersion a1b2c3d4e5f6...
+```
+
+### Bisecting the base runtime (pin runtime, aspnetcore stays latest)
+
+```
+> crank --config benchmarks.yml --scenario json --profile aspnet-perf-lin --application.channel ci --application.runtimeVersion a1b2c3d4e5f6...
+```
+
+### Pinning both
+
+```
+> crank --config benchmarks.yml --scenario json --profile aspnet-perf-lin --application.channel ci \
+    --application.runtimeVersion 1111aaaa2222bbbb... \
+    --application.aspNetCoreVersion 3333cccc4444dddd...
+```
+
+If a requested commit is not found in the cache, crank fails with an error rather than falling back.
+
+### `ci` channel properties
+
+| Property | Default | Description |
+|----------|---------|-------------|
+| `runtimeVersion` | (empty = latest) | On the `ci` channel: a [dotnet/runtime](https://github.com/dotnet/runtime) commit SHA (8–40 hex chars) to resolve from BCS and overlay onto `Microsoft.NETCore.App`. Empty uses the latest cached runtime build on `main`. A feed version string is rejected with an error. |
+| `aspNetCoreVersion` | (empty = latest) | On the `ci` channel: a [dotnet/aspnetcore](https://github.com/dotnet/aspnetcore) commit SHA (8–40 hex chars) to resolve from BCS and place as `Microsoft.AspNetCore.App`. Empty uses the latest cached aspnetcore build on `main`. A feed version string is rejected with an error. |
+
+The BCS configuration key (e.g., `coreclr_x64_linux` for runtime, `aspnetcore_x64_linux` for aspnetcore) is auto-detected per repo from the agent platform. Platforms with no aspnetcore config (there is no macOS/musl/arm32 in v1) fail loud rather than silently skipping.
+
+### Agent configuration
+
+The agent supports these command-line options for BCS:
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--build-cache-base-url` | `https://pvscmdupload.z22.web.core.windows.net` | Base URL for BCS blob storage. |
+| `--build-cache-disabled` | (not set) | Disables BCS integration on this agent. |