Fix MSBuild.exe telemetry delivery in headless CI

[YuliiaKovalova]

Context

MSBuild.exe (.NET Framework, Visual Studio / Build Tools) sent no telemetry from headless CI. Two confirmed causes:

1. Missing dependencies. The VS payload didn't ship Microsoft.VisualStudio.Telemetry.dll or Microsoft.VisualStudio.RemoteControl.dll, and the amd64/arm64 MSBuild.exe.config couldn't locate them. Session creation threw, the exception was swallowed, and nothing was reported.
2. No bounded delivery. TelemetrySession.Dispose() blocks on the SDK's own internal send timeout, which is long and not CI-aware. An ephemeral agent can exit before it completes.

Changes Made

Keeps Microsoft.VisualStudio.Telemetry and the existing destination, consent, and VS/MSBuild/Build event. Adopts the .NET SDK's design ideas (the host owns the session, CI-aware upload, bounded shutdown) but not its OpenTelemetry stack, which would change the destination and add a new .NET Framework dependency closure.

• Deployment: files.swr ships the two missing DLLs. app.amd64.config (used by amd64 and arm64) adds codeBase entries for them and for Utilities.Internal. x86 loads them from its own folder.
• Ownership: the process that builds in-process owns the session: the entry process creates it right before its build, and the server node when it starts. Worker, task-host and RAR nodes don't create one, and the thin client creates one only if it falls back to building in-process. The node mode is read from the command line (/nodemode:N, -nodemode:N, --nodemode:N, and the nmode alias). Inside VS, MSBuild uses the VS session and never disposes it.
• Crashes: every MSBuild.exe process reports its own crash. A process that has no session (a worker, task host or RAR node, or the thin client) creates one just for the crash, bounded by the shutdown budget. In a process that MSBuild is hosted in, such as VS, the host's session is used if there is one.
• Bounded delivery: on exit, CI uploads with DisposeToNetworkAsync, only if the process started events and the session is opted in; outside CI, events are saved as before. Both are capped at 10 s in total (MSBUILD_TELEMETRY_SHUTDOWN_TIMEOUT_MS). The upload gets 80% of the budget. If it fails, or doesn't finish in time and is canceled, the events are saved locally in the rest of the budget, so that a later process can upload them. A second shutdown request waits for the first one within the remaining budget, and the server node's Ctrl+C handler shuts telemetry down before Environment.Exit(0). Telemetry never changes the exit code.
• CI attribution: BuildEngineHost reports Azure DevOps / GitHub Action. Precedence: VS > MSBUILD_HOST_NAME > CI > VS Code.
• Consent: on .NET Framework, MSBUILD_TELEMETRY_OPTOUT and DOTNET_CLI_TELEMETRY_OPTOUT both opt out, in MSBuild.exe and inside Visual Studio, so setting the standard .NET variable is enough. This deliberately changes the behavior inside Visual Studio: before, only MSBUILD_TELEMETRY_OPTOUT applied there. When opted out, no session is created and no MSBuild events are sent. The VS opt-in is still honored, and detecting CI never opts anyone in.
• Diagnostics: MSBUILD_TELEMETRY_DIAGNOSTICS=1 writes initialization, opt-out, failure, and shutdown status to stderr.
• Privacy: crash events and the VS FaultEvent no longer carry exception messages. This deliberately changes the crash event contract. Hang diagnostics identify the project of an active node only by the hash of its file name, without its directory. They report a logger type by name when it is in a Microsoft. namespace, and by the hash of its name otherwise. The BuildCheck custom-check loading failure message has its paths removed, is truncated, and is hashed.
• Docs: VS-Telemetry-Data.md gains a flowchart of what each process sends and how, a table of which process owns a session, and sections on crashes, consent, and delivery.

Testing

The build passes with no warnings. Focused telemetry tests pass on net48 and net11.0, including CI/non-CI shutdown selection, the bounded timeout, and both opt-out variables. A headless E2E harness ran x86/x64 MSBuild.exe from the built VSIX layout (exactly what VS setup installs), simulating CI through a local proxy, with one approved run against the production collector: events arrived with the correct BuildEngineHost/BuildSuccess, no duplicates across multiprocess/server/node-reuse builds, shutdown capped at 10 s under a stalled network, and opt-out / missing-dependency cases correctly disabled telemetry without affecting the build. Not tested: ARM64 at runtime, a clean Build Tools image, real Azure DevOps/GitHub Actions agents.

Review follow-up: new tests cover when the upload happens (CI, events started, opted in), the cancel-then-save fallback and the shared shutdown budget, the crash-only session in MSBuild.exe versus hosted processes, node mode parsing, the hashing of custom logger type names, project file names, and the BuildCheck message, and the plain-text reporting of Microsoft logger types. The key tests were checked to fail when the code they guard is broken. The built MSBuild.exe was also run with the opt-out set and MSBUILD_TELEMETRY_DIAGNOSTICS=1, so that nothing was sent: the entry process and the server node reach telemetry initialization, and the worker node and the thin client (while the server builds) don't. Not re-run after these changes: the proxy and production-collector E2E harness.

Known limitations and follow-ups

• A server node keeps its session between builds, and a live session can't be flushed on demand: the VS telemetry library has no public flush, and DisposeToNetworkAsync ends the session. Its events reach the local store about every 30 s, are uploaded by the process that holds the upload lock, and are saved when the node exits, so an upload right after each build isn't guaranteed. Possible follow-ups: send the build event from the server node back to the thin client, which would upload it when it exits, or let a short-lived server exit after the build in CI, which changes the server's lifetime and needs agreement.
• CI detection is shared with the terminal logger, so a generic variable such as BUILD_ID on a developer machine also triggers the upload on exit. The cost of that false positive is a bounded wait, followed by a local save.
• A node mode that is only supplied through a response file isn't detected. MSBuild itself always passes /nodemode:N on the command line.

Background: how delivery happened before 18.13

Before the upload-on-exit behavior described above, MSBuild.exe only saved events to the local store on exit; it never uploaded them itself. Whether those events still reached the collector depended on which of two mechanisms applied.

In-process hosting (for example, Visual Studio's own build engine). BuildManager.EndBuildTelemetry calls TelemetryManager.Instance.Initialize(isStandalone: false), which resolves the session through TelemetryService.DefaultSession rather than creating one. When MSBuild's Framework assemblies are loaded in-process by a host that already created a default session (Visual Studio does this early at startup for its own telemetry), MSBuild's build event is posted onto that same live session object. MSBuild does not own or dispose it; the host's own long-running session delivers it through its normal flush cadence. Delivery on this path is not changed by this PR, and it was always reliable.

Separate MSBuild.exe processes (standalone, including CI and builds started from a VS Developer Command Prompt). XMake.cs always called Initialize(isStandalone: true) first, so every MSBuild.exe process owned its own session, created with TelemetryService.CreateAndGetDefaultSession, regardless of how it was launched. That session persists events to a storage folder keyed by the collector's instrumentation key (%LOCALAPPDATA%\Microsoft\VSApplicationInsights\vstel<hash>\*.trn), shared by every process on the machine that uses the same collector key. Before this change, delivery from that folder depended entirely on some process acting as sender:

• The owning PersistenceTransmitter tries to acquire a named, cross-process mutex over the folder. Only the process holding the mutex runs a Sender that polls the folder and uploads files.
• A sufficiently long-running process can deliver its own data: the in-memory buffer auto-flushes to disk every 30 seconds (FlushManager.FlushDelay), and the sender polls every 10 seconds when idle (Sender.sendingIntervalOnNoData). But events raised close to exit, including the build's own end event, can still be left in a final .trn file written during session disposal, after the last auto-flush.
• If no delivery happens before exit, the .trn file sits in the shared folder until another process using the same collector key starts, acquires the now-free sender mutex, and uploads it. That process can be Visual Studio itself (devenv.exe, since many builds happen under VS or a VS Developer Command Prompt) or simply a later MSBuild.exe invocation on the same machine. On a reused dev or build machine, a single drain can pick up leftovers from several prior runs if nothing drained them in between.
• This was verified experimentally: a second MSBuild.exe process uploaded a first process's leftover file, and a minimal harness using the same TelemetryService.DefaultSession API that Visual Studio uses delivered a child MSBuild.exe process's file about 7 seconds after the child exited.

[Copilot]

Copilot review overview

🟡 Changes recommended

Diagnostic output can alter the exit code, and critical ownership and shutdown behavior lacks automated regression coverage.

Review effort: Balanced
Findings: 2

Open (2)

• Guard diagnostic stderr writes against output failures · New
• Add coverage for process-mode telemetry ownership · New

What changed in this PR

Enables reliable Visual Studio telemetry delivery from headless .NET Framework MSBuild processes while preserving consent and protecting crash data.

Changes:

• Adds required telemetry assemblies and 64-bit/ARM64 resolution configuration.
• Introduces process ownership, CI-aware upload, bounded shutdown, diagnostics, and host attribution.
• Removes exception messages from crash telemetry and documents the behavior.

File	Description
src/​Shared/​UnitTests/​TestAssemblyInfo.cs	Disables telemetry during tests.
src/​Package/​MSBuild.VSSetup/​files.swr	Packages telemetry dependencies.
src/​MSBuild/​XMake.cs	Manages process-level telemetry ownership and shutdown.
src/​MSBuild/​app.amd64.config	Resolves telemetry assemblies for 64-bit processes.
src/​Framework/​Traits.cs	Adds telemetry diagnostics and timeout settings.
src/​Framework/​Telemetry/​TelemetryManager.cs	Implements bounded save/upload behavior.
src/​Framework/​Telemetry/​CrashTelemetryRecorder.cs	Flushes sessions and sanitizes fault exceptions.
src/​Framework/​Telemetry/​CrashTelemetry.cs	Removes exception-message properties.
src/​Framework/​DebugUtils.cs	Exposes the current process node mode.
src/​Framework/​BuildEnvironmentState.cs	Centralizes CI detection and attribution.
src/​Framework.UnitTests/​CrashTelemetry_Tests.cs	Verifies exception messages are excluded.
src/​Framework.UnitTests/​BuildEnvironmentState_Tests.cs	Tests CI and host detection precedence.
documentation/​wiki/​MSBuild-Environment-Variables.md	Documents telemetry environment variables.
documentation/​wiki/​CollectedTelemetry.md	Clarifies host-specific telemetry behavior.
documentation/​VS-Telemetry-Data.md	Documents collection, consent, delivery, and privacy.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

#	Dimension	Verdict
4	Test Coverage & Completeness	🟠 1 MAJOR
10	Design Before Implementation	🟠 1 MAJOR
6	Logging & Diagnostics Rigor	🟡 Existing thread

✅ 21/24 dimensions clean.

• Test Coverage — add deterministic coverage for CI network disposal and its timeout.
• Design — preserve the declared process-ownership boundary on worker/task-host/RAR crash paths.
• Diagnostics — the existing review thread already covers protecting Console.Error.WriteLine; not duplicated here.

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• api.github.com
• cafe.github.com
• github.com
• patchdiff.githubusercontent.com
• raw.githubusercontent.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.github.com"
    - "cafe.github.com"
    - "github.com"
    - "patchdiff.githubusercontent.com"
    - "raw.githubusercontent.com"

See Network Configuration for more information.

Generated by Expert Code Review (on open) for #15186 · copilot · gpt56 · 1.4K AIC · ⌖ 10.3 AIC · ⊞ 24.5K

[baronfel]

This seems reasonable, but @rainersigwald and @JanProvaznik probably have thoughts/feedback about the dll dependencies and if we can get that through VS perf gates.

[JanProvaznik]

General points:

Ownership: the entry process (or the server node) owns the session. Worker and task-host nodes don't create one. Inside VS, MSBuild uses the VS session and never disposes it.

this goes against the design of the crash/hang telemetry no?
Those kinds of problems are supposed to start the session in the node that crashed because there is no way to at that point send it to the node that owns the singular session.

The server path basically does not work:

A resident MSBuild server saves or uploads its session only at idle expiry (15 min after the last build) or explicit shutdown, so events can be lost on agents recycled sooner. Consider a per-build flush in CI. Also,  Environment.Exit(0)  in the cancel path ( XMake.cs:1541 ) skips  finally .

[JanProvaznik]

 DOTNET_CLI_TELEMETRY_OPTOUT  now also silences MSBuild's events inside VS.  IsOptOut()  is shared by  Initialize(isStandalone: false)  and by  BeginBuild 's collection gate ( BuildManager.cs:555 ), yet the docs say the VS path is unaffected ( VS-Telemetry-Data.md:200 ). Is that intended? If not, apply the SDK variable only when  isStandalone .

[YuliiaKovalova]

It comes from @baronfel 's comment here #15186 (comment), so yes, it's intended.