Add collect-linux mode to dotnet-trace integration

[LoopedBard3]

What

Adds a collect-linux mode to crank's dotnet-trace integration so Linux agents on modern kernels can capture whole-machine, perf_event-based traces with kernel + native frames, without LTTng.

New Job properties (defaults reproduce today's exact behavior):

• DotNetTraceCollectMode (default | collect | collect-linux) — default keeps today's in-process DiagnosticsClient.StartEventPipeSession path; collect and collect-linux shell out to the dotnet-trace CLI.
• DotNetTraceStopTimeoutSec — grace period after SIGINT before the CLI is killed. 0 selects the per-mode default (60s for collect, 180s for collect-linux).

Agent additions in Startup.cs:

• Pinned dotnet-trace 10.0.721401 (latest on nuget.org with collect-linux; ships a linux-arm64 build).
• Lazy dotnet tool install --tool-path on first CLI-mode use, with a post-install --version assert and a single-flight semaphore.
• CollectViaDotNetTraceCliAsync spawns the CLI directly (not via ProcessUtil.RunAsync) so we keep lifecycle control: wait for the benchmark to signal stop → send SIGINT → wait per-mode grace → fall back to Kill(entireProcessTree:true) on expiry.
• Grace-period expiry logs [WRN] dotnet-trace did not finalize within {N}s; trace may be incomplete and keeps the partial trace — does not fail the job.

Provider-string compatibility:

• BuildDotnetTraceCliArgs classifies each token into profile / provider-spec / CLR-keyword-expression using crank's existing TraceExtensions table.
• Required rewrite: cpu-sampling → dotnet-sampled-thread-time in collect mode only (the modern cpu-sampling profile has VerbExclusivity="collect-linux" and is rejected by the collect verb).
• Cosmetic alias rewrites: fusion → assemblyloader, gcheapcollect → managedheapcollect, plus the two gcsampledobjectallcation{high,low} typos.
• All rewrites are grouped into one [INF] dotnet-trace providers normalized: … line per job.
• Empty-providers default is cpu-sampling for collect-linux and dotnet-sampled-thread-time,dotnet-common for collect. Empty-providers default for default mode is unchanged (still cpu-sampling via PR A's preserved behavior).

collect-linux preflight (hard fail — no silent fallback to collect, per design):

• OS == Linux, kernel ≥ 6.4 (parsed from /proc/sys/kernel/osrelease), effective UID == 0 (via Mono.Unix.Native.Syscall.geteuid()).
• Verbatim error text:
  DotNetTraceCollectMode=collect-linux requires Linux, root (effective UID 0), and kernel >= 6.4. Detected: OS={os}, EUID={euid}, kernel={ver}. Set DotNetTraceCollectMode=collect to use EventPipe collection instead.

Docs:

• New flags documented in Documentation.cs + README.md.
• Decision matrix expanded from 2 rows (PR A) → 3 rows: DotNetTrace=true (any OS), DotNetTrace=true + DotNetTraceCollectMode=collect-linux (modern Linux + root), Collect=true (legacy PerfView pipelines).

Why

LTTng-based Collect=true is structurally fragile on modern Ubuntu — cobalt-hosted Ubuntu 24.04 aarch64 returns empty managed-event sequences even with lttng-tools + liblttng-ust-dev installed correctly (likely runtime↔liblttng-ust ABI mismatch). dotnet-trace collect-linux is the perf_event-based replacement: same kernel-sampling capability as perfcollect provided, but without the LTTng dependency. Cobalt hosts (kernel 6.8, agent container runs as root) meet every prerequisite.

Existing --application.dotNetTrace true users keep their exact current behavior. collect-linux is purely opt-in via the new mode flag.

Notes

• collect-linux rejects --buffersize (TreatUnmatchedTokensAsErrors=true in upstream's CollectLinuxCommand; perf_event uses kernel ring buffers, not EventPipe circular buffers). The argv builder strips it in that mode regardless of DotNetTraceBufferSizeMB.
• Pinned tool version targets net8.0 (the tool's own TFM). collect-linux requires the traced process to be .NET 10+. If a user points it at a .NET 8/9 process, the CLI's stderr ("EventPipe IPC command not understood") is captured verbatim into job.Error.
• Effective UID, not real UID. Mono.Unix.UnixUserInfo.RealUserId would return the pre-sudo UID under elevation; geteuid() is the signal that matters for perf_event_open capability checks. (Rubber-duck blind-spot fix from the planning session.)
• Windows collect mode is best-effort. GenerateConsoleCtrlEvent doesn't reliably deliver Ctrl+C to non-console children spawned with UseShellExecute=false; the helper falls back to CloseMainWindow() + the grace-period kill. Windows users almost always want default mode anyway.

Mixed-version compatibility

Direction	Behavior
New controller + old agent	Old agent's deserializer ignores DotNetTraceCollectMode / DotNetTraceStopTimeoutSec. If the controller asks for collect-linux, the old agent silently runs default. Documented expectation.
New agent + old controller	New properties default to today's exact behavior (default mode, 0-as-per-mode-default timeout). Byte-for-byte identical to today.

Validated by JobMixedVersionCompatTests covering legacy and PR-A-era payloads.

Tested

• dotnet build Microsoft.Crank.sln -c Release → clean (0 errors, 0 warnings).
• dotnet test test/Microsoft.Crank.UnitTests → 103 / 103 passing, including 33 new tests across:
  • DotNetTraceCliNormalizationTests (BuildDotnetTraceCliArgs / NormalizeClrEventExpression — defaults per mode, cpu-sampling rewrite, classification, alias rewrites, buffer-size gating)
  • DotNetTraceCollectLinuxPreflightTests (ParseKernelVersion across the 6.4 boundary, error-message contract on non-Linux)
  • Extended JobMixedVersionCompatTests (PR B knob defaults + round-trip)
• ❌ Manual collect-linux validation on cobalt aarch64 Ubuntu 24.04 host — pending reviewer. Manual-validation checklist before un-drafting:
  • dotnet tool install dotnet-trace --version 10.0.721401 --tool-path /tmp/x resolves an arm64 tool on the cobalt host (no x64-only fallback).
  • A simple .NET 10 benchmark with --application.dotNetTrace true --application.dotNetTraceCollectMode collect-linux produces a non-empty .nettrace with both managed and native frames.
  • Same invocation on a non-Linux/non-root environment fails fast with the documented preflight error string (no silent fallback to collect).

Operator notes

dotnet-trace collect-linux reads tracefs to translate perf_event_open records. In a container the /sys/kernel/tracing directory exists but is empty unless the host's tracefs is bind-mounted, surfacing as Error: Tracefs is not accessible: It appears tracefs is not mounted. at runtime. This PR adds -v /sys/kernel/tracing:/sys/kernel/tracing to docker/agent/run.sh so the wrapper script handles this for new agent deployments. EventPipe-based collection (DotNetTraceCollectMode=collect or Collect=true perfcollect) does not need this mount; only collect-linux does. The host must also have tracefs mounted (Ubuntu 24.04 + systemd does this at boot; check with mount | grep tracefs). Operators upgrading an existing agent need to ./docker/agent/stop.sh && ./docker/agent/run.sh to pick up the new mount -- volume mounts cannot be added to a running container. Validated on the cobalt-hosted aarch64 Ubuntu 24.04 (kernel 6.8) host.

The pinned dotnet-trace CLI version is 10.0.721401 (tag v10.0.721401 on dotnet/diagnostics:release/stable, commit 3f576a7). This is the latest stable release that contains both upstream fixes for the no-tty SetCursorPosition(0, -1) crash that occurs when collect-linux is spawned with redirected stdout: dotnet/diagnostics#5771 ("Fix no-tty crash with console capability aware ProgressWriter") and dotnet/diagnostics#5745 ("Handle redirected input/output in collect-linux"). 10.x is not on nuget.org; the agent writes a hermetic NuGet.config next to the tool path and resolves from the dnceng public dotnet-tools feed via --configfile.

Stacking

PR A (#886) has merged into dotnet/crank:main as squash commit d392425. This branch has been rebased onto current main and now contains only the collect-linux commits.

[Copilot]

Pull request overview

Adds a new DotNetTraceCollectMode=collect-linux option to crank’s dotnet-trace integration, enabling perf_event-based whole-machine tracing on modern Linux (kernel + native frames) via the dotnet-trace CLI, without relying on LTTng/perfcollect, while keeping the legacy in-process EventPipe behavior as the default.

Changes:

• Introduces DotNetTraceCollectMode and DotNetTraceStopTimeoutSec Job knobs (defaults preserve existing behavior).
• Adds agent-side dotnet-trace CLI installation + lifecycle management, plus collect-linux prerequisite validation and provider-token normalization for CLI modes.
• Expands documentation and unit tests to cover mixed-version compatibility, CLI arg normalization, and Linux preflight parsing/contract.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Microsoft.Crank.Agent/Startup.cs	Implements CLI-based collect / collect-linux modes, lazy tool install, preflight checks, argv builder, and stop/grace/kill behavior.
src/Microsoft.Crank.Agent/TraceExtensions.cs	Adds CLR keyword-expression recognition helper used for CLI token classification.
src/Microsoft.Crank.Models/Job.cs	Adds DotNetTraceCollectMode + DotNetTraceStopTimeoutSec with legacy-preserving defaults.
src/Microsoft.Crank.Controller/Documentation.cs	Documents the new dotnet-trace mode and stop-timeout CLI flags.
src/Microsoft.Crank.Controller/README.md	Updates controller help text for the new dotnet-trace flags.
docker/agent/run.sh	Adds /sys/kernel/tracing bind mount required for dotnet-trace collect-linux tracefs access.
test/Microsoft.Crank.UnitTests/DotNetTraceCliNormalizationTests.cs	Adds unit tests for CLI argv building, token classification, and alias normalization behavior.
test/Microsoft.Crank.UnitTests/DotNetTraceCollectLinuxPreflightTests.cs	Adds unit tests for kernel-version parsing and the non-Linux error-message contract.
test/Microsoft.Crank.UnitTests/JobMixedVersionCompatTests.cs	Extends mixed-version deserialization/roundtrip tests to include PR-B knobs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[LoopedBard3]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[DrewScoggins]

Still not convinced on the necessity of the euid validation method, and still think we should get rid of the Go-style tuple return, but those aren't pr stoppers.