🏥 Repository Health Dashboard

[github-actions]

🏥 Daily Health Check — 2026-04-27

Monitoring: dotnet/dotnet
Status: 🔴 2 critical · 🟡 2 warnings · 🔵 0 info
Since yesterday: 🆕 0 new · ✅ 2 resolved · 📌 4 unchanged

i️ ADO data covers public CI (dnceng-public) only.
⚠️ Source-build (P8), NuGet feed health (I6), source manifest staleness (I7), repo config (I3, I4, I5) skipped: pre-collected data not available for these checks.

---

🆕 New Findings (0)

✅ No new findings since the last run.

---

🔍 Investigation Results

Deep investigations are dispatched for new 🔴 Critical and 🟡 Warning pipeline findings.
No new qualifying findings this run — no investigations dispatched.

Finding	Severity	Status	Result
dotnet-dotnet public CI 14-day pass rate critical	🔴 Critical	✅ Done	Two concurrent independent failures are driving the pass rate: (1) a recurring SIGN004 signing misconfiguration for a 3rd party assembly in ContainerSigning/118 using Arcade SDK toolset 11.0.0-beta.26210.111, and (2) an ongoing intermittent npm ci crash (exit code 57005 / 0xDEAD) in the aspnetcore build step — the aspnetcore team is actively adding diagnostics but has not yet shipped a fix.
Codeflow PRs blocked >3 days	🔴 Critical	✅ Done	All 4 blocked codeflow PRs require manual intervention — 3 are stuck on unresolved merge conflicts (requiring darc vmr resolve-conflict) and 1 is blocked by a source-build failure on the release/11.0.1xx-preview3 branch.
dotnet-dotnet public CI 14-day pass rate critical (21.4%)	🔴 Critical	✅ Done	The public dotnet-dotnet CI pipeline on main is failing at a critical rate (10/14 builds over 14 days) due to two compounding failures: (1) the src/runtime/build.sh build script exiting with non-zero status (MSB3073), and (2) a SIGN004 error where a newly-introduced 3rd party library is being signed with a Microsoft certificate — a known, previously-resolved pattern.
dotnet-dotnet public CI failed on main	🟡 Warning	✅ Done	Build 1390567 failed on two fronts — (1) the src/runtime/build.sh subprocess exited non-zero (primary, root cause unclear without full logs), and (2) a SIGN004 signing policy violation in ContainerSigning was simultaneously active and has since been fixed by PR #6191 (merged same day). The 14-day pass rate indicates a chronic underlying issue, most likely in the runtime build leg.
Multiple operational issues stale >3 days	🔴 Critical	✅ Done	18 open area-unified-build-OperationalIssue issues are stale (no updates for >3 days), with the oldest unresolved since December 2025, indicating a triage and ownership gap rather than a single root cause — immediate triage, assignment, and closure of resolved or permanently-intermittent issues is recommended.
dotnet-dotnet public CI failed on main	🔴 Critical	✅ Done	The public CI dotnet-dotnet pipeline on main is exhibiting chronic failures driven by two concurrent issues: an ongoing src/runtime/build.sh restore/build failure and a SIGN004 signing-policy violation for a newly-introduced third-party binary in the source-built artifacts.
dotnet-dotnet 14-day pass rate critical (21.4%)	🔴 Critical	✅ Done	Two concurrent failure modes — a recurring SIGN004 3rd-party signing violation and a MSB3073 runtime Linux build script failure — are together driving a critical pass rate on the main branch; the SIGN004 fix follows a well-known pattern and can be applied quickly, while the runtime failure requires investigation of recent codeflow changes.
Codeflow PR dotnet#6120 blocked >3 days	🔴 Critical	✅ Done	PR #6120 passed all CI and Maestro auto-merge checks but remains unmerged due to its branch being out-of-date with the release/10.0.4xx base branch; Maestro re-evaluated the PR today (17:40 UTC) which may indicate an imminent merge or a Maestro queue delay.

---

✅ Resolved (2)

• [Critical] Multiple operational issues stale >3 days — infra:stale-operational-issues:critical — all operational issues cleared ✅
• [Warning] 19+ open operational issues (>10 threshold) — infra:stale-operational-issues:warning — 0 open operational issues now ✅

---

📌 Existing (4)

🔴 [Critical] dotnet-dotnet public CI 14-day pass rate critical (7.1%) — first seen 2026-04-24, 4 occurrences

Fingerprint: pipeline:ado:rolling-build:pass-rate:critical
Category: pipeline · Severity: 🔴 Critical · Check: P4

The 14-day pass rate for dotnet-dotnet on main is 7.1% — well below the 🔴 Critical threshold of 50%.

Metric	Value
14-day total builds	14
Succeeded	1
Failed	12
Pass rate	7.1% 🔴

Failed tasks: Sign Source-Built Artifacts (SIGN004 — 3rd party signing in ContainerSigning/117), Build (MSB3073 in src/sdk on Linux), Build (MSB3073 in src/wpf on Windows).

[ADO pipeline →]((dev.azure.com/redacted)

🔴 [Critical] Codeflow PRs blocked >3 days (7 PRs, oldest: ~54 days) — first seen 2026-04-24, 4 occurrences

Fingerprint: infra:codeflow-backlog:critical
Category: infra · Severity: 🔴 Critical · Check: I2

7 codeflow PRs have been open for >3 days, indicating flow blockages (PR dotnet#6236 merged since last run ✅):

PR	Title	Age
#5229	[main] Source code updates from dotnet/diagnostics	~54 days
#5743	[release/11.0.1xx-preview3] microsoft/vstest	~31 days
#6106	[release/10.0.1xx] dotnet/symreader	~10 days
#6111	[release/10.0.1xx] dotnet/sourcelink	~10 days
#6226	[release/10.0.3xx] dotnet/sourcelink	~4 days
#6232	[main] dotnet/emsdk	~3 days
#6233	[main] dotnet/arcade	~3 days

Open codeflow PRs →

🟡 [Warning] dotnet-dotnet public CI failed on main (24h) — first seen 2026-04-24, 4 occurrences

Fingerprint: pipeline:ado:dnceng-public:dotnet-dotnet:build:failed
Category: pipeline · Severity: 🟡 Warning · Check: P1

The dotnet-dotnet public pipeline (dnceng-public) had 1 failed build out of 1 in the last 24h on main.

• [Build #1397450]((dev.azure.com/redacted) — started 2026-04-27T05:01Z, finished 2026-04-27T10:55Z (~5h54m)
• Failed tasks:
  • Sign Source-Built Artifacts — SIGN004: Signing 3rd party library in ContainerSigning/117
  • Build (Linux) — MSB3073: src/sdk/eng/common/build.sh exited non-zero
  • Build (Windows) — MSB3073: src/wpf/eng/common/build exited non-zero

🟡 [Warning] 23 codeflow PRs open simultaneously (>5 threshold) — first seen 2026-04-24, 4 occurrences

Fingerprint: infra:codeflow-backlog:warning
Category: infra · Severity: 🟡 Warning · Check: I2

Currently 23 open codeflow PRs (warning threshold: >5), increased from 16 yesterday. The increase reflects new forward-flow PRs from roslyn, vstest, aspnetcore, fsharp, and other repos across multiple branches.

Open codeflow PRs →

---

📊 Summary

Category	Checks Run	Findings
ADO Pipelines (public)	P1, P4	🔴 1 critical, 🟡 1 warning
GitHub Actions	P7	✅ No failures (24h)
Codeflow PRs	P5, I2	🔴 1 critical, 🟡 1 warning
Infrastructure (Ops Issues)	I1	✅ 0 open issues
Source-build	P8	⚠️ Skipped (data not available)
NuGet feed health	I6	⚠️ Skipped (data not available)
Source manifest staleness	I7	⚠️ Skipped (data not available)
Repo config (CODEOWNERS, Dependabot, Actions)	I3, I4, I5	⚠️ Skipped (data not available)

---

_{🤖 Generated by DevOps Health Check · 2026-04-27T17:22 UTC · Monitoring dotnet/dotnet}

Generated by DevOps Daily Health Check · ● 433.8K · ◷

[github-actions]

📋 Health Check — 2026-04-21

🆕 0 new · ✅ 0 resolved · 📌 1 unchanged

New: (none)

Resolved: (none)

Existing:

• 🔵 [Info] Unpinned action: dotnet/arcade backport-base.yml@main in .github/workflows/backport.yml (2nd occurrence)

⚠️ ADO pipeline checks skipped — health-data/ directory was not populated by Phase 1. NuGet feed and source-manifest staleness checks also skipped.

Full report →

Generated by DevOps Daily Health Check · ● 1M · ◷

[github-actions]

📋 Health Check — 2026-04-21 (Run #3)

🆕 4 new · ✅ 0 resolved · 📌 1 unchanged

New findings:

• 🔴 Multiple operational issues stale >3 days — 18 open ops issues, oldest from Dec 2025
• 🔴 Codeflow PR #6120 blocked >3 days — nuget/nuget.client → release/10.0.4xx (~77h)
• 🟡 18 open operational issues (>10 threshold)
• 🟡 25 codeflow PRs open simultaneously (>5 threshold)

Resolved: None

Investigations dispatched: 2 (critical findings)

Full report →

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #6172 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Daily Health Check · ● 860.3K · ◷

[github-actions]

🔍 Investigation: Multiple operational issues stale >3 days

Finding ID: infra:stale-operational-issues:critical
Severity: 🔴 critical
Correlation: hc-2026-04-21-1
Executive Summary: 18 open area-unified-build-OperationalIssue issues are stale (no updates for >3 days), with the oldest unresolved since December 2025, indicating a triage and ownership gap rather than a single root cause — immediate triage, assignment, and closure of resolved or permanently-intermittent issues is recommended.

Root Cause

The VMR has accumulated 18 open operational issues, none of which have been updated within the 3-day threshold. The pattern spans a wide date range (Dec 2025 – Mar 2026) and multiple failure categories (OOM, file-lock races, IL format errors, crossgen2 crashes, component detection timeouts). This strongly indicates a process gap: issues are being filed by automated health checks and engineers when failures occur, but there is no consistent triage loop to close them once the underlying condition resolves, assign owners for ongoing investigation, or escalate issues that persist beyond a reasonable SLA. The most impactful unresolved issue is dotnet#5428 (AzureLinux arm64 OOM — 100% failure rate on release/10.0.2xx over 14 days), which has been open 39+ days with only 3 comments and no resolution. Two older issues (dotnet#3978, dotnet#4095) have zero comments and have been open 3–4 months.

Confidence: High — The issue list is directly observable; staleness is measured against updated_at timestamps from the GitHub API.

Blast Radius

• release/10.0.2xx is the most affected branch: AzureLinux_x64_Cross_arm64 builds failing with OOM/restore errors on 10.0.2xx and 10.0.3xx dotnet/dotnet#5428 (100% arm64 OOM failures), Razor parallel build file-lock race on release/10.0.2xx dotnet/dotnet#5770 (40% Razor file-lock race), and likely others
• release/10.0.3xx also affected by AzureLinux_x64_Cross_arm64 builds failing with OOM/restore errors on 10.0.2xx and 10.0.3xx dotnet/dotnet#5428
• internal main build affected by crossgen2 exit code 57005 (0xDEAD) in internal main build 2925590 dotnet/dotnet#5467 (crossgen2 crash, single occurrence)
• scenario tests on 10.0.1xx affected by Intermittent MissingMethodException initializing scenario test run dotnet/dotnet#3660 (MissingMethodException, intermittent)
• Unresolved stale issues create noise in health dashboards and may mask new regressions

Suggested Fix

1. Immediate triage pass — Review all 18 open issues and close any where the underlying build failure no longer reproduces or has been resolved upstream. Candidates: [Operational Issue]: Failed to download releases.json when installing .NET dotnet/dotnet#3978 (releases.json download), [Operational Issue]: Component detection timing out in some source build runs dotnet/dotnet#4095 (component detection timeout), Intermittent MissingMethodException initializing scenario test run dotnet/dotnet#3660 (MissingMethodException if analyzer version mismatch was fixed in a roslyn/runtime update).
2. Escalate AzureLinux_x64_Cross_arm64 builds failing with OOM/restore errors on 10.0.2xx and 10.0.3xx dotnet/dotnet#5428 (AzureLinux arm64 OOM) — This is the most severe active issue with a 100% failure rate on release/10.0.2xx. Assign to @dotnet/runtime-infrastructure or @dotnet/product-construction to assess memory allocation for the arm64 cross-compilation container; this is a recurrence of VMR Build Failure: Job/Task Canceled dotnet/dotnet#1001/VMR Build Failure: Job Timeout (Exceeded Maximum Time) dotnet/dotnet#1002 (closed July 2025) and likely requires a container memory limit increase or Roslyn package size reduction.
3. Establish a triage SLA — Add a regular cadence (e.g., weekly) for the unified-build team to review area-unified-build-OperationalIssue issues: close stale/resolved ones, escalate high-severity ones, and set area-unified-build-OperationalIssue resolution expectations in CONTRIBUTING.md or team docs.

Evidence

Stale issues breakdown (as of 2026-04-21):

Issue	Title	Days Stale	Last Updated
#3978	Failed to download releases.json	~123 days	2025-12-19
#4095	Component detection timing out in source build	~104 days	2026-01-07
#3660	MissingMethodException in scenario test run	~99 days	2026-01-13
#4732	testhost.dll: BAD IL format	~71 days	2026-02-09
#5428	AzureLinux arm64 OOM/restore (100% failure on 10.0.2xx)	~39 days	2026-03-13
#5467	crossgen2 exit code 0xDEAD in internal main	~38 days	2026-03-14
#5770	Razor parallel file-lock race on release/10.0.2xx	~22 days	2026-03-30

Critical active failure — dotnet#5428 error signature:

repo-projects/Directory.Build.targets(606,5): MSB3073 (NETCORE_ENGINEERING_TELEMETRY=Restore)
OOM during NuGet restore of Roslyn packages in arm64 cross-compilation container
Affected: release/10.0.2xx (18/18 builds failed), release/10.0.3xx
```

**#5770 (Razor race — 40% on release/10.0.2xx):**
```
error MSB4018: "GetAssemblyAttributes" task failed unexpectedly.
System.IO.IOException: The process cannot access the file
  'src/razor/artifacts/obj/Microsoft.CodeAnalysis.Remote.Razor/Release/net8.0/Microsoft.CodeAnalysis.Remote.Razor.dll'
  because it is being used by another process.

Related

• Open operational issues (all)
• #5428 — AzureLinux arm64 OOM (most critical active issue)
• #5770 — Razor file-lock race
• #5467 — crossgen2 0xDEAD crash
• Prior art: VMR Build Failure: Job/Task Canceled dotnet/dotnet#1001, VMR Build Failure: Job Timeout (Exceeded Maximum Time) dotnet/dotnet#1002 (arm64 OOM, closed July 2025 — recurrence of AzureLinux_x64_Cross_arm64 builds failing with OOM/restore errors on 10.0.2xx and 10.0.3xx dotnet/dotnet#5428)

---

_{🔍 Investigation Run #24737708952 · Dispatched by health check · hc-2026-04-21-1}

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 188.1K · ◷

[github-actions]

🔍 Investigation: Codeflow PR dotnet#6120 blocked >3 days (nuget/nuget.client → release/10.0.4xx)

Finding ID: infra:codeflow-backlog:critical
Severity: 🔴 critical
Correlation: hc-2026-04-21-2
Executive Summary: PR dotnet#6120 passed all CI and Maestro auto-merge checks but remains unmerged due to its branch being out-of-date with the release/10.0.4xx base branch; Maestro re-evaluated the PR today (17:40 UTC) which may indicate an imminent merge or a Maestro queue delay.

Root Cause

PR #6120 carries nuget/nuget.client source updates (23 files, +204/−10) targeting release/10.0.4xx. All 13 check runs completed successfully, including every Maestro auto-merge policy check (All Checks Successful, Validate coherency, Do not automerge downgrades, Codeflow verification). Despite this, GitHub reports mergeable_state: blocked. The PR's base commit is 5b150b1 but the current release/10.0.4xx HEAD is at 997e4173 — the branch has advanced by at least 3 additional merges since PR creation on 2026-04-18. The most probable cause of the blocked state is a branch-is-out-of-date condition enforced by branch protection rules. Maestro normally auto-updates stale codeflow branches before merging, but the release/10.0.4xx branch has been unusually active with manual interventions (vstest resets dotnet#6136, dotnet#6169; SDK band init dotnet#6090), which may have disrupted Maestro's merge queue. Notably, all Maestro checks were re-run and passed at 17:40 UTC today, suggesting Maestro is actively re-evaluating the PR's readiness.

Confidence: Medium — All CI evidence points to an out-of-date branch as the blocker; no CI failures, no merge conflicts, and no review blocks were found. The exact Maestro queue state is not accessible without direct Maestro/ADO access.

Blast Radius

• release/10.0.4xx codeflow pipeline: 25 total open codeflow PRs across the VMR (per health check data); any that depend on the nuget/nuget.client changes will also be delayed.
• nuget.client updates: 3 upstream NuGet.Client PRs (#7261, #7291, #7296) are not yet reflected in the VMR's release/10.0.4xx branch.
• Build 7.7.0.15 (nuget.client, produced 2026-04-18 05:06 UTC) has not flowed into VMR — any downstream consumers of that build are also blocked.

Suggested Fix

1. Wait and monitor (~1–2 hours): Maestro re-evaluated all checks at 17:40 UTC today and found everything green. Maestro may be in the process of updating the branch and triggering a merge. Check again after the current Maestro cycle completes.
2. Trigger a manual branch update: If the PR is still open after the next Maestro cycle, ask @dotnet/prodconsvcs or use darc to manually update the branch:

   darc update-dependences --repo dotnet/dotnet --branch darc-release/10.0.4xx-e02ceb77-7dee-492f-a4cd-ef27522b4f4a
   

   Or request a /merge command via the Maestro bot on the PR.
3. Investigate vstest disruption: PR [release/10.0.4xx] Reset Vstest to rel/18.7 311075 dotnet/dotnet#6136 ([release/10.0.4xx] Reset Vstest to rel/18.7 311075) has been open since 2026-04-20 and PR [release/10.0.4xx] Initialize SDK band versions dotnet/dotnet#6090 (Initialize SDK band versions, open since 2026-04-16) requires human review from @NikolaMilosavljevic. These unmerged manual PRs may be interacting with Maestro's merge queue ordering. Resolve or merge these to unblock the queue.

Evidence

PR #6120 check runs (all: success):
  ✅ dotnet-unified-build             completed 2026-04-18T14:08:39Z
  ✅ VMR Vertical Build Windows_x64   completed 2026-04-18T13:27:33Z
  ✅ VMR Source-Only Build SB_CentOSStream10 completed 2026-04-18T14:08:35Z
  ✅ Maestro auto-merge - All Checks Successful  completed 2026-04-21T17:40:19Z ← TODAY
  ✅ Maestro auto-merge - No Requested Changes   completed 2026-04-21T17:40:18Z ← TODAY
  ✅ Maestro auto-merge - Validate coherency     completed 2026-04-18T12:43:37Z
  ✅ Maestro auto-merge - Codeflow verification  completed 2026-04-18T12:43:37Z

PR state:      open, draft: false, merged: false
Mergeable:     blocked
Reviews:       none

Base branch evolution (release/10.0.4xx since PR creation):
  5b150b1  ← PR #6120 base (2026-04-18)
  66fba525 ← #6116 fsharp merged 2026-04-20
  1513e8b1 ← additional merges
  997e4173 ← current HEAD (2026-04-21)

Related

• PR #6120 — This PR (nuget/nuget.client → release/10.0.4xx)
• PR #6090 — Initialize SDK band versions (open since Apr 16, awaiting @NikolaMilosavljevic review) — may be affecting queue
• PR #6136 — Reset Vstest (open since Apr 20, manual fix PR) — potential queue interference
• Maestro subscription: [5039ea22-178d-433e-baa4-954e945ddf10]((maestro.dot.net/redacted)
• NuGet build on Maestro: [310908]((maestro.dot.net/redacted)
• No matching area-unified-build-OperationalIssue found for this specific blockage

---

_{🔍 Investigation Run #24737713196 · Dispatched by health check · hc-2026-04-21-2}

Note

🔒 Integrity filter blocked 6 items

The following items were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet/NuGet.Client@de3b92c list_commits: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet/NuGet.Client@ffd4cc1 list_commits: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet/NuGet.Client@461fa93 list_commits: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet/NuGet.Client@106309a list_commits: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet/NuGet.Client@1f130f6 list_commits: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 296.3K · ◷

[github-actions]

📋 Health Check — 2026-04-21 (Run 4)

🆕 4 new · ✅ 0 resolved · 📌 5 unchanged
Status: 🔴 4 critical · 🟡 4 warnings · 🔵 1 info

---

🆕 New findings:

• 🔴 pipeline:ado:dnceng-public:dotnet-dotnet:build:failed — Public CI failed on main: [Build #1388566]((dev.azure.com/redacted) — MSB3073: src/runtime/build.sh + signing failure (SIGN004)
• 🔴 pipeline:ado:rolling-build:pass-rate:critical — 14-day pass rate 21.4% (3/14 builds passing on dotnet-dotnet main) 🔴 below 50% threshold
• 🟡 infra:no-codeowners — No CODEOWNERS file detected in the repo
• 🟡 infra:no-dependabot — No Dependabot configuration detected

✅ Resolved: None

🔍 Investigations dispatched (2):

• hc-2026-04-21-4 → dotnet-dotnet public CI failed on main
• hc-2026-04-21-5 → 14-day pass rate critical (21.4%)

Full report →

Generated by DevOps Daily Health Check · ● 1.1M · ◷

[github-actions]

🔍 Investigation: dotnet-dotnet public CI 14-day pass rate critical: 21.4% (3/14 builds passing)

Finding ID: pipeline:ado:rolling-build:pass-rate:critical
Severity: 🔴 critical
Correlation: hc-2026-04-21-5
Executive Summary: Two concurrent failure modes — a recurring SIGN004 3rd-party signing violation (new untracked library in ContainerSigning/118) and a MSB3073 runtime Linux build script failure — are together driving a critical 21.4% pass rate on the main branch over 14 days; the SIGN004 fix follows a well-known pattern and can be applied quickly, while the runtime failure requires investigation of recent codeflow changes or infrastructure OOM.

Root Cause

The pipeline is failing on two distinct, concurrent errors reported in context_json:

1. SIGN004: Signing 3rd party library ContainerSigning/118 — A 3rd-party library (contained in signing batch 118) is being signed with a Microsoft certificate (MicrosoftDotNet500). This is a well-documented recurring pattern in this repository: new 3rd-party dependencies with empty or non-Microsoft copyright metadata enter the build via codeflow and trigger the Arcade Sign.proj validation. Prior instances: #3199 (Spectre.Console.dll, Nov 2025), #5359 (Antlr4.Runtime.Standard.dll, Mar 2026). No currently open issue tracks this specific batch/library, suggesting a new dependency has recently entered the build. Recent codeflow PRs (#6162 from dotnet/runtime, Apr 21; #6096 from dotnet/sdk, Apr 17) are likely candidates for introducing this dependency.

2. MSB3073: src/runtime/build.sh — The runtime Linux build script is exiting with a non-zero exit code. MSB3073 means an invoked command failed. This maps to similar patterns seen in: #5428 (OOM/NuGet restore errors in AzureLinux_x64_Cross_arm64, open) and #4361 (cross-DAC build-runtime.cmd failure, Jan 2026). With 11/14 builds failing over 14 days, this is a sustained chronic condition not a recent one-off. The root sub-cause (OOM, cross-compilation infrastructure issue, or source regression) cannot be confirmed without access to the ADO build logs.

Confidence: Medium — The SIGN004 pattern is well-understood (High confidence) and matches multiple prior issues exactly. The MSB3073/runtime failure is circumstantially correlated with runtime codeflow activity and the OOM arm64 pattern, but the specific sub-cause cannot be confirmed from GitHub context alone (Medium).

Blast Radius

• Main branch .NET 11 rolling builds: 78.6% of builds are failing, blocking stable nightly SDK output.
• Source-build validation: SIGN004 failures occur specifically in source-build offline legs (e.g., SB_CentOSStream10_Offline_MsftSdk_x64), blocking source-build deliverables.
• Developer experience: Frequent failures on main erode confidence and may delay codeflow from upstream repos.
• Toolset 11.0.0-beta.26210.111: All builds using this toolset snapshot are affected.

Suggested Fix

1. Identify and fix the SIGN004 library (high priority, known fix pattern): Check the ADO build logs for the Sign Source-Built Artifacts step, locate the specific DLL in ContainerSigning/118. Then either: (a) add the library to the exclusion list if it's truly 3rd-party and should not be signed, or (b) update its signing configuration in eng/Sign.props or the relevant arcade signing config to use a 3rd-party certificate. See #5359 for the exact fix pattern — required updating the copyright/signing assignment in the sdk's signing config or via an arcade update.

2. Investigate MSB3073: runtime/build.sh failure: Pull full logs from a recent failed build at [(dev.azure.com/redacted)(dev.azure.com/redacted) Look for the actual error inside runtime/build.sh— common sub-causes are: OOM (check memory telemetry), WASI/external SDK download failures (network connectivity), or a source regression from recent runtime codeflow. If OOM, consider coordinating with@dotnet/runtime` team for memory profiling.

3. Retry builds after SIGN004 fix: The signing failure causes the entire build to abort early; fixing it may unblock builds and reveal whether the MSB3073 is still independently failing or was masked by the signing abort order.

Evidence

context_json failures:
  - "MSB3073: src/runtime/build.sh"          ← Linux runtime build script non-zero exit
  - "SIGN004: Signing 3rd party library ContainerSigning/118"  ← 3rd-party cert mismatch

14-day stats (main branch):
  total: 14, succeeded: 3, failed: 9, pass_rate: 21.4%
  threshold_critical: 50%

Toolset: 11.0.0-beta.26210.111

Prior SIGN004 instances (all closed after signing config updates):
  #3199 — Spectre.Console.dll — Nov 2025
  #5359 — Antlr4.Runtime.Standard.dll — Mar 2026

Similar MSB3073 pattern (still open):
  #5428 — AzureLinux_x64_Cross_arm64 OOM/restore on release branches — Mar 2026

Recent runtime codeflow merges to main:
  #6098 — Apr 20, 2026 (dotnet/runtime)
  #6162 — Apr 21, 2026 (dotnet/runtime, build 20260420.20, commit c87e7f8)

No currently open operational issue tracks these specific failures on main.

Related

• #5359 — SIGN004: Antlr4.Runtime.Standard.dll (closed Mar 2026) — same error class
• #3199 — SIGN004: Spectre.Console.dll (closed Nov 2025) — same error class
• #5428 — AzureLinux_x64_Cross_arm64 OOM with MSB3073 (open) — related MSB3073 pattern
• #4849 — Illegal byte sequence during real signing (open) — related signing infrastructure issue
• ADO pipeline: [dotnet-dotnet public CI]((dev.azure.com/redacted)

---

_{🔍 Investigation Run #24745731392 · Dispatched by health check · hc-2026-04-21-5}

Note

🔒 Integrity filter blocked 5 items

The following items were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #4268 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #4037 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #5915 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #5915 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 689K · ◷

[github-actions]

🔍 Investigation: dotnet-dotnet public CI failed on main

Finding ID: pipeline:ado:dnceng-public:dotnet-dotnet:build:failed
Severity: 🔴 critical
Correlation: hc-2026-04-21-4
Executive Summary: The public CI dotnet-dotnet pipeline on main is exhibiting chronic failures (21.4% pass rate over 14 days) driven by two concurrent issues: an ongoing src/runtime/build.sh restore/build failure (likely a NuGet feed race or runtime regression) and a SIGN004 signing-policy violation for a newly-introduced third-party binary in the source-built artifacts.

Root Cause

Two distinct failures are present in build [1388566]((dev.azure.com/redacted)

1. Runtime build failure (primary) — repo-projects/Directory.Build.targets(622,5): error MSB3073 when invoking src/runtime/build.sh --restor[e].... The error is truncated, but the --restor prefix strongly suggests a package-restore failure during the runtime build step. The pipeline has failed 9 out of 14 times in the last 14 days, a pattern consistent with the tracked issue #5609 ("darc-pub-* feeds can cause package restore races in stable"), where NuGet feed propagation delays cause intermittent build failures on public CI that uses online feeds. The failed build started at 05:01 UTC on April 21; runtime codeflow PR #6162 (283 files, 11,095 additions, build 20260420.20) was not merged until 07:52 UTC, so it was not the trigger for this run.

2. Signing validation failure (secondary) — artifacts/toolset/11.0.0-beta.26210.111/Sign.proj(74,5): error SIGN004: Signing 3rd party library '/__w/1/s/artifacts/tmp/Release/ContainerSigning/118.... SIGN004 indicates Arcade's signing task detected a binary that is classified as a third-party library being submitted for first-party signing. A related (but distinct) signing issue is tracked as #5526. The truncated path suggests a new binary in the container-signing bundle may be missing from the signing-exclusion list.

Confidence: Medium — Strong circumstantial evidence ties the restore failures to the known feed-race issue (dotnet#5609), and the SIGN004 pattern is well-understood. However, the truncated error messages prevent a definitive confirmation without full ADO log access.

Blast Radius

• main branch only for this specific build; however, the chronic failure rate (21.4% pass over 14 days) means most CI runs on main are blocked.
• The signing failure blocks the "Sign Source-Built Artifacts" step, which means even when the build itself succeeds, signing may fail — preventing artifact promotion.
• Any downstream channels depending on dotnet-dotnet public builds on main (e.g., dependency-flow subscriptions consuming VMR build artifacts) are starved.

Suggested Fix

1. Investigate runtime restore failure: Open the full ADO build log at [Build 1388566]((dev.azure.com/redacted), expand the "Build" step, and capture the complete src/runtime/build.sh error. Cross-reference with open issue #5609 — if the error is a PackageNotFound or HTTP 404 during restore, a retry of the build may succeed (intermittent feed race). If the error is reproducible, bisect using the runtime commit range in PR #6162 diff (10b20d5...c87e7f8).

2. Fix the SIGN004 signing violation: Identify the third-party binary at artifacts/tmp/Release/ContainerSigning/118... (full path visible in the ADO log). Determine which codeflow PR introduced it, then either: (a) add the binary to the signing exclusion list in eng/signing-validation.proj or the relevant .signproj file, or (b) remove the binary if it should not be packaged. Reference existing fix pattern in #5526 comments.

3. Consider filing a new operational issue if the runtime restore failure turns out to be distinct from [Operational Issue]: darc-pub-* feeds can cause package restores races in stable branches dotnet/dotnet#5609. The 14-day failure rate of 78.6% is severe and warrants an area-unified-build-OperationalIssue tracking issue if not already covered by an existing one.

Evidence

Failed tasks (from build 1388566, 2026-04-21 05:01–11:03 UTC, duration ~6h):

[1] Build
  repo-projects/Directory.Build.targets(622,5): error MSB3073:
  (NETCORE_ENGINEERING_TELEMETRY=Build) The command
  "/__w/1/s/src/runtime/build.sh --restor[e ...]" exited with code N

[2] Sign Source-Built Artifacts
  artifacts/toolset/11.0.0-beta.26210.111/Sign.proj(74,5): error SIGN004:
  Signing 3rd party library '/__w/1/s/artifacts/tmp/Release/ContainerSigning/118...'

14-day stats: total=14, succeeded=3, failed=9, pass_rate=21.4%

Recent main codeflow activity around build window (04-21 05:01 UTC):

• [main] Source code updates from dotnet/msbuild ([main] Source code updates from dotnet/msbuild dotnet/dotnet#6155) — merged 07:49 UTC (after build start)
• [main] Source code updates from dotnet/runtime ([main] Source code updates from dotnet/runtime dotnet/dotnet#6162) — merged 07:52 UTC (after build start), 283 files, 11,095 additions from build 20260420.20

Related

• #5609 — [Operational Issue]: darc-pub-* feeds can cause package restores races in stable (open)
• #5526 — [Operational Issue]: Validate Signing fails on Linux: Unable to read beyond the end of the stream. (open, different SIGN error but same subsystem)
• PR #6162 — [main] Source code updates from dotnet/runtime (large codeflow, may affect subsequent builds)
• [ADO Build 1388566]((dev.azure.com/redacted) — primary failing build

---

_{🔍 Investigation Run #3 · Dispatched by health check · hc-2026-04-21-4}

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #3314 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 332.7K · ◷

[github-actions]

📋 Health Check — 2026-04-23

🆕 2 new · ✅ 0 resolved · 📌 5 unchanged

New:

• 🔴 dotnet-dotnet public CI 14-day pass rate critical (21.4%) · pipeline:ado:rolling-build:pass-rate:critical
• 🟡 dotnet-dotnet public CI failed on main · pipeline:ado:dnceng-public:dotnet-dotnet:build:failed

Resolved: None

Still active (critical):

• 🔴 5 codeflow PRs open >3 days (oldest: [main] Source code updates from dotnet/diagnostics dotnet/dotnet#5229, 50 days) · infra:codeflow-backlog:critical
• 🔴 Multiple operational issues stale >3 days (19 open) · infra:stale-operational-issues:critical

Full report →

Generated by DevOps Daily Health Check · ● 1M · ◷

[github-actions]

🔍 Investigation: dotnet-dotnet public CI failed on main (24h)

Finding ID: pipeline:ado:dnceng-public:dotnet-dotnet:build:failed
Severity: ⚠️ warning
Correlation: hc-2026-04-23-2
Executive Summary: Build 1390567 failed on two fronts — (1) the src/runtime/build.sh subprocess exited non-zero (primary, root cause unclear without full logs), and (2) a SIGN004 signing policy violation in ContainerSigning was simultaneously active and has since been fixed by PR dotnet#6191 (merged same day). The 14-day pass rate of 21.4% indicates a chronic underlying issue, most likely in the runtime build leg.

Root Cause

The build failed in two independent steps:

Step 1 — Build: repo-projects/Directory.Build.targets(622,5): error MSB3073 — the VMR orchestrator invoked src/runtime/build.sh --restore ... and the subprocess returned a non-zero exit code. The full error is truncated in the collected data. The most recent runtime codeflow in main before the build started was PR #6162 (merged 2026-04-21T07:52 UTC), bringing runtime build 20260420.20 (commit c87e7f8 from dotnet/runtime), with 283 changed files (+11095/−5157). A restore-phase NuGet failure or a compilation regression in that runtime drop are the most likely causes.

Step 2 — Sign Source-Built Artifacts: Sign.proj(74,5): error SIGN004: Signing 3rd party library in artifacts/tmp/Release/ContainerSigning/118. This error indicates that a non-Microsoft-owned binary ended up in the container signing bucket. The root cause is that repositories defining vs in global.json defaulted to desktop (VS) MSBuild for SigningValidation, which incorrectly classifies container artifacts. This was fixed the same day by PR #6191 "Force the use of dotnet msbuild for SigningValidation" (merged 2026-04-22T13:15 UTC).

Confidence: Medium — The signing failure root cause is confirmed (PR dotnet#6191 fixes it). The runtime build failure root cause cannot be confirmed without full build logs; the 21.4% 14-day pass rate suggests it is likely a recurring infrastructure or NuGet-feed issue rather than a single code regression.

Blast Radius

• main branch only for this specific build (build ID 1390567, 2026-04-22).
• The signing fix (PR Force the use of dotnet msbuild for SigningValidation dotnet/dotnet#6191) was applied the same day and should resolve the SIGN004 failure for subsequent builds.
• The 21.4% 14-day pass rate indicates the runtime build failure is likely affecting most recent main builds (and potentially other branches), blocking integration of runtime, aspnetcore, roslyn and other upstream changes.
• The signing issue may affect all branches that have vs in their global.json until they pick up the fix.

Suggested Fix

1. Retry the main build now that PR Force the use of dotnet msbuild for SigningValidation dotnet/dotnet#6191 (SigningValidation fix) is merged. If the SIGN004 error was the root cause for this step's failure and the runtime restored cleanly this time, the build may pass.
2. Investigate the runtime build failure — check the full ADO build log at (dev.azure.com/redacted) to determine whether the runtime restore (src/runtime/build.sh --restore) failed due to a NuGet feed issue or a compilation error introduced in runtime codeflow PR #6162 (commit c87e7f8).
3. Address the 21.4% pass rate — open a tracking operational issue (area-unified-build-OperationalIssue) if not already present, and escalate to @dotnet/product-construction to investigate whether this is a feed availability issue, an OOM/timeout on the runtime build leg, or a persistent code regression.

Evidence

# Failed steps from build 1390567 (2026-04-22T05:00–10:46 UTC, duration ~5h46m)

## Step: Build
repo-projects/Directory.Build.targets(622,5): error MSB3073:
  (NETCORE_ENGINEERING_TELEMETRY=Build) The command
  /__w/1/s/src/runtime/build.sh --restor[e ...]
  [truncated — full error requires ADO log]

## Step: Sign Source-Built Artifacts  
artifacts/toolset/11.0.0-beta.26210.111/Sign.proj(74,5): error SIGN004:
  Signing 3rd party library
  /__w/1/s/artifacts/tmp/Release/ContainerSigning/118[...]

Most recent runtime codeflow before failing build:

• PR #6162 merged 2026-04-21T07:52 UTC
• Runtime build 20260420.20, commit c87e7f8
• 283 files changed, +11095/−5157 lines

Signing fix (merged same day as failure):

• PR #6191: "Force the use of dotnet msbuild for SigningValidation" by @ViktorHofer (merged 2026-04-22T13:15 UTC)
• "Resolves errors when devdiv pipelines use older VS images" — 1-line change to force dotnet msbuild over VS MSBuild in SigningValidation

14-day pass rate:

• 21.4% (approximately 3 passing builds out of 14 — chronic failure pattern)

Related

• PR #6162 — Runtime codeflow (most recent before build, 283 files)
• PR #6191 — SigningValidation fix (already merged)
• ADO build: (dev.azure.com/redacted)
• Existing operational issues: tvOS timeout (#5609-class) — search for runtime build or SIGN004 patterns

---

_{🔍 Investigation Run #24819448094 · Dispatched by health check · hc-2026-04-23-2}

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 767K · ◷

[github-actions]

🔍 Investigation: dotnet-dotnet public CI 14-day pass rate critical (21.4%)

Finding ID: pipeline:ado:rolling-build:pass-rate:critical
Severity: 🔴 critical
Correlation: hc-2026-04-23-1
Executive Summary: The public dotnet-dotnet CI pipeline on main is failing at a critical rate (10/14 builds over 14 days) due to two compounding failures: (1) the src/runtime/build.sh build script exiting with non-zero status (MSB3073), and (2) a SIGN004 error where a newly-introduced 3rd party library is being signed with a Microsoft certificate — a known, previously-resolved pattern.

Root Cause

Two failures are compounding to produce the critical pass rate:

Failure 1 — Runtime Build Script Failure (MSB3073, primary):
repo-projects/Directory.Build.targets line 622 executes src/runtime/build.sh via <Exec Command="$(FullCommand)">. The command is failing (non-zero exit) and MSBuild surfaces this as MSB3073. The error is truncated at --restor, indicating the restore or initial build phase of the runtime is failing. This is a persistent failure pattern (~12/14 days) suggesting a systematic issue rather than a transient flake — likely introduced around April 9. No currently-open operational issue specifically tracks this main branch runtime build failure.

Failure 2 — SIGN004 Third-Party Library Signing (secondary):
The Sign Source-Built Artifacts step fails with error SIGN004: Signing 3rd party library /__w/1/s/artifacts/tmp/Release/ContainerSigning/118 — a new 3rd party dependency is being signed with the MicrosoftDotNet500 certificate when it should use a 3rd party certificate (or be excluded). This is an exact match to prior issues #5359 (Antlr4.Runtime.Standard.dll, Mar 2026) and #3199 (Spectre.Console.dll, Nov 2025) — both resolved by adding signing configuration for the new library.

Confidence: Medium — The runtime build failure (MSB3073) is confirmed but the root cause within runtime/build.sh is unknown without full log access (ADO access required). The SIGN004 pattern is High confidence: it exactly matches previously closed issues and has a well-established fix path.

Blast Radius

• main branch public builds: Critically degraded — 21.4% pass rate, blocking codeflow validation
• Codeflow PRs to main: All codeflow PRs (from runtime, roslyn, aspnetcore, etc.) are validated against this pipeline; persistent failures reduce confidence in merge safety
• Signed artifact delivery: The SIGN004 error prevents the signing step from completing, blocking signed artifact publication
• Downstream consumers: Any team depending on fresh signed SDK artifacts from main is affected

Suggested Fix

1. Investigate the runtime build failure: Access [build 1390567]((dev.azure.com/redacted) in Azure DevOps and expand the Build step logs for src/runtime/build.sh. Look for the actual error (compilation failure, OOM, restore error, or package conflict) before the MSB3073. Check if the failure started correlating with a specific runtime codeflow PR merged to main around April 9–10.

2. Fix the SIGN004 signing error: Identify the 3rd party library at artifacts/tmp/Release/ContainerSigning/118 (check the full ADO log for the Sign Source-Built Artifacts step — the filename will be there). Then follow the same fix pattern as #5359: either add the library to the signing exclusion list in the Arcade SDK's signing config, or assign the correct 3rd party certificate. File this as an operational issue with area-unified-build-OperationalIssue if the specific library is a new dependency from a recent codeflow.

3. Retry builds after fixing: Once the signing config is corrected, retry recent failed codeflow PRs to main to validate the fix. If the runtime build failure is a separate regression, file a targeted issue against dotnet/runtime with the specific error from the full logs.

Evidence

# Context JSON (pre-collected ADO data)
{
  "pipeline": "dotnet-dotnet",
  "branch": "main",
  "pass_rate_14d": 21.4,
  "total_14d": 14,
  "succeeded_14d": 3,
  "failed_14d": 10,
  "latest_build_result": "failed",
  "latest_build_start": "2026-04-22T05:00:53Z",
  "latest_build_finish": "2026-04-22T10:46:40Z"
}

# Failure 1 — Build step (MSB3073)
repo-projects/Directory.Build.targets(622,5): error MSB3073:
  (NETCORE_ENGINEERING_TELEMETRY=Build) The command
  /__w/1/s/src/runtime/build.sh --restor[e...] exited with code <non-zero>

# Directory.Build.targets line 622 (confirmed in VMR):
<Exec Command="$(FullCommand)"
      WorkingDirectory="$(ProjectDirectory)"
      EnvironmentVariables="@(EnvironmentVariables);@(BuildEnvironmentVariable)"
      YieldDuringToolExecution="true">

# Failure 2 — Sign step (SIGN004)
artifacts/toolset/11.0.0-beta.26210.111/Sign.proj(74,5): error SIGN004:
  Signing 3rd party library /__w/1/s/artifacts/tmp/Release/ContainerSigning/118[/<lib>.dll]
  with Microsoft certificate 'MicrosoftDotNet500'.

# Prior SIGN004 issues (same pattern, previously resolved):
# #5359 — Antlr4.Runtime.Standard.dll (closed 2026-03-10)
# #3199 — Spectre.Console.dll (closed 2025-11-03)

Recent codeflow activity (main branch, ~24h window):

• #6196 — [main] Source code updates from dotnet/runtime (merged 2026-04-23T00:12:51Z, AFTER failed build)
• #6097 — [main] Source code updates from dotnet/aspnetcore (merged 2026-04-23T03:55:38Z)
• #6150 — [main] Source code updates from dotnet/roslyn (merged 2026-04-23T03:58:11Z)

Note: The failed build started at 05:00 UTC on April 22 — before the April 22 14:43 runtime codeflow. The persistent 14-day failure rate implies the root cause predates these recent merges.

Related

• dotnet/dotnet #5359 — SIGN004: Antlr4.Runtime.Standard.dll signing (closed, same error class)
• dotnet/dotnet #3199 — SIGN004: Spectre.Console.dll signing (closed, same error class)
• dotnet/dotnet #3340 — Intermittent signing timeout on Linux (closed, different signing failure)
• [ADO Build 1390567]((dev.azure.com/redacted) — Latest failed build (primary investigation target)

---

_{🔍 Investigation Run #24819444988 · Dispatched by health check · hc-2026-04-23-1}

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 618.2K · ◷

[github-actions]

📋 Health Check — 2026-04-24

🆕 6 new · ✅ 0 resolved · 📌 0 unchanged

New findings:

• 🔴 dotnet-dotnet public CI 14-day pass rate critical (21.4%) — pipeline:ado:rolling-build:pass-rate:critical
• 🔴 Codeflow PRs blocked >3 days (4 PRs, oldest: ~50 days) — infra:codeflow-backlog:critical
• 🔴 Multiple operational issues stale >3 days (19 open) — infra:stale-operational-issues:critical
• 🟡 dotnet-dotnet public CI failed on main (1/1 builds, 24h) — pipeline:ado:dnceng-public:dotnet-dotnet:build:failed
• 🟡 20 codeflow PRs open simultaneously (>5 threshold) — infra:codeflow-backlog:warning
• 🟡 19 open operational issues (>10 threshold) — infra:stale-operational-issues:warning

Resolved: None

Investigations dispatched: 2 (pass rate critical, codeflow blockage critical)

Full report →

Generated by DevOps Daily Health Check · ● 672.3K · ◷

[github-actions]

🔍 Investigation: dotnet-dotnet public CI 14-day pass rate critical (21.4%)

Finding ID: pipeline:ado:rolling-build:pass-rate:critical
Severity: 🔴 critical
Correlation: hc-2026-04-24-1
Executive Summary: Two concurrent independent failures are driving the 21.4% pass rate: (1) a recurring SIGN004 signing misconfiguration for a 3rd party assembly in ContainerSigning/118 using Arcade SDK toolset 11.0.0-beta.26210.111, and (2) an ongoing intermittent npm ci crash (exit code 57005 / 0xDEAD) in the aspnetcore build step — the aspnetcore team is actively adding diagnostics but has not yet shipped a fix.

Root Cause

Two independent failure modes are present in the latest build [1392514]((dev.azure.com/redacted)

Failure 1 — SIGN004 Signing (Sign Source-Built Artifacts):
The Arcade signing task (Sign.proj) is rejecting a 3rd party assembly at ContainerSigning/118 because it carries a missing or 3rd-party copyright string but is being signed with the MicrosoftDotNet500 Microsoft certificate. This exact pattern has occurred twice before: Spectre.Console.dll (dotnet#3199, fixed Nov 2025) and Antlr4.Runtime.Standard.dll (dotnet#5359, fixed Mar 2026). In each case a new dependency was introduced into source-build output without updating the signing exclusion/override configuration. The current toolset is 11.0.0-beta.26210.111 and no open tracking issue exists for this recurrence.

Failure 2 — MSB3073 aspnetcore/eng/build (Build step):
The aspnetcore build script exits non-zero due to npm ci crashing with exit code 57005 (0xDEAD) with zero diagnostic output. The upstream tracking issue is dotnet/aspnetcore#62807. As of April 24 2026, the aspnetcore team has merged diagnostics improvements (#66447 — verbose npm logging, pre-flight checks) and re-enabled binary logging (#66448), but a root-cause fix has not yet shipped. Historical analysis shows this failure affects ~12/30 scheduled builds intermittently. A prior related issue (dotnet#5800, npm ETARGET for @microsoft/signalr) was closed April 1 2026; the current 0xDEAD crash is a distinct but related npm instability.

Confidence: Medium — The SIGN004 pattern is well-established (two prior identical occurrences with known remediation). The aspnetcore npm failure is confirmed by upstream repo activity and open tracking issue, but the precise trigger for the crash remains undiagnosed by the aspnetcore team.

Blast Radius

• SIGN004: Blocks all source-build-enabled legs that include the affected ContainerSigning batch. Affects public main builds and likely internal main builds. Does not affect non-source-build legs.
• aspnetcore MSB3073: Affects all Linux legs of the public (and likely internal) rolling build where npm ci is invoked. Windows and macOS (non-arm64) legs appear unaffected based on the prior [Operational Issue]: aspnetcore npm build fails — @microsoft/signalr@>=11.0.0-dev not published (ETARGET) dotnet/dotnet#5800 pattern. Impact: ~40–50% of build legs per run fail.
• Combined: 10/14 builds failed in the past 14 days (pass rate 21.4%), which is critical for release readiness.

Suggested Fix

1. SIGN004 (immediate): Identify the specific 3rd party assembly in ContainerSigning/118 by inspecting the signing log in build [1392514]((dev.azure.com/redacted) Update the signing exclusion list in src/<owner-repo>/eng/Signing.props (or the relevant repo's eng/Signing.props) to either exclude the file or assign the correct 3rd party certificate. File a dotnet/dotnet operational issue with label area-unified-build-OperationalIssue. Reference the prior fixes for Antlr4 (SIGN004: Antlr4.Runtime.Standard.dll signed with Microsoft certificate in source-build offline dotnet/dotnet#5359) and Spectre.Console (Signing 3rd party library with Microsoft certificate 'MicrosoftDotNet500' dotnet/dotnet#3199) as templates.

2. aspnetcore npm crash (short-term): Monitor the aspnetcore upstream for a fix to dotnet/aspnetcore#62807. The verbose-logging commit (#66447) should surface root cause in the next failing build — review the new diagnostic output. Ping @wtgodbe (active on these commits) for status.

3. Retry / unblock: Once the SIGN004 fix is merged, manually re-trigger the failing builds to separate the two failure modes and confirm which legs remain broken due to the npm issue.

Evidence

# context_json fail_tasks:
"Sign Source-Built Artifacts: SIGN004: Signing 3rd party library ContainerSigning/118 (toolset 11.0.0-beta.26210.111)"
"Build: MSB3073 src/aspnetcore/eng/build NETCORE_ENGINEERING_TELEMETRY=Build"

# 14-day stats:
pass_rate: 21.4%  |  total: 14  |  succeeded: 3  |  failed: 10

# Prior SIGN004 issues (closed):
dotnet/dotnet#5359 — Antlr4.Runtime.Standard.dll signed with MicrosoftDotNet500 (Mar 2026)
dotnet/dotnet#3199 — Spectre.Console.dll signed with MicrosoftDotNet500 (Nov 2025)

# aspnetcore upstream activity (Apr 24 2026):
commit f951446 — "Add verbose logging and diagnostics for npm ci failures (#66447)"
  "The npm ci command intermittently fails with exit code 57005 (0xDEAD) in CI,
   producing zero diagnostic output."
  "Relates to https://github.com/dotnet/aspnetcore/issues/62807"
  12/30 scheduled builds affected

commit 441706c — "Enable binary logging in quarantined-tests pipeline (#66448)"
  References "Build FAILED, 0 Warnings, 0 Errors" silent failures

# Prior aspnetcore npm issue (closed Apr 1 2026):
dotnet/dotnet#5800 — aspnetcore npm ETARGET `@microsoft/signalr`@>=11.0.0-dev
  (distinct failure mode, but same build step)

Related

• dotnet/dotnet#5359 — Prior SIGN004 (Antlr4), closed Mar 2026
• dotnet/dotnet#3199 — Prior SIGN004 (Spectre.Console), closed Nov 2025
• dotnet/dotnet#5800 — Prior aspnetcore npm ETARGET failure, closed Apr 2026
• dotnet/aspnetcore#62807 — Upstream npm ci crash tracking issue (open)
• [ADO Build 1392514]((dev.azure.com/redacted) — Latest failing build

---

_{🔍 Investigation Run #24874847710 · Dispatched by health check · hc-2026-04-24-1}

Note

🔒 Integrity filter blocked 7 items

The following items were blocked because they don't meet the GitHub integrity level.

• #6227 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #6097 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #6033 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #5986 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #5967 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #5942 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 406K · ◷

[github-actions]

🔍 Investigation: Codeflow PRs blocked >3 days (4 PRs, oldest: ~50 days)

Finding ID: infra:codeflow-backlog:critical
Severity: 🔴 critical
Correlation: hc-2026-04-24-2
Executive Summary: All 4 blocked codeflow PRs require manual intervention — 3 are stuck on unresolved merge conflicts (requiring darc vmr resolve-conflict) and 1 is blocked by a source-build failure on the release/11.0.1xx-preview3 branch.

Root Cause

Each PR has a distinct but related blocker:

PR	Repo	Branch	Age	Root Cause
#5229	dotnet/diagnostics	main	~50 days	Merge conflict in .devcontainer/devcontainer.json — Codeflow verification check failing since 2026-03-23. No update in 32+ days.
#5743	microsoft/vstest	release/11.0.1xx-preview3	~27 days	Source-build failure — VMR Source-Only Build SB_CentOSStream10_Online_MsftSdk_x64 failed ([build #1364124]((dev.azure.com/redacted) All Checks Successful is failing. 90+ vstest upstream PRs are accumulating in this single stale codeflow PR.
#6106	dotnet/symreader	release/10.0.1xx	~6.5 days	Conflict + opposite codeflow: Conflict detected at PR creation; subsequently an opposite codeflow (VMR→symreader) merged on 2026-04-21, further blocking this PR. Build checks all pass.
#6111	dotnet/sourcelink	release/10.0.1xx	~6.5 days	Same pattern as dotnet#6106: Conflict at creation + opposite codeflow merged on 2026-04-21. Build checks all pass.

Confidence: High — All 4 PRs show mergeable_state: "blocked", and the Maestro bot comments directly identify the conflict/build-failure cause on each PR with actionable remediation instructions.

Blast Radius

• dotnet/diagnostics → main: Source updates from diagnostics are not flowing into the VMR main branch. Any diagnostics fixes are not reflected in the VMR for 50 days.
• microsoft/vstest → release/11.0.1xx-preview3: The preview3 release branch is missing ~90 upstream vstest PRs. The source-build failure may also affect other PRs targeting this branch.
• dotnet/symreader + dotnet/sourcelink → release/10.0.1xx: Two repos are blocked on the same release/10.0.1xx branch. The opposite codeflow that merged on 2026-04-21 means additional changes are now pending on top of these unresolved PRs.

Suggested Fix

1. PR [main] Source code updates from dotnet/diagnostics dotnet/dotnet#5229 (diagnostics/main — CRITICAL, 50 days): Manually resolve the conflict using:

   git clone https://github.com/dotnet/dotnet && cd dotnet
   ./eng/common/darc-init.sh
   darc vmr resolve-conflict --subscription fffd23ac-cce6-400c-919a-5fed1f9cb1f5
   # Resolve .devcontainer/devcontainer.json conflict, commit & push

2. PR [release/11.0.1xx-preview3] Source code updates from microsoft/vstest dotnet/dotnet#5743 (vstest/release/11.0.1xx-preview3 — HIGH, 27 days): Investigate the source-build failure at [ADO build #1364124]((dev.azure.com/redacted) (job: SB_CentOSStream10_Online_MsftSdk_x64). Given the large vstest diff (534 changed files, +11k/−6k lines), a vstest-introduced source-build incompatibility is likely. Consider closing this PR and triggering a fresh codeflow once the upstream issue is fixed: darc trigger-subscriptions --id 72a1750f-05fe-4d7d-b9b7-b11b6fa82f2b.

3. PRs [release/10.0.1xx] Source code updates from dotnet/symreader dotnet/dotnet#6106 & [release/10.0.1xx] Source code updates from dotnet/sourcelink dotnet/dotnet#6111 (symreader + sourcelink/release/10.0.1xx — MEDIUM, 6.5 days): Both have build checks passing (All Checks Successful: ✅) but are stalled on Codeflow verification. Per Maestro's guidance, either:

   • Merge as-is (Maestro will handle the opposite codeflow in a follow-up PR), or
   • Close and re-trigger: darc trigger-subscriptions --id 5d5c84f6-685a-48f4-98a9-282076fd8553 (symreader) and darc trigger-subscriptions --id 87b56f49-8b6f-4357-83d2-575263b3649d (sourcelink)

Evidence

PR dotnet#5229 — Maestro conflict comment (2026-03-23):

⚠️ Action Required — Conflict detected
A conflict was detected when trying to update this PR with changes from build 304113
of dotnet/diagnostics. Conflicts in: .devcontainer/devcontainer.json
```

**PR #5743 — Failed check:**
```
dotnet-unified-build (VMR Source-Only Build SB_CentOSStream10_Online_MsftSdk_x64): FAILURE
Build: (dev.azure.com/redacted)
```

**PRs #6106 & #6111 — Maestro opposite-codeflow comment (2026-04-21):**
```
⚠️ While this PR was open, the source repository has received code changes from this
repository (an opposite codeflow merged). The codeflow cannot continue until this PR
is closed or merged.

All 4 PRs: mergeable_state: "blocked", opened by dotnet-maestro[bot], no human activity on oldest PRs in 30+ days.

Related

• Open Codeflow PRs (20 total open, 4 blocked >3 days)
• Codeflow PR documentation
• Existing operational issues reviewed: none directly related to this codeflow backlog pattern

---

_{🔍 Investigation Run #24874850869 · Dispatched by health check · hc-2026-04-24-2}

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• VMR v10.0.201 uses shared components from 10.0.3. Official binaries use 10.0.5 dotnet/dotnet#5915 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by DevOps Health — Deep Investigation · ● 372.4K · ◷