📈 CI Daily Pulse

[radical]

🔴 main red at tip · 🔴 release/14.0 red at tip · 🔴 outerloop red at tip · 🔴 internal main red at tip · 🔴 PR completed tip red (47%) — 6 items for a dev

🔴 For a dev — FluentUI package validation is repeatedly breaking rolling main

What: The current FluentUI package fails NuGet signature validation during Windows restores. The failure is currently deterministic across four consecutive rolling pushes.

Do: A dependency/feed dev should fix the package timestamp/signature or publish a consumable version to the mapped feed. failed run · Directory.Packages.props:10

• 🔴 Homebrew cask validation intermittently fails because the generated cask uses a deprecated parameter; it failed 2 of the last 4 pushes and is red at the tip (run).
• 🔴 Outerloop dashboard checks still fail broadly on both OSes after all four attempts (run).
• 🔴 Maven still cannot compile the staged Java sibling on Linux or Windows after all four attempts (run).
• 🔴 Agent initialization still rejects the Playwright CLI package provenance after all four attempts (run).
• 🔴 The Redis starter sample still never exposes its expected dashboard resources after all four attempts (run).

_{The first release/14.0 run still has two single-run failures, internal main has reached ten consecutive failures with its cause unconfirmed, and the latest completed PR run is red while a fresh bot rerun is active — see § 🧩 below.}

_{Window: Wed Sep 9 09:55 → Thu Sep 10 21:55 UTC · deltas vs 7-day average · Sep 10, 2026.}

📊 Last 36h by lane — activity, pass rate & Δ vs 7d

Lane	36h activity	36h pass rate (Δ vs 7d)	Latest build
🌳 GH CI — main	🟩🟥🟥🟥🟥🟥🟥	🔴 8% (1/12) · ▼ 4.2	🔴 Sep 10 19:14
🌿 GH CI — release/14.0	🟥🟥	🔴 0% (0/1, 95% CI 0–79%) · ▬ 0.0	🔴 Sep 10 17:25
🧪 GH CI outerloop — main	🟥🟥	🔴 0% (0/1, 95% CI 0–79%) · ▬ 0.0	🔴 Sep 10 02:11
🏗️ Internal (AzDO) — main	🟥🟥🟥🟥🟥🟥	🔴 0% (0/9, 95% CI 0–30%) · ▼ 43.5	🔴 Sep 10 19:33
🔀 GH CI — PR validation ⭐	🟩🟩🟩🟩🟩🟥🟥🟥🟥🟥🟥	🔴 47% (34/73) · ▼ 17.3	🔴 Sep 10 20:31

🧱 infra share: unavailable this run. Internal main has ten consecutive failed builds and remains red at the tip at build 3072363.

_{bar length = sample size / confidence (short = read its % with care) · Δ vs 7d = 36h rate minus the lane's 7-day average · Latest build = red/green of the lane's most recent completed in-window run, linked to it · lane names link to the workflow / pipeline.}

🧩 What's broken & why

Six things need a human: rolling main has two repeatable package/installer breaks, and four independent outerloop families exhausted every retry.

	What's broken	Who's on it	Since
🔴	The current FluentUI package fails restore validation in four consecutive rolling main runs.	Needs a dependency/feed dev — no auto-rerun on rolling	repeated in the last 4 pushes; red at tip
🔴	The generated Homebrew cask intermittently fails audit because it emits a parameter Homebrew has deprecated (run).	Needs an installer dev — no auto-rerun on rolling	failed 2 of the last 4 pushes; red at tip
🔴	Dashboard UI checks remain broadly broken on both OSes in the fourth-attempt outerloop jobs.	Needs a dashboard dev — all outerloop retries exhausted	last failed ~20h ago; tip still red
🔴	Maven cannot compile the staged Java sibling on Linux or Windows in the outerloop run.	Needs a CLI/Java dev — failed through all four attempts	last failed ~20h ago; tip still red
🔴	Agent initialization rejects the upstream Playwright CLI package across the outerloop checks.	Needs a CLI/package dev — tracked in #19159; retries exhausted	last failed ~20h ago; tip still red
🔴	The Redis starter sample starts, but its expected resources never appear in the dashboard in the outerloop run.	Needs a templates/runtime dev — retries exhausted	last failed ~20h ago; tip still red
🟡	The first release/14.0 run passed all Kafka tests but its Windows test process exited abnormally afterward (job).	Watch — too early to call; no auto-rerun on rolling release	first seen ~4.5h ago; red at tip
🟡	The same release/14.0 run timed out waiting for a Deno debugger breakpoint (job).	Watch — too early to call; no auto-rerun on rolling release	first seen ~4.5h ago; red at tip
🟡	Internal main is red at the tip after ten consecutive failed builds.	Watch — cause unconfirmed this run	latest finished ~47m ago
🟡	One PR turns package vulnerability warnings into build errors across many jobs after exhausting four attempts (run).	PR author owns it; the rerun bot is already trying the newer PR tip	last completed ~1.5h ago

🧱 Product vs infra: rolling main is blocked by the FluentUI package signature and a Homebrew cask incompatibility, while outerloop still has four reproducible product/test failures after all retries. The release failures look flaky or environmental, internal causes remain unconfirmed, and the latest completed PR failure is isolated to that PR's audit configuration.

⏱️ Rolling main is also slower today: build-time p50 is ~52m versus ~36m over the 7-day baseline.

exact tests, jobs & counts

🔴 Rolling GH CI — main package restore — 1/12 completed runs passed in the 36h window. The FluentUI failure reproduced in four consecutive pushes:

• Latest run 34519184641, commit 9c2af96f: Aspire.Dashboard.Tests.csproj failed with NU3027 because Microsoft.FluentUI.AspNetCore.Components 5.0.0-preview.26251.2 was not timestamp-signed for long-term validity. The restore also reported a missing dictionary key for SQLitePCLRaw.core 3.0.5. The FluentUI version is set at Directory.Packages.props:10.
• Run 34519005883 hit the same NU3027 failure in Aspire.Hosting.RemoteHost.Tests.csproj.
• Run 34512081344 hit the same NU3027 failure in Aspire.Hosting.RemoteHost.Tests.csproj; its restore also saw one transient Azure DevOps 503.
• Run 34508764231 failed on macOS and Linux with NU1100 for the FluentUI component and icon versions and on Windows with the same NU3027 signature rejection.
• The four triggering commits changed unrelated workflow, test, Front Door, or dev-tunnel areas, strengthening the package-feed/signature diagnosis.

🔴 Rolling GH CI — main Homebrew validation — cask preparation failed in 2 of the last 4 pushes:

• Runs 34519184641 and 34512081344 both failed brew audit --except=installed --tap=local/aspire.
• Runs 34519005883 and 34508764231 passed the same job, so this is intermittent while runner Homebrew versions converge, not deterministic.
• Verbatim error: the 'verified' parameter has been deprecated; use the 'url' stanza without it.
• The generated stanza comes from eng/homebrew/aspire.rb.template:8-9.

🔴 Outerloop dashboard — attempt 4 failed 40/53 tests on Linux and 40/53 on Windows; both dashboard jobs failed in attempts 1, 2, 3, and 4.

• Representative timeout: Aspire.Dashboard.Tests.Integration.Playwright.AppBarTests.AppBar_Change_Theme timed out after 30 seconds at AppBarTests.cs:24-29.
• Other failures span app-bar layout, authentication redirects, resource menus, accessibility surfaces, focus indicators, and viewport checks on both OSes. Representative errors include Locator expected to be visible, No control part found for fluent-button, and 30-second operation timeouts.

🔴 Outerloop Maven/JDK — Aspire.Cli.Tests.Projects.JavaAppHostToolchainResolverTests.ApplyToRuntimeSpec_ForMavenReactor_StagesAnUninstalledSiblingFromAnEmptyLocalRepository failed 1/13 tests on Linux and 1/13 on Windows in attempt 4; both CLI jobs failed in all four attempts.

• Verbatim error: Fatal error compiling: error: release version 21 not supported.
• Assertion: JavaAppHostToolchainResolverTests.cs:406.

🔴 Outerloop agent initialization — 3/15 CLI end-to-end tests failed in attempt 4; the job failed in all four attempts.

• Tests: AspireNew_WithAgentInit_InstallsPlaywrightWithoutErrors, AgentInit_InstallsPlaywrightCli_AndGeneratesSkillFiles, and AgentInit_CwdDiffersFromRoot_PlacesSkillsInWorkspaceRoot.
• Verbatim error: Provenance verification failed for @playwright/cli@0.1.19: AttestationFetchFailed.
• Assertions: NewWithAgentInitTests.cs:112-125, PlaywrightCliInstallTests.cs:55-61, and PlaywrightCliInstallTests.cs:110-116. Tracking issue: #19159.

🔴 Outerloop Redis starter template — all 3 selected tests failed in attempt 4; the templates job failed in all four attempts.

• Tests: ResourcesShowUpOnDashboard and both ApiServiceWorks variants on StarterTemplateWithRedisCacheTests.
• Verbatim error: Expected rows not found: apiservice, webfrontend, cache.
• Assertion: TemplateTestsBase.cs:162-257.

🟡 release/14.0 Windows Kafka job — the job reported Passed! for all 492 tests (474 succeeded, 18 skipped, 0 failed), then three post-test commands exited -1073741502. This is one run with no product-test failure, so it is too early to call a recurring break.

🟡 release/14.0 Deno debugger E2E — the job had 0 passing and 1 failing test:

• attaches js-debug to a Deno resource and hits a TypeScript breakpoint timed out after 240000 ms waiting for main.ts:2, with Last error: <none>.
• Test setup and assertions: denoDebugger.e2e.test.ts:24-76.
• An earlier ChromeDriver metadata request hit ECONNRESET but succeeded on retry, so that recovered download is not the final test failure.

🟡 Internal (AzDO) — main had 0 succeeded or partially succeeded builds out of 9 completed in-window. Builds 3071054, 3071262, 3071391, 3071497, 3071585, 3071656, 3072001, 3072274, and 3072363 failed; build 3070652 immediately before the window makes ten consecutive failures. Aggregate counts only per internal-privacy policy; causes were not inspected.

🟡 PR validation — 39/73 counted runs were red in the 36h window. The latest completed run, 34526846888 for #20052, exhausted four attempts because NU1902/NU1903 package-vulnerability warnings became errors across many jobs. A newer PR run is currently in progress on its second bot-triggered attempt.

_{Attention dots: 🔴 needs a human now · 🟡 keep an eye on it · ⚪ looks resolved. These are distinct from the lane table's 🔴/🟢 run-outcome dots.}

🔁 Reruns & flaky tax

The auto-rerun bot carried 147 of 154 reruns and rescued 17 PR runs. Human-only reruns rescued none, so PR flakiness remains overwhelmingly automated toil with manual tax at the edge.

rerun actor split (36h)

🤖 Auto-rerun bot: 147 reruns across 62 PR runs → 17 rescued red→green; 36 still red, 9 cancelled.

🧑 Manual (human) reruns: 7 reruns across 3 PR runs (1 branches the bot also reran) → on the 2 human-only branches: 0 rescued, 2 still red, 0 cancelled.

🔗 Combined: 65 distinct PR runs needed a rerun in the window (18 went green, 38 still red, 9 cancelled); 1 PR branches needed both bot + human.

_{Sources: gh api .../ci.yml/runs (pass rates + rerun split, fetched in bounded UTC-day buckets), az pipelines build list def 1602 (internal, aggregate counts only), and failed GitHub job logs for live-cause confirmation. ciinsights was unavailable in this session, blocking fresh failure-cluster and infra-share joins; all visible causes were confirmed directly from GitHub job data and logs. Window = last 36h. Backlog & trend: see the weekly CI Health report (#18231). Generated Sep 10, 2026 21:55 UTC.}

[joperezr]

🤖 Aspire triage pass — 2026-06-16 06:54 UTC

Area: �rea-engineering-systems
Kind: docs (informational CI health report — [pipeline trial] per title)
Repro: n/a — this is a generated dashboard, not a bug report
Affected version(s): n/a (CI health snapshot, window: last 36h ending 2026-06-16)
Severity: S4 — informational; no product regression. Body flags real CI breakage (SqlServer conformance cluster, outerloop on main red 7+ days), but those are tracked as separate test-failure issues.

Urgency: ⚪ backlog — no milestone
Reason: informational dashboard / pipeline trial output, not a fix candidate. CI health concerns it surfaces are already being tracked under �rea-engineering-systems work.

Suggested owner(s): Jose Perez Rodriguez (@joperezr) (score 6)
Evidence:

• joperezr merged Merge release/13.4 to main after v13.4.0 release #17817, Merge release/13.3 to main after v13.3.5 release (supersedes #17337) #17343 in �rea-engineering-systems (last 90d)
• Issue author (Ankit Jain (@radical)) is the originator of this pipeline trial and is excluded from owner suggestions per policy

Confidence: medium
Needs human?: no — informational; Ankit Jain (@radical) is driving the trial and can dismiss/close when the experiment is over.

Dry-run mode — no labels or milestone applied. Reply /triage apply to promote, or edit and reapply.

[radical]

🔧 Post-run review — 2026-06-18 19:48 UTC

First run of the new two-tier Pulse spec (aspire-ci-pulse.prompt.txt + shared aspire-ci-report-common.prompt.txt, both committed today). The prior body here was produced by the older combined prompt, so a straight diff looks like a big regression — most of it is the intended split, but two things are worth a human's eye.

Regressions vs previous run: none that are true regressions — but two intended discontinuities a watcher will notice:

• Dropped sections (intended): the trend PNG, the <details> per-day grid, and the old "Newly…" flaky block are gone. The new spec explicitly moves trend/grid/build-time to the Weekly (🩺 CI Health — Last 7 days #18231); the Pulse is now current-state-only. Lane labels also moved to the canonical 🌳🌿🧪🏗️🔀 set. Deliberate, not a regression.
• Pass rates jumped sharply (methodology correction, NOT a real swing): PR validation reads 83% (76/92) vs the prior body's 38% (76/200); main reads 30% (6/20) vs 26% (6/23). Same passing counts — the denominator shrank because the new shared counting rule excludes cancelled / in-progress / queued runs, and PR ci.yml has heavy cancellation (65 cancelled in 36h alone). The new numbers are spec-correct; the jump reflects the rule, not reality. Calling this out so nobody reads it as "CI suddenly healed overnight."

Readability / improvement ideas:

• The headline blockquote is dense (lane + 3 tests + rerun split + two healthy lanes + watch clause). Skimmable but near the limit — accept: consider trimming the "PR + internal are healthy" clause to a half-sentence, since the table shows it. Reject if the contrast is the point.
• The activity bars are in a fenced code block so the emoji columns align on every client — reads cleanly. Keep.

Data-gathering issues this run:

• ciinsights query_test_failures threw An error occurred invoking 'query_test_failures'; get_failure_summary worked and supplied the top breakages + infra/flaky counts, so the Pulse was unaffected. The connection status also reported a DB index-creation warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists) — API healthy, but flag if failure data ever looks stale.
• ciinsights test data can lag the last ~30 min (noted in-body); the rerun split is computed live from gh api .../ci.yml/runs, so it's current.
• First --paginate | jq pass silently produced per-page arrays (mangled counts); re-running with --slurp fixed it. The 36h ci.yml window held 194 runs across pages — a non-slurped read would have undercounted.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, infra-share rule. It says NN% (failed-infra/failed-total) / "share of failed builds", but the only figure ciinsights exposes is infrastructureFailureCount vs failingTestCount (test-result level: this run, 77/134 = 57%), not a build-level ratio. Pin the metric to one definition + one ciinsights field so successive runs don't silently switch denominators (build-level vs test-result-level). Small wording fix, durable payoff.

[radical]

🔧 Post-run review — 2026-06-18 20:48 UTC

Regressions vs previous run: none. The diff vs the 19:46 run is entirely genuine state movement — window slid ~1h, MongoDB ReplicaSet overtook the CLI TypeScript-apphost smoke as the single loudest test (11 vs 8), rerun split drifted (56/17 → 55/16), infra share recomputed (77/134 → 78/136), and 7d-avg deltas moved a fraction. No lost structure, no flipped statuses, no format churn. (Also trimmed a few trailing blank lines the prior body had accumulated.)

Readability / improvement ideas:

• (accept/reject) The headline now distinguishes the single loudest test (MongoDB ReplicaSet) from the loudest family (Aspire.Cli.*) in the 👀 Watch clause. This reads better than the prior run, which conflated the two. Suggest keeping this single-test-vs-family split as the headline convention.

Data-gathering issues this run (the valuable one):

• The rerun-split fetch via gh api .../ci.yml/runs?...&created=>2026-06-17... silently truncated to 82 of 169 runs on the first attempt because the > in the created filter was not URL-encoded — the shell/gh mangled the operator and the API ignored the lower bound, returning only one partial slice. I caught it only because I cross-checked against the gh run list count (94 completed PR runs in 36h) and against the API's own total_count (169). Without that reconcile, the rerun split would have been undercounted by ~50% (≈27 bot reruns instead of 55), badly distorting the flaky-tax section.
  • Fix that worked: URL-encode the operator (created=%3E2026-06-17T08%3A38%3A49Z) and --paginate, then assert jq -s length == .total_count.

Suggested prompt change (for a human to apply):

• File aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section (or the "Fetch-enough-then-filter" section, which today only warns about gh run list --limit undercounts, not the runs-API created filter). Add one line: "When using gh api .../ci.yml/runs with a created=>… bound, URL-encode the operator (%3E) and verify the fetched run count equals the response's total_count — an unencoded > is silently dropped and truncates the window, halving the rerun split." This is a recurring trap (it will bite every run that fetches the rerun split this way), so it clears the "helps every future run" bar.
• Minor: the prompt never states the infra-share denominator. I inferred infra / (infra + failing) from the prior run's 77/134; this run's 78/136 matches the same formula. Worth one explicit line in the "Infra-share line" section so the figure is reproducible and doesn't drift between runs/authors.

[radical]

🔧 Post-run review — 2026-06-18 21:48 UTC

Regressions vs previous run: none. The diff vs the 20:46 run is entirely genuine state movement — window slid ~1h; MongoDB ReplicaSet ticked 11→12; the CLI TypeScript-apphost smoke (LatestCliCanStartStableChannelTypeScriptAppHost) aged out of the 36h window and StartAsync_…GracefulSignaler… took the #3 slot; rerun split drifted (55/16 → 58/20); infra share recomputed (78/136 → 79/135); 7d-avg deltas moved a fraction as the baseline window also slid. No lost structure, no flipped statuses, no format churn.

Readability / improvement ideas: none new — the headline + first table still convey "what's on fire" without scrolling.

Data-gathering issues this run (the new one):

• ciinsights.query_test_failures failed outright this run (An error occurred invoking 'query_test_failures'), and get_connection_status reported the backend DB as unhealthy (index ... 'IX_TestResults_FullyQualifiedName_CapturedAt' already exists — a stuck server-side migration). I fell back to ciinsights.get_failure_summary (its topFailingTests array) for the §3 breakages list, which worked. Without that fallback the loudest-breakages section would have had no source.
• This is plausibly recurring while the DB stays in that migration state, so it clears the "helps every future run" bar.
• I avoided the prior run's gh api .../ci.yml/runs created=> truncation trap by fetching paginated pages (per_page=100&page=N, 6 pages) and filtering by timestamp in code instead of an API-side created bound — confirms that prior finding's value.

Suggested prompt change (for a human to apply):

• (new — common file, "Reruns & the flaky tax" / data-sources) Add one line: "If ciinsights.query_test_failures errors, fall back to get_failure_summary.topFailingTests for the loudest-breakages list (same fail-count ranking, narrower window control)." Codifies a working mitigation for a tool that was down this run.
• (carried forward from 20:46, still unapplied — restated here because that comment is now resolved) (a) "Fetch-enough-then-filter": warn that an unencoded > in a runs-API created=>… bound is silently dropped and halves the rerun split — URL-encode (%3E) and assert fetched count == total_count, or fetch by page and filter in code. (b) "Infra-share line": state the denominator explicitly — infra / (infra + failing) (this run 79/135) — so the figure is reproducible and doesn't drift between authors.

[radical]

🔧 Post-run review — 2026-06-18 22:52 UTC

Regressions vs previous run: none. Diff against the prior body (published ~1h earlier) is all genuine state movement — window slid forward ~1h, main 6/20→6/21, internal main 14/16→14/17, PR validation 82/98→89/104, latest failed internal build 3003323→3003488, rerun counts 58/32→59/33. No section, column, link, or legend was dropped; no frozen value moved (the Pulse has no frozen history); no status emoji flipped without a real change.

Readability / improvement ideas:

• (accept/reject) The Watch clause was sharpened from last run's broad Aspire.Cli.* to the specific graceful-shutdown Signaler…Escalates family (3 named tests, repeating). This is more actionable — a reader can grep three exact test names. Suggest keeping Watch clauses this specific going forward rather than naming a whole namespace.
• The report is skimmable in ~15s: headline → lane table → reruns → breakages. No change needed.

Data-gathering issues this run:

• recentTrends.last24h.newFailingTests is a names-only list with NO per-test counts, and a large block in it is often ONE build's worth of a whole test class failing once — not N independent signals. This run it listed ~33 DistributedApplicationPipelineTests.* entries; cross-checking via search_tests showed every one had totalRuns=1, failures=1, all at the same timestamp (2026-06-17T23:03:34) — a single-build setup/compile blip, not a repeating breakage. Had I trusted the raw block I'd have featured a phantom "loud new family." Always validate newFailingTests entries against search_tests/failCount before featuring them.
• PR-validation 7d denominator needed a much larger fetch than the common prompt's --limit 300. At ~90–150 runs/day, 300 runs only spanned ~2.5 days (back to Jun 16) — short of the 7-day window. Refetched at --limit 1200 (reached Jun 7). The "fetch-enough-then-filter" rule held, but the suggested 300 floor silently undercounts this lane's 7d average.
• ciinsights get_connection_status reported a DB index warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists); all queries still returned complete data. One-off, no impact this run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Fetch-enough-then-filter" section: add a note that for the 7-day average on PR validation specifically, --limit 300 is insufficient (covers only ~2–3 days); use --limit ~1200 (or paginate to the 7d start) for that lane. The 36h window is fine at 300.
• aspire-ci-report-common.prompt.txt (or the Pulse §3 / headline Watch guidance): add a caution that recentTrends.newFailingTests carries no counts and a bulk block is frequently a single failed run of one test class — verify against search_tests before treating it as a flaky/broken family or featuring it in the Watch clause.

[radical]

🔧 Post-run review — 2026-06-18 23:52 UTC

Regressions vs previous run: none. Diffed the prior body against this one — every section, column, link, the bar-length legend, and both <sub> lines are intact. The only changes are state-driven value updates (window slid ~1h, so the MongoDB ReplicaSet 36h count fell 12→8 as older fails aged out; internal main 82→88%, PR validation 86→85%, infra share 59→60%). Pulse freezes no history, so there is no frozen-cell drift to check.

Readability / improvement ideas:

• (accept/reject) The report can be built from one ciinsights call: get_health_summary returns both failureBreakdown (testFailures + infrastructureFailures + total → the 🧱 infra-share fraction) and topFailingTests (the §3 loudest-breakage list). This run, query_test_failures failed outright (server-side DB index error: IX_TestResults_FullyQualifiedName_CapturedAt already exists), but get_health_summary + get_failure_summary covered everything. Suggest the prompt name get_health_summary as the primary §1-infra/§3-breakages source so a future run doesn't get blocked on the flaky query_test_failures endpoint.

Data-gathering issues this run:

• Rerun split is methodology-sensitive and the prompt under-specifies it. The window moved only ~1h vs the prior run, yet bot reruns jumped 59→89 and human 20→23. Two unpinned choices drive this: (a) what counts as one "rerun" — I used Σ(run_attempt−1) per run, attributed to the latest-attempt actor; (b) how to identify a PR — 38 of 80 rerun-bearing ci.yml runs were fork PRs with an empty pull_requests array, so I fell back to head_branch as the PR-identity key (raising distinct-PR from 23 to 48, much closer to the prior run's 46). Without these pinned, two runs an hour apart can legitimately disagree by ~50%.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Reruns & the flaky tax" section: pin the counting unit and the join key explicitly — (1) a "rerun" = each attempt beyond the first, i.e. Σ(run_attempt−1); (2) attribute a run's reruns to its latest-attempt triggering_actor (github-actions[bot] = bot, else human); (3) when pull_requests is empty (fork PRs), key the PR by head_branch. This is additive — it doesn't contradict the existing "split by who triggered it" rule, just makes it reproducible run-to-run.
• aspire-ci-pulse.prompt.txt / common data sources: note get_health_summary as the single source for both the infra-share fraction and the loudest-breakage list, with query_test_failures flagged as currently erroring.

[radical]

🔧 Post-run review — 2026-06-19 18:52 UTC

Regressions vs previous run: none. All deltas vs the prior body reflect genuine state changes. Structure preserved (header, headline, single lane table, infra line, legend, rerun split, loudest breakages, footer). The PR-validation activity bar shrank 12→11 cells — expected, not a regression: bar length is log-scaled to sample size and the lane's 36h run count dropped (109→72). The Internal AzDO release/13.4 row was omitted correctly (0 builds in window) and the omission is annotated inline so a reader doesn't think the lane was dropped silently.

Readability / improvement ideas:

• (accept/reject) The headline now carries two distinct test clusters (analyzer pair + EF Core Enrich) plus the "loudest by count vs most-recently-active" nuance. It's accurate but dense — at ~5 lines it's near the edge of skimmable. Consider a soft cap: lead with the single loudest cluster, push the "most-recently-active" distinction to §3 only. Reject if the recency caveat is considered load-bearing for "what's on fire now."

Data-gathering issues this run:

• Rerun-split attribution is an approximation. I bucketed reruns by triggering_actor of the latest attempt and counted run_attempt − 1 reruns per run, all assigned to that one actor. A run rerun first by the bot then by a human (or vice-versa) is attributed entirely to whoever triggered last; mixed-actor runs are only partially captured (I could detect "needed both" = 1 only via PR-set overlap, not per-attempt). This likely explains some of the large magnitude swing from the prior run (bot 89/human 23 → bot 43/human 16) — part real drop, part method variance. The prompts do not pin an exact counting method, so two runs can compute this differently.
• Infra-share denominator is ambiguous. The common prompt defines 🧱 infra share = failed-infra/failed-total as "share of failed builds", but ciinsights returns failure-occurrence counts (infrastructureFailureCount vs failingTestCount). I used 62/102 (infra ÷ infra+test signals); the prior run used 83/138 — a different denominator entirely. The metric isn't reproducible run-to-run without pinning the source + denominator.
• ciinsights get_test_history returned empty for an exact FQN (...MSBuildTests.AspireExportAnalyzers...) that get_failure_summary simultaneously reported with 34 fails in the same 36h window. So I could not use it to attribute the loudest test to a specific lane; the headline characterizes the analyzer pair by suite/family from the summary alone, not by lane. get_flagged_tests(problemType=broken) also did not surface the 34-fail pair in its top rows (it ranks multi-context broken tests; the pair appears single-context), so neither tool corroborated a lane for it.
• ciinsights get_connection_status reported the API healthy but the DB "unhealthy" (a duplicate-index warning, IX_TestResults_FullyQualifiedName_CapturedAt). All queries still returned data; flagging in case it foreshadows stale/partial results on a future run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section: pin the exact rerun-counting method — e.g. "count reruns as Σ(run_attempt − 1) over completed ci.yml runs in the window; attribute each run's reruns to the triggering_actor of the latest attempt (a known limitation for mixed bot+human runs); bot = login ending [bot]." Without this, the bot-vs-human magnitudes are not comparable across runs.
• Same file, "Infra-share line" section: pin the denominator and source — state whether it's failed builds or failed signals, and that ciinsights failingTestCount / infrastructureFailureCount are the inputs (or specify the build-level alternative). This run (62/102) and the prior (83/138) used different denominators for the same metric.

[radical]

🔧 Post-run review — 2026-06-19 20:48 UTC

Regressions vs previous run: none. Diff vs the prior body shows only genuine state movement (window advanced ~2h; main 50%→53%, internal 87%→88%, PR-val 85%→90%, release/13.4 100%→50%, rerun split 16→19 human). All sections, columns, lane links, the activity bars, the Wilson-CI legend, and the sources footer were preserved. No frozen-history cells exist in this tier, so nothing to drift.

Readability / improvement ideas:

• (accept/reject) Pin the infra-share denominator to the build-level breakdown. This run reports 86% (67/78) of failed builds; the previous run reported 61% (62/102) of failed CI signals. The jump is partly real (more infra noise) but partly a denominator basis flip — "failed builds" (78, from ciinsights get_health_summary.buildHealth.failureBreakdown) vs "failed CI signals" (102). The common prompt's spec literally says "share of failed builds", so the 78-build basis is the spec-aligned one. Recommend always sourcing this single figure from get_health_summary failed / infrastructureFailures so the headline % stops swinging on methodology between runs.
• (accept/reject) The headline now carries two competing "loudest" signals (the analyzer pair by count, the conformance family by recency). It reads fine, but a future tweak could state the rule explicitly in the prompt — "lead with by-recency when the by-count leader has gone quiet (last-failed > Nh ago)" — so the choice isn't re-litigated each run.

Data-gathering issues this run:

• Rerun PR attribution is a lower bound. 26 of 45 rerun runs (run_attempt>1) in the 36h window had an empty pull_requests[] (typical for cross-fork PR runs), so the "PRs rescued / still red / distinct PRs" counts undercount. The rerun counts (43 bot / 19 human, from run_attempt) are reliable; the PR-level figures are not. Worth a one-line caveat in the rerun section, or computing distinct PRs via the head-branch/gh pr lookup instead of the embedded pull_requests[].
• ciinsights DB reported "unhealthy" (get_connection_status.lastError: duplicate-index error) and query_test_failures threw. get_failure_summary / get_health_summary / get_flagged_tests all worked, so the report is complete — but if the summary endpoints had also failed there is no fallback for the test-level breakdown. No action needed unless it recurs.
• gh run list caps at 1000 even with --limit 1500; for PR validation that still reaches back ~10 days (oldest Jun 9), covering the 7-day delta baseline. Fine today, but a busier week could push the 7d window past the 1000-run cap — sanity-check oldest-run-reaches-window-start still holds (it did).

Suggested prompt change (for a human to apply): aspire-ci-report-common.prompt.txt, the "Infra-share line" section — add one sentence pinning the denominator to the ciinsights get_health_summary build-level failed count, so the figure is reproducible run-to-run. Optionally, the "Reruns & the flaky tax" section — note that pull_requests[] is often empty on fork PR runs and PR-level rescue counts are therefore lower bounds.

[radical]

🔧 Post-run review — 2026-06-19 21:48 UTC

Regressions vs previous run (20:46): none. The diff is clean — every change tracks a genuine ~1h window shift: main 53%→56% (9/16), Internal 88%(14/16)→82%(14/17) as failed build 3004254 entered the window, PR 90%→89%, infra 86%→85%. No lost sections/columns/links, no format churn, no frozen-history drift (this tier keeps none).

Readability / improvement ideas:

• (accept/reject) The headline + first table still answer "what's on fire" without scrolling — keep as is.
• (accept) I tightened the headline to "weakest real-volume lane" because release/13.4 at 50% (1/2) is numerically lower but a 2-run sample; calling main "the weakest lane" flat was slightly misleading. Minor wording only — no structural change.

Data-gathering issues this run:

• ciinsights query_test_failures errored twice ("An error occurred invoking…"); fell back to get_failure_summary, which carries the fail counts the Pulse needs. Recurring-tool note, not a one-off.
• gh run list --workflow ci.yml --event pull_request --limit 400 only reached Jun 16 (~3.5d) — short of the 7d window needed for the PR Δ. Had to page gh api .../ci.yml/runs?...&created=%3E<7d> (591 runs) to get the true 7-day average (79.5%). The run-list path also slightly undercounted the 36h PR window (58/65 vs the paginated 59/66). The paginated API is authoritative for the high-volume PR lane.
• Rerun-split is sensitive to attribution method. Over a near-identical window the split swung run-to-run: 20:46 run reported 43 bot / 19 human, this run 46 bot / 14 human (per-run_attempt triggering_actor, 60 rerun-attempts total). The total (62 vs 60) is stable; the bot/human ratio is not, because last-actor-per-run and per-attempt-actor attribute the same reruns differently.
• The "13 distinct PRs" figure is a floor: 25 of 43 rerun runs had an empty pull_requests[] (fork PRs), so they can't be mapped to a PR number. This recurs every run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt → "Reruns & the flaky tax": pin the exact attribution unit so the bot/human split is reproducible. Recommend stating: count one rerun per run_attempt ≥ 2, attributed to that attempt's triggering_actor (not the run's latest actor). Without this the split wobbles run-to-run (43/19 vs 46/14) and the takeaway ratio (~2.3× vs ~3.3×) is not comparable across Pulses.
• Same file → "Fetch-enough-then-filter": add that for the PR-validation 7-day average, gh run list --limit falls short of 7d; use gh api --paginate .../ci.yml/runs?event=pull_request&created=%3E<7d-iso> instead. (The existing note covers undercounting the window but not that the run-list path can't reach 7d for this lane at all.)

[radical]

🔧 Post-run review — 2026-06-19 22:58 UTC

Regressions vs previous run: none. Structure is intact — same header/headline/lane-table/rerun/breakages/footer, all columns, links, the 🧱 infra line, and both legends preserved. Every changed value tracks a genuine state change since the prior run (the 36h window slid ~1h: Jun 18 09:40→21:40 became 10:41→22:41). No frozen-history drift (Pulse freezes nothing), no dropped sections, no format churn.

Readability / improvement ideas:

• Accept/reject: In the headline, the analyzer pair (last failed 01:33, ~21h ago) is named first as "loudest signal repo-wide" while the currently firing cluster comes second. A reader scanning for "what's on fire now" has to read past the loud-but-stale pair. Consider leading the headline with the currently-firing cluster and demoting the stale-but-high-count pair to a "(loudest by count, but quiet ~21h)" clause. Minor — the current order is defensible since count is the headline metric.
• Otherwise the top table + headline answer "what's on fire" without scrolling; no fact is written twice across sections.

Data-gathering issues this run:

• Rerun split is not reproducible run-to-run. Prior run reported 13 distinct PRs / 10 bot-rescued / 14 human reruns; this run, over a window shifted by only ~1h, computed 32 distinct PRs / 21 bot-rescued / 16 human reruns — a >2× swing that 1h of new data cannot explain. Root cause is methodology, not data: the shared prompt (lines 81–83) defines the shape of the split but not (a) the PR-identity key (I used head_branch for pull_request events; run-id vs branch vs PR-number give different counts), (b) whether push (rolling main/release) reruns count toward "PRs", or (c) the exact "rescued red→green" predicate (I used "final conclusion == success on a run that had ≥2 attempts"). Two runs picking different conventions will disagree wildly.
• gh api .../runs?created=>=YYYY-MM-DD over-fetches by up to a day. The created filter is date-granular, so it pulls runs from 00:00 of the start day — ~10h before a 36h window that starts mid-morning. I corrected by fetching each rerun attempt's run_started_at (via /runs/{id}/attempts/{n}) and filtering to the exact window; without that step the bot/human rerun counts inflate.
• Prior "latest failed internal build" (3004254) is not in definition 1602. az pipelines build list --definition-ids 1602 for main never returns 3004254 (max id seen: 3004196, partiallySucceeded). The current latest failed 1602/main build is 3003583 (Jun 19 00:08 UTC), which is what I linked. The prior link may have pointed at a build outside def 1602 or a transient — worth a "confirm the linked build is in def 1602" guard.
• ciinsights connection reported the DB as "unhealthy" (index ... 'IX_TestResults_FullyQualifiedName_CapturedAt' already exists) while still answering queries correctly. Transient/cosmetic this run, but flagging in case it later returns stale or empty results.

Suggested prompt change (for a human to apply): In aspire-ci-report-common.prompt.txt, the shared "Reruns & the flaky tax" section (lines ~79–86), pin the methodology so the split is reproducible:

• Define the PR unit explicitly (e.g. "distinct head_branch of pull_request-event runs; push reruns are excluded from the PR counts but their rerun attempts still count toward N").
• Define "rescued red→green" (an attempt ≥2 whose run's final conclusion is success after an earlier failure).
• Add the window-filtering note: filter rerun attempts by each attempt's run_started_at, not by the run's created date — the created=>= API filter is date-granular and over-counts the pre-window hours.

This refines an underspecified rule (no reversal in the prompt's git log); it does not contradict an existing convention.