Repository navigation
The High CPU and Long-Running Query cards name the maintenance and the job (#3495, #3497) - #3511
Merged
Merged
Conversation
First night of live dogfood alerting: a High CPU page delivered honestly — 100% total CPU, correctly split SQL/other — and the cause still took two more instrument reads to establish: an RDS-managed BACKUP DATABASE ... WITH COMPRESSION seventeen minutes in (program RdsAdminService, ASYNC_IO_COMPLETION), visible the whole time in the active-session snapshot the service had already collected on the same sweep. Everything the triage needed was knowable by the alert pass at fire time; the card could not say the one sentence that closes it. When the High CPU condition fires, the fire branch now reads the same active-session surface the operator read by hand — the long-running-query alert's own store read, threshold zero, every noise opt-out OFF and an empty exclusion list, deliberately: excludeBackups exists to keep maintenance off THAT alert, and this probe wants exactly the population it removes — and appends one line per system-maintenance session to detail_text: Active maintenance: BACKUP DATABASE (RdsAdminService), 17m 14s elapsed, ASYNC_IO_COMPLETION BACKUP DATABASE/LOG, RESTORE DATABASE/LOG and ALTER INDEX are the recognized shapes, matched on the trimmed statement HEAD rather than contains-anywhere, so the annotation under-claims (a maintenance statement buried mid-batch costs only its line) rather than naming maintenance on a card where none runs. Concurrent sessions get one line each, longest first, capped at three with the omission stated — the #3494 discipline — and the longer prose rides the splitter that fix installed, so it costs Slack section blocks, never delivery. ANNOTATION, NEVER SUPPRESSION: the tiers stay, the page still fires, the line states what IS (kind, program, elapsed, wait) and never a verdict. A backup pinning CPU at 04:30 with the store idle is routine; the same pin at 14:30 under checkout load is a capacity finding with a named cause — that judgment belongs to the reader. Bounds and degrades: the probe runs only inside the cooldown-gated fire branch (one bounded read per delivered page, never per sweep; muted fires included so the history row matches), and a failed read logs at Warning, is counted on distinguishable from a condition gone blind — and leaves the card byte-identical to today, which the no-maintenance arm pins exactly. Fingerprint-inert by construction: High CPU carries no incident fingerprint and its cooldown keys on (server, metric), so a re-fire with a different elapsed is the same incident it always was.
Second night of live dogfood alerting: the nightly index-maintenance window opened and the Long-Running Query alert did its job — a dozen honest cards, each showing ALTER INDEX ... REORGANIZE past the threshold — and each leaving the reader to reconstruct "that is the maintenance job, running as scheduled, merely long" from the statement shape, the hour, and prior knowledge, twelve times, every night, because index maintenance recurs. The session data already identified the program as a SQL Agent job step; the card never surfaced WHICH job. Now a card whose session's program_name carries the standard 'SQLAgent - TSQL JobStep (Job 0x<hex> : Step <n>)' form gains one field, directly under the raw Program it explains: Running under Agent job: nightly index maintenance, step 3 (Reorganize fragmented indexes) The identity mechanics live in one shared place (AgentJobStepQuery), because a wrong job-id conversion here does not error — it resolves nothing, silently, on every card. The hex in program_name is sysjobs.job_id CONVERTed to binary(16): SQL Server's GUID layout, Data1–Data3 little-endian, the same conversion the collector's CDC filter already does server-side via TRY_CONVERT — and .NET's Guid(byte[]) constructor reads exactly that layout, so the parse does no byte swapping and the round trip is PINNED in AgentJobStepQueryTests (job_id AB6D9F63-3B01-4E15-9F34-B0A0F0B355A2 <-> 0x639F6DAB013B154E9F34B0A0F0B355A2) rather than trusted. Resolution is FIRE-time, deliberately, over capture-time: no schema rung on either store, no degrade baked into collected rows, nothing on quiet sweeps — the lookup sits inside the cooldown-gated fire branch, scoped to the sessions the card will show (the builder's display cap, now named), deduped, and answered in ONE msdb round trip (a VALUES join over sysjobs + sysjobsteps; INNER on sysjobs so a deleted job stays honestly unresolved, LEFT on sysjobsteps so a missing step still names the JOB). Both hosts wire the resolver on the recently-failed-job check's exact live-msdb seam and permission arm: a login without SELECT on msdb.dbo.sysjobs and sysjobsteps (expected for read-only monitoring accounts — SQLAgentReaderRole is not the remedy) degrades to an empty map, logged Info with the grant named, and the card renders the unresolved form — 'Running under Agent job: (name unresolved) Job 0x<hex>, step <n>' — which still states the fact the parse alone establishes and carries the raw marker an operator can match against the Program field by eye. A null resolver renders the pre-#3497 card byte-identically; both arms pinned. ANNOTATION, NEVER SUPPRESSION — the #3495 contract, sibling card: the thresholds stay, every card still fires, the annotation states what IS and never a verdict. And it is fingerprint-INERT by construction: the #1140 dedup key hashes (server, type, query_hash) and the annotation is a rendered field, so a card that re-fires with a different elapsed — or with the job name freshly resolved, or freshly unresolvable — folds into the same incident, which per-fingerprint delivery cooldowns and PagerDuty correlation key on; pinned through the engine, not merely stated. CHANGELOG entries for both siblings are buffered in CHANGELOG.pending.md per the parallel-lanes process; CHANGELOG.md itself is untouched.
… read is counted The #3495 probe records its swallowed failure under its own name and clock - and the read-failure censuses hold the population by pinned count, per-file scope, and narrated addition. AlertEngine's fourteenth site joins the ledger the way the evaluator's eighth did on #3466: counted, because a fault folded into 'no maintenance ran' is the quiet-is-not-clean misreading wearing a card.
erikdarlingdata
enabled auto-merge (squash)
September 17, 2026 14:18
… on the monitored server, exempt by the failed-jobs precedent The census's deeper arms caught what the count fix alone could not: the LRQ block's two awaits shared one stopwatch (a resolver fault would have recorded the store read's elapsed on top of its own), and the resolver's catch was neither counted nor exempt. Restart between the awaits; the catch joins the exemption table keyed on its own log line - it reads the monitored server's msdb through the host resolver, not the store, the Recently-failed-job precedent one seam over, and the card's unresolved form keeps the gap visible where a count would double-book a read that is not the store's.
…lt that must not double-bill is the NEXT one The census's pairwise walk flagged resolver-to-FireAsync; the earlier fix guarded store-read-to-resolver, the pair above it. Both boundaries now restart, and the faithful replication of the walk shows every consecutive pair in the long-running queries block carrying one.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3495; #3497 is closed manually at merge (one PR, two sibling issues — the watcher maps one). Changelog entries ride the batched entries PR.
Two cards learn to say the sentence that closes the triage
Both issues came from the first nights of live dogfood alerting, both carry the same owner-written contract, and it is honored verbatim in code comments and both entries: annotation, never suppression. The tiers stay where they are, every card still fires — a backup pinning CPU at 04:30 on an idle store is routine, the same pin at 14:30 under load is a capacity finding with a named cause, and that judgment belongs to the reader, who can only make it in one glance if the card states the fact.
#3495 — the High CPU card names active maintenance. When the condition fires, the engine probes the same active-session surface the operator read by hand on the evidence night — the exact adapter seam the LRQ alert already uses, with the noise opt-outs deliberately OFF because the probe wants exactly what those filters exist to remove — and appends one line per maintenance session (
Active maintenance: BACKUP DATABASE (RdsAdminService), 17m elapsed, ASYNC_IO_COMPLETION), capped with a stated omission. No maintenance running renders the card byte-identical to today, pinned. The probe's swallowed failure is counted on the read-failure surface under its own name and clock — the fourteenth AlertEngine site, narrated in the census — because "no maintenance ran" and "the probe went blind" must never look alike.#3497 — the Long-Running Query card names the Agent job. A session whose program name carries the SQLAgent job-step form gets its job id parsed — the hex is msdb's
binary(16)GUID layout, Data1–3 little-endian, prior art cited from the CDC filter and the byte order pinned with a live-verified round-trip — and resolved through one msdb round trip per delivered card, scoped to the shown sessions, deduped, via a host delegate wired in both SKUs. The card gainsRunning under Agent job: <name>, step <n> (<step name>)directly under the Program field. Permission denied routes through the shared msdb-denial precedent and degrades to the unresolved form carrying the raw marker byte-for-byte; a deleted job stays honestly unresolved; a missing step row still names the job; non-Agent sessions render byte-identically, pinned.The property that took the longest to verify: fingerprint inertness
Neither annotation may enter any dedup identity — a card re-firing with a different elapsed must fold onto its incident, not mint a new one. Traced and pinned at both the engine and builder level: the LRQ dedup key is built from the query list the annotation never touches; High CPU has no incident fingerprint at all; every downstream consumer (webhook cooldown, delivery filter, the PagerDuty key) keys on DedupKey and none on detail text. The #3494 prose splitter carries the longer detail text by design, and every existing exact-string detail pin in both suites holds because the empty-annotation arms contribute zero bytes.
Eighteen new tests across three suites: the parse (13 malformed arms, byte-order round-trip), both cards' annotated and byte-identical arms, every degrade arm, the cap's stated omission, resolver scoping and dedup, and fingerprint inertness twice. Tests cannot run on macOS; the byte-order contract was verified with a live dotnet check and the census walks replicated statically. CI is the first suite execution, review-dark per the known window.