Skip to content

Three PostgreSQL pages observe truncation off limit + 1 through one shared helper, the census sweeps the inference class instead of a roster, and four payload rules become facts and inventories, so three tools stop guessing truncation from a full page and four of the eleven payload rules stop being sentences (partial #3653) - #3699

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3653-payload-census
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Part of #3653 (the "11-rule payload contract as census" line, from #3541's checklist, and the three >= limit residue sites PR #3679 named). Does not close it.

The lie

Three PostgreSQL tools still guessed truncation from a full page, and four of the eleven payload rules were sentences, not tests.

get_pg_deadlocks, get_pg_plan_capture_readiness and get_pg_index_bloat each published truncated = rows.Count >= limit — the inference #3594 named and every other honest page in the repository had replaced. A page that is exactly limit long cannot tell "that was all of them" from "there was at least one more", and the flag said more either way. Two of the three made it worse by acting on the flag: the readiness tool WITHHOLDS unsatisfied_facets when truncated, and the collector emits exactly six facets, so a caller asking for limit = 6 — the whole set — was told the page was cut and had the one figure the tool exists to produce withheld. The index tool withholds answered_count / estimated_count / exactly_measured_count the same way, for a candidate census that happened to be exactly limit long. Beside the readiness site, get_pg_plans had the quieter cousin: a Take(limit) over a read already cut at limit, and no word on the wire about the cut at all.

The census that pins this rule — McpPageContractTests.EveryPagedTool_ObservesTruncation_AndNeverInfersIt_OnEitherSku — was green the whole time. Why it missed them is the first thing this PR answers: it walks PagedTools, the ten tools #3541 A3 named, and its #3653 sibling walks one file (DarlingMcpPgServerStateTools). The three tools were in neither roster, so their bodies were never read. The regex was never the gap: TruncationInferred matches both spellings they used, the local (var truncated = rows.Count >= limit;) and the object-initializer member (truncated = rows.Count >= limit,), and the discriminator test now witnesses both against the shipped lines. A roster is the right instrument for the POSITIVE contract (which tools must observe, bind and publish their bounds — it names what a paged tool is) and the wrong instrument for the anti-pattern, which is a defect wherever it appears.

The fix

One helper, four first consumers. McpHelpers.BoundPage<T>(fetched, limit) is the one place (rows fetched at limit + 1, limit) becomes (page, truncated): truncated = fetched.Count > limit, the page is the first limit rows in the reader's order (the same list when nothing was cut), and every count a caller takes off the page is a count of the page. The four tools fetch limit + 1 and bind through it. On the index tool the page is bound BEFORE the empty branch, the coverage verdict and the reach classifier, so rows.Count keeps meaning what it meant at every downstream site; PgCappedRead.Classify still receives the page count and the cap, and its documented-pessimistic ReturnedRows < Limit → Complete test is not this change's to soften. The two plan tools' wire-shape builders (BuildPlansJson, BuildReadinessJson) receive the limit + 1 fetch and bind inside, so their existing wire-shape tests keep exercising the real projection. get_pg_plans now publishes truncated and a note that says the window holds more shapes, ranked below these by total duration, and that query_id pins one statement wherever it ranks. Every limit parameter description says what it bounds and names the flag, in #3679's wording. The readers were already parameterised (LIMIT $4 / $5 / $4 / $4), so no Storage file changes.

The census now catches the class. NoTool_InfersTruncationFromItsCap_OnEitherSku sweeps every tool body on both SKUs (244 methods; floor 150, the same population control the A13 Math.Abs sweep uses) and the whole file below the last tool, where two of the three lived. Run against the three files at their dev state it fails naming exactly get_pg_deadlocks, get_pg_index_bloat, get_pg_plan_capture_readiness — the positive control. It does NOT flag limit_reached = x.Count >= limit (five PostgreSQL tools): that key says the page is as long as the cap, which is true by construction, and its prose says "more MAY exist" — a weaker dialect rather than an inference, and whether it collapses into an observed truncated is the vocabulary lane's call. It is held in the truncation-key inventory instead, and the discriminator test witnesses the non-match. A Darling-only roster (PgPagedToolsThroughTheHelper) pins the four first consumers: limit + 1 in the body, McpHelpers.BoundPage in the body or the named builder, truncated published, the reader's cap parameterised, the description naming the bound. The census accepts EITHER the helper OR the proven inline spelling (var truncated = x.Count > limit;) and says so — forty-odd inline sites are not swept onto the helper tonight, other lanes hold those files.

The pin that encoded the lie. DarlingMcpPgPlanToolsTests.ACappedResult_WithholdsTheUnsatisfiedSummary_RatherThanDescribingThePage handed the builder two facets at limit = 2 and asserted truncated — the >= limit inference as its own proof. It now hands three (a truncated page of two, third row the signal) and a new fact asserts the other boundary: two at limit = 2 is complete, and unsatisfied_facets is an empty ARRAY, not null — absent-because-withheld and absent-because-none are different answers. This is the one test file outside the brief's list I touched; the pin was falsified by the fix and a load-bearing pin is a granted boundary.

The four rules, as census

McpPayloadContractCensusTests (new; one region per rule, each fact stating its population, its floor and its roster; rosters are EQUALITIES so a hit that vanishes without leaving the roster is as red as a new one):

  • Errors one shape — pinned as "every catch in every tool method returns through a SHARED shape": McpHelpers.FormatError (directly or via Task.FromResult on the synchronous tools), McpHelpers.Status("error", …), the write tools' Outcome("invalid", …), or a gate computed above the catch. 254 catch returns swept; one ad-hoc site rostered shrink-only (list_servers returns an interpolated sentence — another lane's file). The rule itself is not met: the two shared shapes are a bare STRING and a JSON envelope, executed side by side in TheTwoSharedErrorShapes_AreASentenceAndAnEnvelope — 200-odd tools answer an exception with prose an agent keyed on status reads as successful text, 30-odd with {"status":"error"}. The split is pinned as a file-grain inventory (21 files return the envelope: every PostgreSQL tool file plus the collector-cost tool; every Lite file and every SQL Server-family Darling file returns the sentence) for the vocabulary lane to collapse.
  • Refuse what you cannot honor — pinned: every tool declaring hours_back / limit / top / days_back / as_of (156 methods) reaches a McpHelpers validator in its own span or a same-file helper it calls (the nine health-parser tools share CollectAsync), or is one of five rostered inline days_back refusals (no ValidateDaysBack exists; they refuse, they do not clamp — three with a bare sentence, the errors split one layer down). No tool clamps a bounded parameter against a ceiling or literal — zero hits, so the fact carries the population floor and a witness that the matcher fires on Math.Clamp(limit, 1, MaxTop) / Math.Min(hours_back, 168) / top = Math.Max(1, top) and passes the page-count idiom Math.Min(rows.Count, limit). The Math.Abs arm stays where The daily summary stops painting purged months green, and every MCP filter is part of the query: parallel_only/min_dop/blocking_only cut before the page, a negative window is refused, an unknown source names the accepted set (#3541 A9/A13) #3641 put it. The timestamp arm: no tool source parses a timestamp under the host culture; the two general (non-Exact) DateTime.TryParse calls that remain (create_mute_rule's expires_at, update_mute_rule's BuildMuteRuleUpdate) carry InvariantCulture + AdjustToUniversal | AssumeUniversal and are rostered shrink-only — they accept every shape the framework accepts rather than the AsOfFormats allowlist ResolveAsOf refuses against, and a control shows the allowlist refusing 08/18/2026 14:30 while the general parser under the same styles accepts it.
  • Name = truth — NOT mechanically pinnable in general; the two greps MCP payload-contract campaign: the agent-facing API tells the truth (~30 defects + an 11-rule contract) #3541 names are pinned as inventories and the rest is stated as review-checklist. Every total_* assigned from a .Count/.Length in a tool body (10 sites, both SKUs) is rostered with the WHOLE set it counts (get_mute_rules' every rule, the default-trace and health-parser tools' every event in an uncapped window, get_analysis_facts' every collected fact) — a new one must prove its set is whole or rename *_returned. Every reader constant whose OUTER statement anchors on = (SELECT MAX(collection_time | capture_time) …) projects the anchor column so a stamp is possible (15 anchored constants on both SKUs; a depth-zero walker finds the owning SELECT list; anchors inside a CTE or subquery pick a SET and are not snapshots) — two rostered exceptions, both known: IndexUsageSql (the A10 object-stats residual McpLatestSnapshotStampTests.UnstampedLatestReadsPendingA10 already holds) and IndexUsageMatchCountSql (a COUNT(*) with no row to stamp).
  • One vocabulary — NOT collapsed here; the INVENTORY the vocabulary lane will shrink is pinned, at KEY grain so another tool joining an existing dialect changes nothing while a new spelling or a collapse moves the roster. Truncation keys: 14 spellings across both SKUs (truncated, shown, limit_reached, scan_truncated, truncation_note, is_partial, partial_count, history_capped, capture_was_truncated, chain_may_be_truncated, chain_truncation_note, query_text_may_be_truncated, answered_rows_withheld, created_rows_withheld) — the doc says which are dialects of one fact and which are different facts. Severity words: 8 spellings in three casings (Healthy/Warning/Critical/Unknown, warning/unknown, CRITICAL/HEALTHY) with the files that spell each; the NoData/No Data pair is pinned by its named instance — get_daily_summary on both SKUs publishes health_band = HealthBand.ToString() beside overall_health = DailyHealthBandCalculator.Label(…), the same band under two spellings in one object, executed against the shared calculator. Statement-terminal literal LIMIT n (n > 1): seven, all in Lite's service layer, none behind a tool that takes limit (six viewer-only reads; GetPlanCacheSnapshotAsync's LIMIT 30 is a ceiling over a dozen cache types behind a tool with no cap) — zero on Darling.
  • Writes report what happened — already pinned per write tool by MCP write tools report what happened: every server you add lands in a counter, a partial name cannot delete the wrong server, an omitted field is never a clear, and a mute that matched nothing says so (#3541 A14) #3615 (DarlingMcpServerAdminToolsSurfaceTests.EveryAddStatus_HasExactlyOneSummaryCounter_AndNoCounterIsUnnamed, Aggregate_CountsEveryResultOnce_AndTheCountersSumToRequested); a cross-write census is deferred until the envelope lane's mute-write rewrite lands tonight, and the file says so.

Files whose names carry ForcePlan are outside every sweep by standing decision, not by allowance.

What it does NOT do

  • It does not rename deadlock_count / facet_count / plan_shapes / index_count to *_returned. They are page counts under a neutral noun, not total_*; the vocabulary lane owns renames.
  • It does not sweep the forty-odd inline var truncated = x.Count > limit; sites onto BoundPage. Both spellings are the observation; the census accepts both.
  • It does not touch limit_reached (five tools) or history_capped (runs.Count >= RunsReadPerTable, the same shape against a collector constant). Both are inventoried, not flagged.
  • It does not collapse the two error shapes, the truncation dialects or the severity casings. It produces the rosters the collapse will shrink.
  • It does not touch PgCappedRead.Classify's pessimistic Complete test, CHANGELOG.md, any other Mcp/*Tools.cs, server-tabs.js, DarlingWebEndpoints.cs, or any PgTarget* file.

Tests

All executed on this Mac through the built Darling.Tests.dll (xunit v3 in-process runner, -p:EnableWindowsTargeting=true, WindowsDesktop framework entry stripped from the runtimeconfig), then first executed in CI:

  • McpPayloadContractCensusTests — 23 facts, 23 green. Every roster above was first written from a survey, then run, and the census corrected the survey in three places (a fifth inline days_back refusal in get_collector_stall_probes; five truncation keys emitted from builders below the last tool; three UPPER-case severity spellings in the Darling service's own readers).
  • McpPageContractTests — 40 green (+NoTool_InfersTruncationFromItsCap_OnEitherSku, EveryPgPagedToolThroughTheHelper_…, EveryPgPagedReadThroughTheHelper_BindsItsCapAsAParameter_NeverALiteral, a seven-row BoundPage theory, the widened discriminator witness). Positive control: with the three tool files at dev state on disk the sweep fails naming exactly the three sites.
  • McpPageContractLivePostgresTests.PgPagedTools_FlipTruncatedExactlyAtTheBoundary_AgainstDevPostgres — against timescale/timescaledb:2.28.1-pg18 (PostgreSQL 18.4) through the real tool methods: five distinct deadlock reports (one seen twice) at limit = 4 truncated / 5 complete; six readiness facets at 5 truncated with unsatisfied_facets null / 6 complete with the three unsatisfied named in causal order; four plan shapes at 3 truncated with the note / 4 complete; three answered indexes at 2 truncated with answered_count withheld / 3 complete with answered_count = 3, coverage's candidate_index_count 3 on both pages. Positive control: with BoundPage flipped to >= the live test fails on the first limit = N arm, the two new builder pins fail, and two of the seven theory rows fail.
  • DarlingMcpPgPlanToolsTests — 14 green (the reshaped pin plus two boundary facts). DarlingCappedReadLivePostgresTests (6), DarlingPgIndexBloatNanLivePostgresTests (2), DarlingPgIndexBloatCoverageLivePostgresTests, DarlingPgIndexBloatRotationReadLivePostgresTests, DarlingMcpPgPlanQueryIdLiveTests, DarlingPgReadSqlParsesLiveTests — the pre-existing suites that call the changed tools against live Postgres — all green through the limit + 1 fetch. PgCappedReadSurfaceTests (12), PgIndexBloatCoverageTests (16), PgLoggingCollectorOffTests, RepoFileAdoptionTests, CommentFilterAdoptionTests, McpLatestSnapshotStampTests (30), AsOfWindowAnchorTests (29) — green.

PerformanceMonitor.Common, Darling.Storage, Darling.Service, Darling.Viewer, Darling.Tests, PerformanceMonitorLite, Lite.Tests, deprecated/Dashboard.Tests build -c Release -p:EnableWindowsTargeting=true with 0 warnings from this change (Lite.Tests carries one pre-existing xUnit2000 on dev at LogTailOverlapThresholdPinTests.cs:95).

Which component(s) does this affect?

  • Lite
  • Darling (four PostgreSQL MCP tools; PerformanceMonitor.Common/Mcp/McpHelpers.cs is shared, compiled into both SKUs, and gains one helper nothing on Lite calls yet)
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (-c Release -p:EnableWindowsTargeting=true, every touched project and every consumer of PerformanceMonitor.Common)
  • I have tested my changes against a live store (PostgreSQL 18.4 / TimescaleDB 2.28.1 via the gated live tests above; the tools are PostgreSQL-target reads and the change is in the store read, not the target read)
  • I have not introduced any hardcoded credentials or server names

…hared helper, the census sweeps the inference class instead of a roster, and four payload rules become facts and inventories (partial #3653)

get_pg_deadlocks, get_pg_plan_capture_readiness and get_pg_index_bloat published
truncated = rows.Count >= limit, the #3594 inference; the readiness and index tools
withheld their summaries on the strength of it for pages that were the whole set,
and get_pg_plans beside them Take(limit)'d a read already cut at limit and said
nothing. McpHelpers.BoundPage turns (fetched at limit + 1, limit) into (page,
truncated) in one place; the four tools are its first consumers and every count
downstream is still a count of the page.

McpPageContractTests missed the three because EveryPagedTool_ObservesTruncation
walks the ten tools #3541 A3 named and its #3653 sibling walks one file - a roster
gap, not a regex gap; the matcher catches both spellings and now witnesses the
object-initializer one. NoTool_InfersTruncationFromItsCap_OnEitherSku sweeps every
tool body on both SKUs with a population floor and fails on the three at dev.

McpPayloadContractCensusTests pins the four unpinned rules as far as a census can
honestly go: every catch returns through a shared shape (one ad-hoc site rostered)
and the two shared shapes - a sentence and a JSON envelope - are an inventory for
the vocabulary lane; every bounded parameter reaches a shared validator or one of
five inline days_back refusals, and nothing clamps a parameter; every total_*
assigned from a Count names a whole set (rostered), every latest-anchored reader
projects its anchor (two known exceptions); truncation-key dialects, severity-word
spellings and statement-terminal literal LIMITs are exact inventories. The
DarlingMcpPgPlanToolsTests pin that handed the builder two rows at limit 2 and
expected truncated encoded the inference as its own proof and becomes the boundary
pair.

Executed here through the built Darling.Tests.dll and against PostgreSQL 18.4 /
TimescaleDB 2.28.1; positive controls: the three files at dev state fail the sweep
by name, and BoundPage flipped to >= fails the live boundary, the builder pins and
the helper theory.
@erikdarlingdata
erikdarlingdata merged commit 3676970 into dev Sep 19, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3653-payload-census branch September 19, 2026 06:53
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…le, Opus full tier (#3710) (#3718)

* REVIEW-TRAPS.md: the repository's recurring defect shapes, one line each with the PR that bit (#3710)

Seven layers (SQL, collectors and deltas, analysis, alerting and notifications,
storage and MCP payloads, tests and censuses, workflows and CI, repository
mechanics), fifty-odd lines, every one citing the pull request or issue that
verified it: the >= limit truncation inference (#3594/#3679/#3699), ORDER BY
3 + 4 folding to a constant (#3613), MAX(max_dop) across plans (#3648), integer
division in cntr_value_per_second (#3653 Q9), per-collection alert firing
against a shorter cooldown (#3628/#3636), restart zeros in deltas and baselines
(#3595/#3698/#3705/#3708), ScoreAll skipping amplifiers on a zero base and the
non-virtual GetActiveEdges (#3688), the PgTargetFactKeys censuses (#3688),
pg_total_relation_size vs the heap and the two TimescaleDB views that hide
materializations and history (#3610/#3585), the -infinity crash backoff (#3629),
the Slack block budget and the surrogate-pair cut (#3612/#3618/#3625/#3644),
LCK_M_SCH_M folding into LCK so no SCH_M fact exists (#3709), one wait profile
per run (#3709), and the CRLF round-trip. Each citation was read before it was
written down; a wrong citation is worse than none.

The review workflow reads this file before the PR body and must say, per trap
the diff is near, whether it applies here and why.

* Claude review: blind read before the body, callers rule, structured verdict with a ledger, and an Opus/Sonnet tier by changed path (#3710)

The prompt runs in two phases. Phase 1 forbids gh pr view / git log / git show
until the reviewer has read the diff, every changed region in context, the
callers of every changed public symbol (Grep, both SKUs and both test projects)
and .github/REVIEW-TRAPS.md, and written its own account. Phase 2 reads the
body and grades every claim VERIFIED / UNVERIFIED / REFUTED, then reports the
DIVERGENCE between the two accounts. The verdict is a fixed block -- Claims,
Riskiest lines with the pinning test or UNPINNED, Traps near this diff,
Divergence, Reviewer checklist -- ending in one machine-readable ledger line,
posted through gh pr review --body-file - with a quoted heredoc (no Write tool;
the allowlist is unchanged and still read-only).

The #3650 verdict check now fetches the newest verdict in the window by id and
fails the job when the body lacks the "## Verdict:" header or the ledger, or
when the ledger's verdict disagrees with the review's API state (a
"CHANGES REQUESTED" body submitted with --comment is the shape the guard would
wave through). The guard's own arm is untouched and still keys on the state.

A dorny/paths-filter@v4 step (the repo's pin style) routes changes under the
shared libraries, Darling Storage/Analysis/Service, Lite Services/Mcp/Database,
install/ and every .sql file to the full tier -- claude-opus-5[1m], 60 turns --
and everything else to claude-sonnet-5 at 30 turns; one decision step emits
tier/model/max_turns, one review step consumes them (two steps would double the
"review posted" accounting the guard does), and the prompt carries the tier and
model into the ledger. A filter that dies degrades to FULL, not light.

Step name "Claude review" and the ALWAYS POST sentinel are unchanged: both are
read by claude-review-guard.yml.

* Review tier: Alerting and Notifications read at the full tier (#3710)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant