get_store_metrics: the job_history block said 'recording' and never said whose eyes the rows are visible to — it now evaluates the view's ownership filter for its own reader and proves rows exist where it can (#3574)

[erikdarlingdata]

The lie, precisely

get_store_metrics' job_history block reports recording: true. That is the #3175 guard, and it is true: it reads the effective value of timescaledb.enable_job_execution_logging and the GUC is on. But it answers "is the instrument switched on", and the reader who follows the tool's own redirect to timescaledb_information.job_history is asking "will I see its rows". Between those two questions sits a filter nothing on this surface mentioned. The view's definition on TimescaleDB 2.28.1 ends with:

WHERE (pg_catalog.pg_has_role(current_user,
           (SELECT pg_catalog.pg_get_userbyid(datdba)
              FROM pg_catalog.pg_database
             WHERE datname = current_database()),
           'MEMBER') IS TRUE
    OR pg_catalog.pg_has_role(current_user, owner, 'MEMBER') IS TRUE);

A row is visible only to a member of the database owner's role or of the job's owner role. Jobs are owned by whoever created them (bgw_job.owner regrole NOT NULL DEFAULT current_role), which for every policy this product adds is the service's owner role. The base table is not a back door — pre_install/tables.sql ends with REVOKE ALL ON _timescaledb_internal.bgw_job_stat_history FROM PUBLIC. And the trap is that the two views a reader checks first — timescaledb_information.jobs and job_stats — have no WHERE clause at all. A role that can see all 110 jobs and every one of their total_runs reasonably assumes it can see their history, reads zero rows, and concludes recording is broken. Measured on a production store this morning: the GUC effective on, all 110 jobs owned by the service's owner role, two independent reads as the least-privilege admin role counted 0 rows ever, and the owner role's read of the same view returned every hourly run. A stuck-then-recovered compression job's run rows were declared unknowable in a real postmortem while they sat in the table.

Recording was never broken. The GUC never lied. The block never said whose eyes the rows are visible to, and never proved rows exist.

What the block says now, and why each field

The reader gains a second statement, DarlingStoreMetricsReader.JobHistoryEvidenceSql, read after the GUC probe on its own checkout and failure-isolated on its own — the two fail independently, so a count that times out cannot make the GUC read unknown. It is not attempted where the GUC probe said the view does not exist (plain PostgreSQL → NotApplicable, not a fault-shaped Unreadable). In one round trip it returns:

• reader_role — current_user. Every count that follows is a count through that role's eyes, because the view decides per role what it shows.
• reader_is_database_owner_member, job_count, history_visible_job_count, visibility — the view's own two pg_has_role tests, evaluated live for this reader (database-owner membership, and per-job owner membership from the unfiltered jobs view), and the verdict derived exactly as the view combines them: All (a census), Partial (a census of the jobs this reader owns, stated as a fraction), None (the view shows this connection nothing by construction). Unknown when nothing was observed or there are no jobs to be an owner of.
• rows_observed, observed_window_hours, newest_row_at — what this reader actually saw: history rows with a start inside a fixed 24-hour window, and the newest start it was shown over all time (null when it was shown none — a count and a nullable are two different facts, and the whole issue is one being read as the other).
• jobs_run_in_window, newest_run_started_at — the population half, from the unfiltered job_stats: how many jobs started a run inside the same window. A zero is readable only when the instrument would have caught the event AND the event had a chance to occur; this is the second condition, measured rather than assumed.
• contradiction — the new finding class as one bool, true only when all four conditions hold: the GUC says recording, the reader's visibility is All, jobs started runs in the window, and the view showed no rows.
• evidence — Observed / Unreadable / NotApplicable, so a count this read did not obtain never reads as zero.

Why 24 hours and not days_back. The question is current — is the instrument writing now — and a 30-to-400-day forecasting window would let ten days of rows written after a heal hide a recording that stopped yesterday. Every job this product schedules runs at least daily (CAGG refreshes and the compression tick hourly, retention daily), so any 24-hour window on a live store contains starts, which is what lets job_stats prove the population half. And the view's own Job History Log Retention Policy drops rows after one month by default, so a window past that would count a table the retention job had already trimmed and call the trimming "no rows".

Why the bind is timestamptz with Kind=Utc, explicitly. This is the inverse of the store's naive-UTC discipline, on purpose: bgw_job_stat_history.execution_start and bgw_job_stat.last_start are declared TIMESTAMPTZ in TimescaleDB's own catalog, so here the naive bind would be the bug — the parameter would be converted at the session zone and the window would skew by the host's offset. The type is stated rather than inferred so a caller's wrong Kind cannot quietly flip it.

TimescaleDB inserts the history row at job start when the GUC is on (bgw_job_stat_history_mark_start, 2.28.1) and updates it at finish, so a run that started inside the window while logging was on has a row inside the window — no waiting on a finish. Failures are written regardless of the GUC (the same file's bgw_job_stat_history_update), which the GUC-off arm of the note now states so a non-zero count with the GUC off is read as failures, not as logging being secretly on.

The note names the rule, and the contradiction arm is loud

Every arm on which the view exists — including the GUC-off arms, because a reader who heals the GUC and then checks as the wrong role walks into the same trap one step later — now carries: history rows are visible only to members of the job's owner role or of the database owner (with the predicate quoted), while jobs and job_stats show every role every job, so a zero from a role outside those memberships contradicts nothing. Then, per what the evidence established about this reader:

• None — "This block read as 'mcp', which is a member of NEITHER, so the view shows this connection NOTHING by construction: rows_observed = 0 here is the filter, not the table … The service's owner role — the role that created the jobs — sees the rows … Meanwhile job_stats, which is not filtered, says 9 job(s) started a run in the last 24 hours."
• All, rows seen — "… so the count is a census: 12 row(s) with a start in the last 24 hours … 'recording' is therefore a measurement here, not a GUC echo."
• All, zero rows, jobs ran, GUC on — CONTRADICTION: "the GUC says recording, this role can see every job's history, jobs started runs inside the window, and the view showed NONE of them — do not read this zero as quiet. The one benign cause is logging switched on AFTER the last of those starts (the managed v11 heal lands on a service-owned server start and writes nothing for earlier runs), which the next hourly run settles: re-read after it. A zero that persists while jobs_run_in_window climbs means the instrument is not writing what the GUC says it is, and that is a finding, not a quiet hour." The benign cause is named rather than ruled out; the existing "covers runs from the moment logging was turned on" sentence stays in front of it.
• All, zero rows, nothing ran — "… this zero proves nothing either way; it is not a clean result, it is an absence of information."
• Partial — the fraction, a census of those jobs only, no verdict.
• Unreadable evidence — "'recording' above is the GUC's word alone, not a measurement."
• NotApplicable — adds nothing; the job_history is silently empty on clusters predating ~2026-08-17, because its GUC lives in the one postgresql.conf block EnsureConfAppended cannot heal #3175 NotRegistered note is byte-for-byte what it was.

The #3175 arms (On, Off by default, Off by explicit override, NotRegistered, Unreadable) are intact and still pinned distinct.

The finding the lane made on the way in, and why it shaped the design

The issue's fix shape assumed the block could count through "the service's own (owner-member) connection". In managed mode it cannot. DarlingMcpHostService builds the MCP pool from the dedicated least-privilege mcp role — DarlingManagedRoles grants it SELECT and a few narrow writes, never membership in the owner — so on every managed store this very connection is exactly the reader the view filters out. A bare count(*) would have reported rows_observed = 0 on every managed store and manufactured the false contradiction the issue is about. That is why the read evaluates the predicate for itself rather than counting and assuming, and why visibility is a first-class field: on a managed store the block now says None, says the zero is the filter and not the table, and names the role that can see — which is the sentence that would have ended this morning's postmortem in a minute. On a bring-your-own store whose connection string is the owner role, visibility is All and the flag is fully self-proving.

Making the managed block self-proving too needs owner-pool evidence — the hourly StoreSelfMetrics sweep (owner pool) persisting the owner's count into the series as a new object_kind, or a worker-published in-memory reading. Both change an on-disk or cross-component contract and neither is taken here; the block is honest about what it can and cannot see, and the choice is left open in the report on the issue.

What this does NOT do

• It does not change recording semantics. The GUC probe, the v11 conf heal and the four job_history is silently empty on clusters predating ~2026-08-17, because its GUC lives in the one postgresql.conf block EnsureConfAppended cannot heal #3175 states are untouched.
• It does not grant anything. No role gains membership, no SECURITY DEFINER wrapper is created, the mcp role's surface is exactly what it was. The fix is that the block reports the filter with a measurement, not that it bypasses it.
• It does not make recording: true a measurement on a managed store — it says, in the field and in the note, that it is a GUC echo there and why.
• It does not touch CHANGELOG.md (entry proposed separately).

Elsewhere in the diff

• TimescaleSupport.cs (HeaviestHourlyRefreshObservedCeilingSeconds doc): the sentence "No SHIPPED read touches job_history" was true until this PR; it now says no shipped read takes a duration from it, and names this count as the one shipped read that touches the view and evaluates its filter.
• Darling/README.md (Background workers → after the job_stats check): a paragraph telling an operator, before they read job_history by hand, who it shows itself to and which role to read as.
• DarlingMcpInstructions.cs (store self-metrics paragraph): the per-run route is named as ownership-filtered and the block's new fields are pointed at.
• Tool description: the ownership filter, the trap, the new fields and the managed-mode mcp fact, asserted together with the shipped SQL.

Tests (first execution is CI — this Mac cannot run net10.0-windows; dotnet build of Darling.Tests and the Viewer is clean at 0 warnings)

DarlingMcpStoreMetricsToolsTests: the evidence SQL pins (both pg_has_role tests in the view's terms, all three views, one bind applied to both halves, the -infinity NULLIF, the 24-hour constant); the explicit timestamptz/Kind=Utc bind as a source pin; Visibility derived as the view combines its tests (db-owner short-circuit, None, Partial, All-by-owners, Unknown on no jobs / not observed); the contradiction's four conditions with each flipped alone; the note distinct per evidence shape with the rule and the reader on every arm, CONTRADICTION only where all four hold, NotApplicable adding no text; failure isolation without a store (a Port=1 data source → evidence Unreadable while the GUC reading it was handed is untouched, and a pre-cancelled token proving NotRegistered returns before touching the connection); the description asserted with the shipped predicate. DarlingStoreMetricsJobLoggingLivePostgresTests (gated on DARLING_TEST_PG): the evidence probe against a live TimescaleDB, inserting nothing and pinning no count — the null-versus-zero contract, cross-checked against direct reads of the same views, and Visibility == All with the visible count equal to the job count whenever the connection is a database-owner member.

[claude]

Reviewed the diff (DarlingStoreMetricsReader.cs, DarlingMcpStoreMetricsTools.cs, DarlingMcpInstructions.cs, TimescaleSupport.cs doc comment, Darling/README.md, plus the test suite).

Correctness: Traced the JobHistoryEvidenceSql query and the Visibility/ContradictsRecording derivation logic carefully against the view's real predicate (pg_has_role(..., datdba...) OR pg_has_role(..., j.owner...)). The derivation matches: DB-owner membership short-circuits to All, otherwise the per-job count is compared against the total job count for None/Partial/All, and Unknown correctly covers both "not observed" and the vacuous "no jobs" case. ContradictsRecording requiring Visibility == All (not Partial) is correct — a partial-visibility reader can't rule out that the jobs it can't see are the ones that ran. The two reads (GUC probe, evidence probe) are genuinely failure-isolated on separate checkouts, and NotApplicable short-circuits before hitting a view that doesn't exist on plain Postgres. The live-Postgres test (JobHistoryEvidenceProbe_...) cross-checks the derived fields against direct queries of the same views rather than just pinning shapes, which is a good source-of-truth check.

Style/convention note (non-blocking): GetJobHistoryEvidenceAsync binds $1 as timestamptz with Kind=Utc (DarlingStoreMetricsReader.cs:546-552), which is the inverse of CONTRIBUTING.md's Darling/Postgres rule ("Timestamps are naive UTC... strip DateTimeKind before binding"). The deviation is deliberate and extensively justified in the doc comment (these are TimescaleDB's own TIMESTAMPTZ catalog columns, not the app's naive-UTC schema), and there's a dedicated test (TheEvidenceRead_BindsAnExplicitTimestampTz_...) pinning it. Worth flagging explicitly since it contradicts the letter of the written convention, even though the reasoning for the exception holds up.

Parity: This is Darling-only by architecture (central Postgres/TimescaleDB store), and the existing CrossAppMcpToolInventoryPinTests.cs guard already documents get_store_metrics as having no Lite twin. No drift introduced.

Security: All reads are parameterized/static SQL, no string interpolation of untrusted input, no new secrets or file/process/network surface. current_user and catalog views only.

No substantive issues found — nicely defensive design (separate failure isolation per read, explicit null-vs-zero contract, live-store round-trip tests).

[claude]

LGTM — reviewed the job_history ownership-filter evidence feature (query logic, visibility/contradiction derivation, failure isolation, Lite/Darling parity, and security surface); no substantive issues, one non-blocking style note posted inline about the deliberate timestamptz-binding exception to the naive-UTC convention.