Repository navigation
job_history is silently empty on clusters predating ~2026-08-17, because its GUC lives in the one postgresql.conf block EnsureConfAppended cannot heal #3175
Description
timescaledb_information.job_history is empty on any store whose cluster predates ~2026-08-17, and a maximum over it returns zero rows — which reads as "no run exceeded the line" rather than as "this instrument is off".
The mechanism
job_history requires timescaledb.enable_job_execution_logging = on. The product sets it in the v1 postgresql.conf block — the one block EnsureConfAppended (DarlingManagedPostgres.cs:1313) cannot heal, because its marker is already present on any pre-existing cluster, so the append is skipped. Every later setting got its own marker for exactly this reason; #1681 put this one in v1.
Measured, two stores, same binary, opposite answers
| cluster initdb | markers present | GUC line | effective | job_history rows |
|---|---|---|---|---|
| 2026-07-17 (four days before 3.2.0) | all ten | absent | off, source = default |
1 row for 110 jobs |
| 2026-08-17 | all ten | present | on |
39,020 |
Why it matters more than a missing setting
Every job_history-based conclusion is scoped to the newer store and does not generalise. That includes the census behind #3174 and #3168 (the 594 → 778 → 896 growth series), the light-refresh census that falsified OtherHourlyRefreshObservedCeilingSeconds at 226.8 s, and any future maximum question about a background job.
On an older store the same query returns nothing, and nothing distinguishes that from a clean result. This is the failure shape the rest of the codebase guards against explicitly — an absence that reads as health.
It is also a prerequisite: the refresh-slot surfacing work (#3044's expired premise) wants a maximum, and the only other instrument that answers one is the hourly collect.store_metrics background-job series. So either the GUC is healed, or every max-answering read has to be built on store_metrics and say why.
Direction
Give the setting its own marker so EnsureConfAppended can heal it, the way every post-v1 setting is handled. Then decide what to do about clusters that already exist: the GUC needs a reload, and a store that has been running without it has no history to recover — so the honest outcome is "logging starts now", stated rather than implied.
Do not simply widen the v1 marker — that block is shared, and matching it more loosely would re-apply unrelated v1 content to healed clusters.
Also worth pinning
Nothing currently detects the condition. A read that depends on job_history cannot tell an empty table from a quiet fleet, so whatever heals this should also make the dependency legible — either a check that the GUC is on before trusting a maximum, or a documented statement that the read requires it.
Found while investigating #3044's surfacing premise. Related: #1681, #3119, #3166, #3168, #3174, #3044.
Activity
Give timescaledb.enable_job_execution_logging its own conf marker, so…
erikdarlingdata commentedon Sep 8, 2026
Claude posting for Erik Darling
Closed by PR #3177, merged to
dev. Closing by hand: merges land ondevwhile closing keywords resolve against the default branch.The GUC now has its own marker (
v11) and is moved out ofBuildConfAppendrather than duplicated — a copy would cost nothing at runtime and would leave the repo asserting the setting in the block that provably cannot deliver it.Widening the v1 marker was measured wrong rather than argued wrong. In a throwaway container, a conf carrying an operator's
shared_preload_libraries = 'timescaledb,pg_stat_statements'came back serving'timescaledb'alone once the v1 block was appended behind it — the setting is list-valued and the last assignment replaces the list.listen_addresseswould likewise re-assert loopback over a conf-configured exposure.Three pins, and one of them is the guard whose absence let this through:
ConfV1Block_ContentIsFrozen_...now fails on anything added to v1.EveryConfMarker_IsDistinct_AndNoneIsASubstringOfAnothercloses the other route, sinceEnsureConfAppendedasks withContainsand v1's marker is nearly a prefix of every later one.Reload: the append suffices where it matters —
EnsureConfAppendedruns beforepg_ctl start, so on a service-owned start the setting is live on the very start that writes it (measuredcontext = 'sighup'). The adopted-listener path issues no reload deliberately, consistent with v9/v10: signalling a postmaster this service did not start is the same class of act as stopping one, and a reload would apply this block while leaving restart-only settings from the same heal inert. For existing clusters logging starts then and there is nothing to recover — stated in the log line, not implied. AnALTER SYSTEM ... = offinpostgresql.auto.confstill wins, which is why the new check reads the effective value rather than the marker.Two findings worth more than the fix. The documentation half already existed and was passing while dead:
TheDescription_...NotAMaximumhas requiredget_store_metrics' description to namejob_historysince #3119, and passed the entire time that route returned nothing on older stores — a pin asserting a pointer cannot tell you the thing pointed at answers. And no shipped read queriesjob_history; every product read usesjob_stats, deliberately. So this gap affected our investigations rather than the product's own surfaces.Unverified: the heal against a real pre-existing field store, and the container measurements are TimescaleDB 2.30.0 / PG 17 rather than the bundled 2.28.1 / PG 18.
Say the census scope is a property of the read, now that the GUC heals
Re-derive the refresh grid so contention cannot depend on list positi…
get_store_metrics: the job_history block now says whose eyes its rows…