Repository navigation
LivePostgresStoreFixture keeps using the connection after CREATE EXTENSION timescaledb kills the backend, masking the real cause #1922
Description
Hit while setting up a local rig to run the live Query Store tests for #1892. Cost about twenty minutes to diagnose because the reported error names neither the cause nor the file that caused it.
What happens
LivePostgresStoreFixture.InitializeAsync runs, in order, on ONE connection:
await PgMigrations.MigrateAsync(connection, cancellationToken);
TimescaleAvailable = await TimescaleSupport.TryEnableAsync(connection, null, cancellationToken);
_ = await DarlingModuleMap.EnsureTableAsync(connection, null, cancellationToken);
_baselineRelations = await RelationsAsync(connection, cancellationToken);
TryEnableAsync issues CREATE EXTENSION IF NOT EXISTS timescaledb. If the TimescaleDB library is present on disk but not in shared_preload_libraries, that statement does not merely error — it terminates the backend:
extension script file "timescaledb--2.28.1.sql", near line 63
server closed the connection unexpectedly
TryEnableAsync catches, logs "TimescaleDB not available — running in plain-PostgreSQL mode", and returns false, which is correct and by design. But the connection it was handed is now dead, and the next two calls use it. The failure surfaces as:
Collection fixture type 'Darling.Tests.LivePostgresStoreFixture' threw in InitializeAsync
---- System.InvalidOperationException : Connection is not open
at Darling.Tests.LivePostgresStoreFixture.RelationsAsync(...) LivePostgresStoreFixture.cs:line 251
Every live test in the collection then fails with that message. It names RelationsAsync, which is fine; the actual cause is three calls earlier and is not mentioned.
Why it is a real gap rather than "configure your rig"
The fixture stores TimescaleAvailable as a field, so a store WITHOUT TimescaleDB is an explicitly supported state — and this is the main way that state is reached on a fresh local rig, since the bundled pg-runtime.zip ships the extension files but a hand-rolled initdb has no shared_preload_libraries. The fixture supports the outcome but not the path to it.
It is also the signature #1794/#1810 chased: InvalidOperationException: Connection is not open from a helper frame, masking the real reason. That work fixed it for test TEARDOWN via LiveStoreCleanup; this is the same failure in fixture SETUP.
Suggested fix
Two small options, either enough:
- Have
InitializeAsyncre-open (or open a second connection) afterTryEnableAsync— the cheapest, and honest about the fact that the call can destroy the connection. - Have
TryEnableAsyncre-open the connection on its failure path before returningfalse, so its documented contract ("returns false, carry on in plain-PostgreSQL mode") is actually usable by callers. This is the better fix — the product's own startup path calls it the same way, so a Darling service started against a PostgreSQL with the extension present-but-unpreloaded would hit the identical dead connection right after migration.
Option 2 is worth checking against the service startup path before choosing; if DarlingWorker opens a fresh connection after TryEnableAsync anyway, then the exposure really is test-only and option 1 suffices.
Either way, a one-line note in the live-test setup docs that a rig needs shared_preload_libraries = 'timescaledb' would have saved the whole detour.
Rig recipe that works, for whoever hits this next
initdb -D <data> -U postgres --pwfile=<pw> -E UTF8 --auth-host=md5 --auth-local=md5
then append to postgresql.conf:
shared_preload_libraries = 'timescaledb'
timescaledb.telemetry_level = off
then start, CREATE ROLE darling LOGIN PASSWORD '...' (the migrations grant to it and fail with 42704: role "darling" does not exist otherwise), and CREATE DATABASE darling_test OWNER darling. Verified on the bundled PostgreSQL 18.4 + TimescaleDB 2.28.1: 4002 passed, 10 skipped.
Activity
erikdarlingdata commentedon Jul 31, 2026
Shipped in #1930 - fixture-side only, because the product path was PROVEN safe rather than assumed: DarlingWorker's TryEnableAsync connection is dedicated to its block and disposed there, and every downstream call gates on the flag, which is false on any backend kill since the catch fires before DetectAsync. Two exposed test sites (the filed fixture plus DarlingDeltaSeederTests, which had the wider blast radius) now enable via LiveTimescaleProbe, shaped after LiveStoreCleanup - the safest connection to risk is one nobody else holds. Measured on a deliberately unpreloaded cluster: before, the whole collection died with the masked 'Connection is not open' at RelationsAsync; after, 15 pass and the single failure is the good diagnostic someone had already written but which was unreachable until now. The product's safety is pinned by TimescaleEnableConnectionSafetyTests so it stays a fact rather than something that was true once. A bonus finding from review round one: CREATE EXTENSION lands in the FIRST schema on the search path (timescaledb is relocatable at install), so the SET search_path in the probe is load-bearing and its comment now says the real reason. Option 2 (hardening TryEnableAsync to reopen) deliberately not taken - no exposed caller justifies changing shipped behavior.
Fix the three erikdarlingdata#1895 residue items: erikdarlingdata#1891,
Enable TimescaleDB through LiveTimescaleProbe rather than a raw CREAT…
Give timescaledb.enable_job_execution_logging its own conf marker, so…
The fleet card's collection-health half is served from an hourly cont…
The daily summary's month read probes each day once and hashes its qu…
Route the quiesce live tests' TimescaleDB enable through LiveTimescal…