Skip to content

LivePostgresStoreFixture keeps using the connection after CREATE EXTENSION timescaledb kills the backend, masking the real cause #1922

Closed

Description

@erikdarlingdata

Hit while setting up a local rig to run the live Query Store tests for #1892. Cost about twenty minutes to diagnose because the reported error names neither the cause nor the file that caused it.

What happens

LivePostgresStoreFixture.InitializeAsync runs, in order, on ONE connection:

await PgMigrations.MigrateAsync(connection, cancellationToken);
TimescaleAvailable = await TimescaleSupport.TryEnableAsync(connection, null, cancellationToken);
_ = await DarlingModuleMap.EnsureTableAsync(connection, null, cancellationToken);
_baselineRelations = await RelationsAsync(connection, cancellationToken);

TryEnableAsync issues CREATE EXTENSION IF NOT EXISTS timescaledb. If the TimescaleDB library is present on disk but not in shared_preload_libraries, that statement does not merely error — it terminates the backend:

extension script file "timescaledb--2.28.1.sql", near line 63
server closed the connection unexpectedly

TryEnableAsync catches, logs "TimescaleDB not available — running in plain-PostgreSQL mode", and returns false, which is correct and by design. But the connection it was handed is now dead, and the next two calls use it. The failure surfaces as:

Collection fixture type 'Darling.Tests.LivePostgresStoreFixture' threw in InitializeAsync
---- System.InvalidOperationException : Connection is not open
   at Darling.Tests.LivePostgresStoreFixture.RelationsAsync(...) LivePostgresStoreFixture.cs:line 251

Every live test in the collection then fails with that message. It names RelationsAsync, which is fine; the actual cause is three calls earlier and is not mentioned.

Why it is a real gap rather than "configure your rig"

The fixture stores TimescaleAvailable as a field, so a store WITHOUT TimescaleDB is an explicitly supported state — and this is the main way that state is reached on a fresh local rig, since the bundled pg-runtime.zip ships the extension files but a hand-rolled initdb has no shared_preload_libraries. The fixture supports the outcome but not the path to it.

It is also the signature #1794/#1810 chased: InvalidOperationException: Connection is not open from a helper frame, masking the real reason. That work fixed it for test TEARDOWN via LiveStoreCleanup; this is the same failure in fixture SETUP.

Suggested fix

Two small options, either enough:

  1. Have InitializeAsync re-open (or open a second connection) after TryEnableAsync — the cheapest, and honest about the fact that the call can destroy the connection.
  2. Have TryEnableAsync re-open the connection on its failure path before returning false, so its documented contract ("returns false, carry on in plain-PostgreSQL mode") is actually usable by callers. This is the better fix — the product's own startup path calls it the same way, so a Darling service started against a PostgreSQL with the extension present-but-unpreloaded would hit the identical dead connection right after migration.

Option 2 is worth checking against the service startup path before choosing; if DarlingWorker opens a fresh connection after TryEnableAsync anyway, then the exposure really is test-only and option 1 suffices.

Either way, a one-line note in the live-test setup docs that a rig needs shared_preload_libraries = 'timescaledb' would have saved the whole detour.

Rig recipe that works, for whoever hits this next

initdb -D <data> -U postgres --pwfile=<pw> -E UTF8 --auth-host=md5 --auth-local=md5

then append to postgresql.conf:

shared_preload_libraries = 'timescaledb'
timescaledb.telemetry_level = off

then start, CREATE ROLE darling LOGIN PASSWORD '...' (the migrations grant to it and fail with 42704: role "darling" does not exist otherwise), and CREATE DATABASE darling_test OWNER darling. Verified on the bundled PostgreSQL 18.4 + TimescaleDB 2.28.1: 4002 passed, 10 skipped.

Activity

  1. erikdarlingdata commented on Jul 31, 2026

    @erikdarlingdata
    OwnerAuthor

    Shipped in #1930 - fixture-side only, because the product path was PROVEN safe rather than assumed: DarlingWorker's TryEnableAsync connection is dedicated to its block and disposed there, and every downstream call gates on the flag, which is false on any backend kill since the catch fires before DetectAsync. Two exposed test sites (the filed fixture plus DarlingDeltaSeederTests, which had the wider blast radius) now enable via LiveTimescaleProbe, shaped after LiveStoreCleanup - the safest connection to risk is one nobody else holds. Measured on a deliberately unpreloaded cluster: before, the whole collection died with the masked 'Connection is not open' at RelationsAsync; after, 15 pass and the single failure is the good diagnostic someone had already written but which was unreachable until now. The product's safety is pinned by TimescaleEnableConnectionSafetyTests so it stays a fact rather than something that was true once. A bonus finding from review round one: CREATE EXTENSION lands in the FIRST schema on the search path (timescaledb is relocatable at install), so the SET search_path in the probe is load-bearing and its comment now says the real reason. Option 2 (hardening TryEnableAsync to reopen) deliberately not taken - no exposed caller justifies changing shipped behavior.

  2. added a commit that references this issue on Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions