perf(datastore): fill connection pools before startup finishes

[vroldanbet]

What

SpiceDB now waits for its Postgres and CockroachDB connection pools to fill to the configured minimum number of connections before it finishes starting up. If a pool cannot be filled, the process fails to start with an error saying which pool fell short and by how much.

Why

When Kubernetes rolls out new SpiceDB pods, it marks a pod Ready as soon as its health check passes, and the load balancer starts sending it traffic. Until now that could happen while the pod's connection pools were still empty. The pgx pool constructor does not actually connect to anything — it kicks off a background goroutine to open the connections and returns immediately — so SpiceDB could finish starting and report itself healthy with nothing in its pools.

The new tests measure this against a database container on loopback, the most favourable case there is: Postgres returned from its constructor with 4 of 20 read connections open, CockroachDB with 8 of 10. On a real network that gap scales with the round trip.

The result is that the first burst of traffic onto a brand-new pod has to open every connection it uses, one per request, at once. That is a thundering herd against the database at exactly the moment a rollout is shifting traffic onto that pod, and it shows up to users as a latency spike or as errors. Rolling pods is supposed to be routine.

The minimum connection count is a statement about how much capacity an instance needs before it should take traffic. So an instance that cannot reach it now fails to start, rather than starting cold and finding out under load.

One trade-off to be explicit about, because the failure mode is new even though the misconfiguration behind it is not: during a rolling deploy maxSurge means old and new pods coexist, so a database sized for exactly N pods cannot satisfy N+1 sets of warm pools, and the rollout will now stall on a pod that fails to start instead of on a pod that starts and then returns connection errors under load. That is an under-provisioned connection limit surfacing at deploy time rather than at peak, and the pod does not survive either way.

How

Both datastore constructors now open the minimum number of connections and hand them straight back to the pool before returning. There is no API in pgx to ask for this (LazyConnect was removed on purpose and a hook for it was declined upstream), so it is a short loop in our code, shared between the two engines.

Details worth a reviewer's attention:

• The target is min(MinConns, MaxConns). A minimum above the maximum is a configuration SpiceDB accepts today with only a warning, and the maximum is a hard limit, so warming to the raw minimum would hang until the timeout and then refuse to start a server that was perfectly able to serve traffic.
• Warm-up has its own 30-second budget, separate from the rest of start-up, so that "the pool could not be filled" does not depend on how long the preceding checks happened to take.
• The read and write pools are warmed at the same time, not one after the other. This matters most for CockroachDB, which rate-limits new connections.
• The failure message names the pool and the counts, e.g. read connection pool did not reach its configured minimum of 20 connections (established 10); lower the minimum connection count for this pool, or raise the connection limit on the database: ... FATAL: sorry, too many clients already.
• A datastore built with no pool options now keeps one connection, not a core count's worth. ConfigurePgx used to default the minimum to the maximum unconditionally. Every path that serves real traffic sets both bounds from the --datastore-conn-pool-* flags, so production is unchanged — but a caller that passes no pool options (the test harness, and embedders using SpiceDB as a library) inherited pgx's own max(4, NumCPU) maximum as a minimum, which with this PR would mean 16 read plus 16 write connections established synchronously per datastore on a 16-vCPU machine. Those callers need a pool that works, not a pool that is full, so the default is now one connection with the rest filling on demand. This keeps the test suites off the warm-up path while still exercising it once per datastore.
• Also fixes a data race alongside this: the Postgres credentials-provider hook wrote its result into the constructor's own error variable, and pgx runs that hook on one goroutine per connection. It was always wrong; warming the pools makes those goroutines overlap the constructor rather than run after it.

What it costs

Constructor wall time with the shipped defaults (20 read, 10 write connections). Latency injected with a TCP proxy that delays every forwarded chunk.

	before	after
Postgres, loopback	11.5ms	16.6ms
Postgres, 10ms RTT	52.2ms	52.6ms
Postgres, 50ms RTT	219.5ms	219.8ms
Postgres, 200ms RTT	828.6ms	821.0ms
Postgres, 16 serial read replicas, loopback	191ms	251ms
Postgres, 16 serial read replicas, 50ms RTT	3.72s	3.65s
CockroachDB, loopback	206ms	2.01s

Past loopback the Postgres cost vanishes, because the connections were already being opened in parallel in the background and the checks the constructor already runs cover the time.

CockroachDB is the outlier, and it is not connection setup: it rate-limits new connections to one per --datastore-connect-rate (100ms by default), so twenty connections take 1.9 seconds whoever opens them. That time was always being spent — previously after start-up, slowing down the traffic it overlapped. Warming the two pools concurrently is what keeps this at 2.0s instead of 3.0s.

The test suites pay almost none of this, which matters because the CockroachDB suite is the CI critical path. Constructing a CockroachDB datastore through the test harness, on a 12-core machine (medians of 5 runs):

	median
before this PR	261ms
with warm-up, minimum defaulted to the maximum	1.44s
with warm-up, minimum defaulted to one (shipped)	431ms

End to end, go test -tags datastore ./internal/datastore/crdb/... on the same machine took 942s with the minimum defaulted to the maximum and 747s as shipped — back in line with the 758s that #3337 brought the CockroachDB job down to, so this does not give back that win.

Not changed here

MySQL and Spanner have the same shape of problem and are left alone in this PR. MySQL uses database/sql, which has no minimum-connections concept at all (the --datastore-conn-pool-*-min-open flag is silently ignored for it) and opens connections purely on demand; only the one connection its start-up queries use is warm. Spanner's client library has removed its session pool entirely in favour of a single multiplexed session, created on a background goroutine the constructor does not wait for — but its readiness check runs a real query, which forces that session to exist, and it is one session rather than N connections.

References

• pgxpool.NewWithConfig returns after starting go func() { p.createIdleResources(...) } — pgx v5.9.2, pgxpool/pool.go:334-340
• No synchronous warm-up exists upstream: pgxpool: Callback for after pool initialization jackc/pgx#1660
• Builds on fix(postgres): stop startup cancelling its own connection warm-up #3333, which stopped that background fill from running on a cancelled context. This PR targets that branch.

[codecov]

Codecov Report

❌ Patch coverage is 67.02128% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/datastore/postgres/postgres.go	40.63%	18 Missing and 1 partial ⚠️
internal/datastore/crdb/crdb.go	20.00%	11 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: 8fe51ed	Previous: b4a0719	Ratio
BenchmarkSerializeUnion/subs=128 (github.com/authzed/spicedb/pkg/query) - MB/s	156.26 MB/s	38.98 MB/s	4.01
BenchmarkDeserializeRealistic (github.com/authzed/spicedb/pkg/query) - MB/s	83.74 MB/s	21.98 MB/s	3.81

This comment was automatically generated by workflow using github-action-benchmark.