Repository navigation
Adaptive connection-budget awareness: probe server max_connections/numbackends and back off daemon polling/concurrency under pressure (CritterWatch-surfaced) #3397
Description
Activity
Concrete impact datapoint from today, strengthening the case: our deployment's own
db-applystep (the out-of-band schema apply that walks all 512 shard databases before pods roll) failed for lack of free connections while the cluster was functionally idle — the per-database daemon polling from 3 nodes held ~1,400 of 2,400. Scaling the API down from 3 to 2 replicas freed enough headroom for the retry to succeed. That inversion — adding a node broke deployments, removing one fixed them — is exactly the failure mode a server-level connection budget would prevent: the apply job (or the daemon) would see utilization near the ceiling and back off/pace instead of failing.Thanks — this direction is a keeper, and the db-apply datapoint is exactly the failure mode worth designing against. Here's the plan we're committing to, sliced into "measure + surface" first, "adapt" second, plus two questions at the end.
Phase 1 — measure + surface (targeting Wolverine 6.19.0)
Observability only, no behavior changes:
- Server identity derives from the
DatabaseDescriptormetadata Wolverine already carries per store (Engine+ServerName), so there's no new connection-string parsing. One gap to close along the way:ServerNametoday is host-only, no port. - A cheap per-server probe for the used side of the budget —
select sum(numbackends) from pg_stat_databaseon PostgreSQL,sys.dm_exec_connectionson SQL Server — deduped by server key inside the existing per-node metrics sweep (the Sweep durability metrics sequentially per node instead of per-database timers (#3375) #3384 sweeper is already sequential per node), so it's one query per server per interval no matter how many tenant databases live there. - The max side is explicit configuration, not the probed
pg_settingsvalue. Any pooler in front (pgBouncer et al.) makes the server'smax_connectionsmisleading as a client budget, so you declare the budget per server; a probed max may serve as a clearly-labeled fallback when nothing is configured. - Scope: active under the sharded-database tenancy model (
MultiTenantedWithShardedDatabases) — the many-databases-per-server shape is where per-server budgeting earns its keep. PostgreSQL and SQL Server parity from the start. - Surfacing: a per-server used/max snapshot published alongside the existing
PersistedCountson the node metrics feed (which CritterWatch consumes on ServiceUpdates), plus matching OpenTelemetry gauges tagged by server.
This phase is also the instrument that answers the connection-breakdown question still open on #3376 — shipping it first helps both issues.
Phase 2 — adapt (opt-in, after #3376 resolves)
One correction to the issue as filed: the knobs you named —
SlowPollingTime/FastPollingTimeand the epic-486 concurrency governors (MaxConcurrentEventLoadsPerDatabase,MaxConcurrentBatchWritesPerDatabase) — live in JasperFx.Events'DaemonSettings, not Wolverine. So the back-off hooks have to ship in a JasperFx release (the same one already earmarked for the #3376 lifecycle work). Runtime mutability is split today: the high-water agent re-reads its polling times on every wait, so cadence is adaptable already, but the governors are semaphores sized at construction and Wolverine's own durability agent captures its polling period into timers at startup — both need structural hooks before anything can adapt them.The design itself, all default-off:
- A
ConnectionBudgetpolicy roughly as you sketched (Enabled,HighWaterUtilization,BackoffFactor) — plus a low-water mark and smoothing for hysteresis, since a single threshold will flap. - A node-local monitor per server key; each node adapts independently off the shared server signal, no cross-node coordination needed.
- Repeated probe failure counts as maximal pressure, not missing data — a probe that can't get a connection is the strongest possible signal, and it's precisely your db-apply scenario.
- Actuation: scale the durability polling cadence in Wolverine, and pace the daemon's high-water polling and load/batch-write governors through the new JasperFx hooks.
And as you anticipated, #3376's resolution changes the baseline any tuning works against, so this phase waits on it.
Phase 3 — CritterWatch control loop (noted, not scoped)
The control-queue pacing commands ("this server is under reporting load, back off") are a good closing-the-loop idea; we're treating that as a later phase rather than scoping it now.
Two questions for you
- Is your claims server talking to Postgres directly, or through pgBouncer/a pooler? That affects which numbers we document as meaningful for your topology (behind a transaction-pooling pgBouncer, server
numbackends≠ client connections). - The two questions still open on Wolverine-managed distribution: every node connects to every tenant database — scope daemon connections to owned agents #3376 — the
AddAsyncDaemonregistration alongside managed distribution, and thepg_stat_activitybreakdown (high-water query vs ordinary application traffic). Those answers gate the whole adapt phase.
- Server identity derives from the
- added a commit that references this issue
on Jul 14, 2026 I'm closing this for now until further testing
Feature request following #3375 (merged direction in #3384) and #3376, from the same 512-tenant-database production deployment. Idea in one line: let a node measure the database server's connection budget and adapt its polling/concurrency to it — and surface that budget through CritterWatch.
Motivation
After the metrics sweeper (#3384) and disabling the CritterWatch feed, our claims server still sits at ~1,400 connections of 2,400 (3 nodes × ~460): per-database daemon connections dominate, every backend younger than 10 minutes (constant reconnect churn), and 497 of 512 databases are connected from more than one node (that part is #3376's territory). The knobs to relieve pressure all exist —
SlowPollingTime/FastPollingTime,MaxConcurrentEventLoadsPerDatabase,BatchWriteThrottle,UpdateMetricsPeriod— but they are static: tuned for the worst case they waste responsiveness in quiet hours, tuned for responsiveness they starve heavy reporting/import workloads that need burst connection capacity on the same server.Proposal
SELECT setting FROM pg_settings WHERE name = 'max_connections'once, plussum(numbackends) FROM pg_stat_databaseper interval. One query per node per interval; the daemon already polls far more than that.ConnectionBudgetpolicy onDurabilitySettingswithEnabled,HighWaterUtilization,BackoffFactor— conservative defaults, opt-in.PersistedCountsper store; adding a per-serverused/maxbudget metric to the ServiceUpdates feed gives operators visibility and alerting, and CritterWatch's existing control queue could later carry explicit pacing commands ("this server is under reporting load, back off") for a human/automated override. That closes the loop: measure → show → act.Why server-level matters
With hundreds of tenant databases on one physical server, each database's pollers behave "reasonably" in isolation while their sum starves everything else — connection pressure is a server-scoped resource being consumed by database-scoped actors. The budget probe is the cheapest possible way to make those actors aware of the shared constraint.
Happy to prototype this behind an opt-in flag once #3376 lands (owned-agent scoping changes the baseline the tuning would work against).