Skip to content

feat: add wildcard database template support#794

Open
hikionori wants to merge 3 commits intopgdogdev:mainfrom
hikionori:feat/wildcards-for-users-and-databases
Open

feat: add wildcard database template support#794
hikionori wants to merge 3 commits intopgdogdev:mainfrom
hikionori:feat/wildcards-for-users-and-databases

Conversation

@hikionori
Copy link

@hikionori hikionori commented Feb 27, 2026

Summary

Support wildcard database and user templates (name = "*") to dynamically route
unknown (user, database) pairs to a real PostgreSQL server without explicit
per-database configuration and without a proxy restart.


Problem

PgDog has no way to connect users to databases that aren't explicitly listed in
pgdog.toml. Any user/database pair not in the config gets an immediate "no
such database" error, forcing operators to restart the proxy every time a new
database is provisioned.


Solution

Add wildcard support for both databases and users entries. A database or
user whose name field is "*" acts as a catch-all template. When a client
connects to an unknown (user, database) pair, add_wildcard_pool looks for
the best-matching wildcard template, creates a pool on demand, and registers it
in the global DATABASES store — all without a restart or config reload.


Configuration

Minimal wildcard setup

# pgdog.toml — match any database for any user
[[databases]]
name = "*"
host = "127.0.0.1"
port = 5432
passthrough_auth = "enabled_plain"   # client credentials forwarded to PG
# users.toml
[[users]]
name = "*"
database = "*"
password = "secret"

With limits and automatic eviction

[general]
# Cap dynamic pool creation (0 = unlimited, default)
max_wildcard_pools = 100
# Remove idle dynamic pools after 5 minutes (0 = disabled, default)
wildcard_pool_idle_timeout = 300

Implementation

Wildcard matching

Four match priorities (highest wins):

Priority Pattern Notes
1 Exact user, wildcard database (user/*) Known user, any database
2 Wildcard user, exact database (*/db) Any user, specific database
3 Wildcard user, wildcard database (*/*) Full catch-all
4 No match Connection refused

exists_or_wildcard() API

New method on Databases returns true if a pool already exists or if a
wildcard template would match the (user, database) pair. Used during the
client-login handshake to decide whether to proceed without triggering pool
creation yet.

Dynamic pool creation (add_wildcard_pool)

  • Holds LOCK for the whole operation to prevent duplicate pools from racing
    threads.
  • Re-checks whether the pool already exists (double-checked locking).
  • Stores a full ConfigAndUsers snapshot at construction time so
    wildcard-pool creation is immune to concurrent SIGHUP reloads.
  • Injects the real database name into the wildcard template before calling
    new_pool.
  • Passthrough auth: client credentials are propagated to the backend
    connection config so the pool can authenticate to PostgreSQL and the
    proxy-level credential check succeeds — fixing a bug where passthrough auth
    would fail silently when a wildcard pool was created dynamically.

Pool limits (max_wildcard_pools)

wildcard_pool_count is tracked inside Databases and reset to zero on every
config reload, so the limit applies per-epoch. Once the limit is reached,
further connections return Ok(None) (no error, treated as "no such database")
until a slot is freed by eviction or SIGHUP.

Pool eviction (wildcard_pool_idle_timeout)

Dynamically-created pools are tracked in Databases::dynamic_pools: HashSet<User>.
A background task started once in init() (via WILDCARD_EVICTION: Lazy<()>)
periodically calls evict_idle_wildcard_pools():

  • Iterates dynamic_pools.
  • Calls Cluster::total_connections() — sum of idle + checked_out across
    all shards.
  • Removes pools with zero connections: cluster.shutdown(), removes from
    dynamic_pools, decrements wildcard_pool_count so new pools can fill the
    freed slot immediately, without a config reload.
  • All mutations use the standard clone-modify-store pattern on DATABASES.

When wildcard_pool_idle_timeout = 0 (default), automatic eviction is
disabled and pools only leave DATABASES on SIGHUP or restart.

SIGHUP / config-reload behaviour

reload_from_existing rebuilds Databases from the updated config file.
Wildcard pools are absent from the new config, so replace_databases drops
them, their connections drain naturally, and wildcard_pool_count resets to
zero. The next client connection recreates the pool from the (potentially
updated) wildcard template.


Tests

Unit tests (14, all passing — cargo nextest run --test-threads=1 wildcard)

Test What it covers
test_wildcard_database_simple_query End-to-end simple query through a wildcard pool
test_wildcard_exists_or_wildcard exists_or_wildcard returns true for wildcard-matched pairs
test_wildcard_add_pool_dynamic add_wildcard_pool creates and registers a pool on demand
test_wildcard_nonexistent_pg_database Pool creation succeeds for a non-existent Postgres DB
test_max_wildcard_pools_limit_enforced Third pool rejected (Ok(None)) when limit=2
test_max_wildcard_pools_zero_means_unlimited 5 pools created without error when limit=0
test_max_wildcard_pools_counter_resets_on_reload Counter resets after config reload
test_evict_idle_wildcard_pools_removes_idle_pool Idle pool (0 conns) removed by eviction
test_evict_idle_wildcard_pools_decrements_count Counter drops to 0; new pool fills the freed slot
test_evict_idle_wildcard_pools_noop_on_empty Static pools untouched when dynamic_pools is empty
test_wildcard_db_templates_populated wildcard_db_templates built correctly from * databases
test_wildcard_users_populated wildcard_users built correctly from * users
test_find_wildcard_match_priority Correct priority ordering among match types
test_no_wildcard_when_absent has_wildcard() is false when no * entries exist

Integration tests (integration/wildcard/)

Cover end-to-end scenarios against a live Postgres instance:

  • Trust auth + wildcard database routing
  • Passthrough auth with wildcard pools
  • Concurrent wildcard pool creation (race-condition safety)
  • Error handling for non-existent databases

Files changed

File Change
pgdog-config/src/general.rs max_wildcard_pools, wildcard_pool_idle_timeout fields
pgdog/src/backend/databases.rs Databases struct, add_wildcard_pool, evict_idle_wildcard_pools, WILDCARD_EVICTION static, from_config, reload_from_existing docs, exists_or_wildcard
pgdog/src/backend/pool/cluster.rs Cluster::total_connections()
pgdog/src/config/mod.rs load_test_wildcard, load_test_wildcard_with_limit test helpers
pgdog/src/frontend/client/query_engine/test/wildcard.rs 14 unit tests
integration/wildcard/ Integration test suite

Support wildcard database templates (name="*") to dynamically route unknown
database names to a real PostgreSQL server without explicit per-database
configuration.

Features:
- Wildcard database templates: name="*" entry matches any unmapped database
- Wildcard users: name="*" and/or database="*" for flexible auth routing
- Dynamic pool creation: pools created on-demand during client login
- Passthrough auth integration: client credentials correctly forwarded to
  backend in dynamically created pools

Implementation includes:
- exists_or_wildcard() API to check if pool exists or can be created
- add_wildcard_pool(user, db, passthrough_password) for dynamic creation
- Priority-based wildcard matching (explicit > user wildcard > db wildcard > both)
- Integration tests for trust/passthrough auth, concurrency, error handling
- Unit tests for wildcard matching and dynamic pool creation logic

Fixes passthrough auth failures when creating wildcard pools by ensuring
client passwords are propagated to the backend connection configuration.
@CLAassistant
Copy link

CLAassistant commented Feb 27, 2026

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants