feat: add wildcard database template support#794
Open
hikionori wants to merge 3 commits intopgdogdev:mainfrom
Open
feat: add wildcard database template support#794hikionori wants to merge 3 commits intopgdogdev:mainfrom
hikionori wants to merge 3 commits intopgdogdev:mainfrom
Conversation
Support wildcard database templates (name="*") to dynamically route unknown database names to a real PostgreSQL server without explicit per-database configuration. Features: - Wildcard database templates: name="*" entry matches any unmapped database - Wildcard users: name="*" and/or database="*" for flexible auth routing - Dynamic pool creation: pools created on-demand during client login - Passthrough auth integration: client credentials correctly forwarded to backend in dynamically created pools Implementation includes: - exists_or_wildcard() API to check if pool exists or can be created - add_wildcard_pool(user, db, passthrough_password) for dynamic creation - Priority-based wildcard matching (explicit > user wildcard > db wildcard > both) - Integration tests for trust/passthrough auth, concurrency, error handling - Unit tests for wildcard matching and dynamic pool creation logic Fixes passthrough auth failures when creating wildcard pools by ensuring client passwords are propagated to the backend connection configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Support wildcard database and user templates (
name = "*") to dynamically routeunknown
(user, database)pairs to a real PostgreSQL server without explicitper-database configuration and without a proxy restart.
Problem
PgDog has no way to connect users to databases that aren't explicitly listed in
pgdog.toml. Any user/database pair not in the config gets an immediate "nosuch database" error, forcing operators to restart the proxy every time a new
database is provisioned.
Solution
Add wildcard support for both
databasesandusersentries. A database oruser whose
namefield is"*"acts as a catch-all template. When a clientconnects to an unknown
(user, database)pair,add_wildcard_poollooks forthe best-matching wildcard template, creates a pool on demand, and registers it
in the global
DATABASESstore — all without a restart or config reload.Configuration
Minimal wildcard setup
With limits and automatic eviction
Implementation
Wildcard matching
Four match priorities (highest wins):
user/*)*/db)*/*)exists_or_wildcard()APINew method on
Databasesreturnstrueif a pool already exists or if awildcard template would match the
(user, database)pair. Used during theclient-login handshake to decide whether to proceed without triggering pool
creation yet.
Dynamic pool creation (
add_wildcard_pool)LOCKfor the whole operation to prevent duplicate pools from racingthreads.
ConfigAndUserssnapshot at construction time sowildcard-pool creation is immune to concurrent SIGHUP reloads.
new_pool.connection config so the pool can authenticate to PostgreSQL and the
proxy-level credential check succeeds — fixing a bug where passthrough auth
would fail silently when a wildcard pool was created dynamically.
Pool limits (
max_wildcard_pools)wildcard_pool_countis tracked insideDatabasesand reset to zero on everyconfig reload, so the limit applies per-epoch. Once the limit is reached,
further connections return
Ok(None)(no error, treated as "no such database")until a slot is freed by eviction or SIGHUP.
Pool eviction (
wildcard_pool_idle_timeout)Dynamically-created pools are tracked in
Databases::dynamic_pools: HashSet<User>.A background task started once in
init()(viaWILDCARD_EVICTION: Lazy<()>)periodically calls
evict_idle_wildcard_pools():dynamic_pools.Cluster::total_connections()— sum ofidle + checked_outacrossall shards.
cluster.shutdown(), removes fromdynamic_pools, decrementswildcard_pool_countso new pools can fill thefreed slot immediately, without a config reload.
DATABASES.When
wildcard_pool_idle_timeout = 0(default), automatic eviction isdisabled and pools only leave
DATABASESon SIGHUP or restart.SIGHUP / config-reload behaviour
reload_from_existingrebuildsDatabasesfrom the updated config file.Wildcard pools are absent from the new config, so
replace_databasesdropsthem, their connections drain naturally, and
wildcard_pool_countresets tozero. The next client connection recreates the pool from the (potentially
updated) wildcard template.
Tests
Unit tests (14, all passing —
cargo nextest run --test-threads=1 wildcard)test_wildcard_database_simple_querytest_wildcard_exists_or_wildcardexists_or_wildcardreturns true for wildcard-matched pairstest_wildcard_add_pool_dynamicadd_wildcard_poolcreates and registers a pool on demandtest_wildcard_nonexistent_pg_databasetest_max_wildcard_pools_limit_enforcedOk(None)) when limit=2test_max_wildcard_pools_zero_means_unlimitedtest_max_wildcard_pools_counter_resets_on_reloadtest_evict_idle_wildcard_pools_removes_idle_pooltest_evict_idle_wildcard_pools_decrements_counttest_evict_idle_wildcard_pools_noop_on_emptydynamic_poolsis emptytest_wildcard_db_templates_populatedwildcard_db_templatesbuilt correctly from*databasestest_wildcard_users_populatedwildcard_usersbuilt correctly from*userstest_find_wildcard_match_prioritytest_no_wildcard_when_absenthas_wildcard()is false when no*entries existIntegration tests (
integration/wildcard/)Cover end-to-end scenarios against a live Postgres instance:
Files changed
pgdog-config/src/general.rsmax_wildcard_pools,wildcard_pool_idle_timeoutfieldspgdog/src/backend/databases.rsDatabasesstruct,add_wildcard_pool,evict_idle_wildcard_pools,WILDCARD_EVICTIONstatic,from_config,reload_from_existingdocs,exists_or_wildcardpgdog/src/backend/pool/cluster.rsCluster::total_connections()pgdog/src/config/mod.rsload_test_wildcard,load_test_wildcard_with_limittest helperspgdog/src/frontend/client/query_engine/test/wildcard.rsintegration/wildcard/