Add auth_query config and make pass in pools optional

Add `auth_query` feature

Adds a feature that allows setting auth passthrough for md5 auth.
It adds 4 new general config parameters:

- `auth_query`: An string containing a query that will be executed on boot
to obtain the hash of a given user. This query have to use a placeholder `$1`,
so pgcat can replace it with the user its trying to fetch the hash from.
- `auth_query_user`: The user to use for connecting to the server and executing the
auth_query.
- `auth_query_password`: The password to use for connecting to the server and executing the
auth_query.
- `auth_query_database`: The database to use for connecting to the server and executing the
auth_query.

The behavior is, at boot time, when validating server connections, a hash is fetched per server
and stored there. When new server connections are created, that hash is used for creating them,
if the hash could not be obtained for whatever reason, it falls back to the password set.

Client connections are also authenticated using the obtained hash.

Fix error message info parameters (username <-> pool_name)

Make reporter optional in server definition

We use server connections for pools but also for fetching auth
hashes if query_auth is set up. As we do not want to mess with stats
this change allows reporter to be optional.

Change AuthPassthrough::fetch_hah to use Address username

Allow query_auth configuration reload

This adds support for reloading query_auth configuration.

Current implementation:

- Checks whether `auth_query` configuration is active, if so, it fetches
  (or refetches) hashes from servers and sets up pools. When it detects a
  password change, the pool is dropped and a new one is created, current
  established connections will be left connected.
- When we go from 'auth_query' configured to unconfigured the reload logic
  detects it and drops Md5 hashes stored in connection pools, so cleartext
  passwords are used instead.

Add tests for `query_auth` reload functionality

This change adds tests for query_auth configuration reload and also
does some refactoring.

- Now postgres containers in `docker-compose` are set to use md5 auth.
  Note that this still uses scram if the password is stored using scram.

- Current tests are:
    - Test that activating the functionality can be added without restarting.
    - Test that with a failing `query_auth`, clear text passwords are used.
    - Test that with a correct `query_auth` and without clear text passwords,
      auth works as expected.
    - Test that when we change a password in postgres and reload, the new password is obtained,
      and the pool rotated.
    - Test that we can use and ENV var to set `auth_query_password`.

Change auth_query config to be per pool instead of global.

Given that we can connect to different username/databases/servers using
connection pools, it makes sense that `auth_query` should be configured in a
per pool basis.

This change implements that and also leaves the global parameters so pool
configuration can inherit global ones.

Note that now, `auth_query_database` is dropped in favor of the pool configured
database.

Author: magec

.circleci/pgcat.toml

@@ -50,6 +50,10 @@ tls_private_key = ".circleci/server.key"
 admin_username = "admin_user"
 admin_password = "admin_pass"
 
+# auth_query = "SELECT * FROM public.user_lookup('$1');"
+# auth_query_user = "md5_auth_user"
+# auth_query_password = "secret"
+
 # pool
 # configs are structured as pool.<pool_name>
 # the pool_name is what clients use as database name when connecting

.circleci/query_auth_test.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+# Query auth test
+
+set -e
+set -o xtrace
+
+export LOCAL_IP=$(hostname -i)
+
+# In config file we have this commented:
+#  [general]
+#  ...
+#  auth_query = "SELECT * FROM public.user_lookup('$1');"
+#  auth_query_user = "md5_auth_user"
+#  auth_query_password = "secret"
+#  ...
+
+# Before (sets up auth_query in postgres and pgcat)
+PGDATABASE=shard0 PGPASSWORD=postgres exec_in_servers postgres tests/sharding/query_auth_setup.sql
+PGDATABASE=shard0 PGPASSWORD=postgres exec_in_servers postgres tests/sharding/query_auth_setup_function.sql
+
+sed -i 's/^# auth_query/auth_query/' .circleci/pgcat.toml
+
+# TEST_WRONG_AUTH_QUERY BEGIN
+#    When auth_query fails...
+PGDATABASE=shard0 \
+    PGPASSWORD=postgres \
+    psql -e -h 127.0.0.1 -p 5432 -U postgres -c "REVOKE ALL ON FUNCTION public.user_lookup(text) FROM public, md5_auth_user;"
+
+kill -SIGHUP $(pgrep pgcat) # Reload config
+sleep 0.2
+
+#    ... we can still connect.
+echo "When query_auth_config is wrong, we fall back to passwords set in cleartext."
+psql -U sharding_user -h 127.0.0.1 -p 6432 -c 'SELECT 1'
+
+# After
+PGDATABASE=shard0 \
+    PGPASSWORD=postgres \
+    psql -e -h 127.0.0.1 -p 5432 -U postgres -c "GRANT EXECUTE ON FUNCTION public.user_lookup(text) TO md5_auth_user;"
+# TEST_WRONG_AUTH_QUERY END
+
+# TEST_AUTH_QUERY BEGIN
+#    When no passwords are specified in config file...
+sed -i 's/^password =/# password =/' .circleci/pgcat.toml
+kill -SIGHUP $(pgrep pgcat) # Reload config
+sleep 0.2
+
+#    ... we can still connect
+echo "When no passwords are specified in config file, and query_auth is set, we can still connect"
+psql -U sharding_user -h 127.0.0.1 -p 6432 -c 'SELECT 1'
+# TEST_AUTH_QUERY END
+
+# TEST_PASSWORD_CHANGE BEGIN
+#    When we change the password of a user in postgres...
+PGDATABASE=shard0 \
+    PGPASSWORD=postgres \
+    psql -e -h 127.0.0.1 -p 5432 -U postgres \
+    -c "ALTER USER sharding_user WITH ENCRYPTED PASSWORD 'md5b47a59331e93a520d20e90fc8a3355a4'; --- another_sharding_password"
+
+#    ... we can connect using the new password
+echo "When we change pass in postgres the new hash is fetched after a connection error."
+PGPASSWORD=another_sharding_password psql -U sharding_user -h "${LOCAL_IP}" -p 6432 -c 'SELECT 1'
+# TEST_PASSWORD_CHANGE END
+
+# After
+PGDATABASE=shard0 PGPASSWORD=postgres exec_in_servers postgres tests/sharding/query_auth_teardown_function.sql
+PGDATABASE=shard0 PGPASSWORD=postgres exec_in_servers postgres tests/sharding/query_auth_teardown.sql
+sed -i 's/^auth_query/# auth_query/' .circleci/pgcat.toml
+sed -i 's/^# password =/password =/' .circleci/pgcat.toml
+
+kill -SIGHUP $(pgrep pgcat)
+sleep 0.2

.circleci/run_tests.sh

@@ -14,11 +14,17 @@ function start_pgcat() {
     sleep 1
 }
 
+# Executes an sql file in every postgres server set up in tests
+# exec_in_servers user file
+function exec_in_servers() {
+    PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 5432 -U $1 -f $2
+    PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 7432 -U $1 -f $2
+    PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 8432 -U $1 -f $2
+    PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 9432 -U $1 -f $2
+}
+
 # Setup the database with shards and user
-PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 5432 -U postgres -f tests/sharding/query_routing_setup.sql
-PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 7432 -U postgres -f tests/sharding/query_routing_setup.sql
-PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 8432 -U postgres -f tests/sharding/query_routing_setup.sql
-PGPASSWORD=postgres psql -e -h 127.0.0.1 -p 9432 -U postgres -f tests/sharding/query_routing_setup.sql
+PGPASSWORD=postgres exec_in_servers postgres tests/sharding/query_routing_setup.sql
 
 PGPASSWORD=sharding_user pgbench -h 127.0.0.1 -U sharding_user shard0 -i
 PGPASSWORD=sharding_user pgbench -h 127.0.0.1 -U sharding_user shard1 -i
@@ -34,11 +40,15 @@ toxiproxy-cli create -l 127.0.0.1:5433 -u 127.0.0.1:5432 postgres_replica
 start_pgcat "info"
 
 # Check that prometheus is running
-curl --fail localhost:9930/metrics
+#curl --fail localhost:9930/metrics
 
 export PGPASSWORD=sharding_user
 export PGDATABASE=sharded_db
 
+# Query auth test
+echo 'THIS IS A TEST'
+source .circleci/query_auth_test.sh
+
 # pgbench test
 pgbench -U sharding_user -i -h 127.0.0.1 -p 6432
 pgbench -U sharding_user -h 127.0.0.1 -p 6432 -t 500 -c 2 --protocol simple -f tests/pgbench/simple.sql

.rustfmt.toml

@@ -0,0 +1 @@
+edition = "2021"

Cargo.lock

@@ -4,7 +4,6 @@ version = "0.6.0-alpha1"
 edition = "2021"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
-
 [dependencies]
 tokio = { version = "1", features = ["full"] }
 bytes = "1"
@@ -37,6 +36,8 @@ exitcode = "1.1.2"
 futures = "0.3"
 socket2 = { version = "0.4.7", features = ["all"] }
 nix = "0.26.2"
+postgres-protocol = "0.6.4"
+fallible-iterator = "0.2"
 
 [target.'cfg(not(target_env = "msvc"))'.dependencies]
 jemallocator = "0.5.0"

Cargo.toml

@@ -0,0 +1,132 @@
+#
+# PgCat config example.
+#
+
+#
+# General pooler settings
+[general]
+# What IP to run on, 0.0.0.0 means accessible from everywhere.
+host = "0.0.0.0"
+
+# Port to run on, same as PgBouncer used in this example.
+port = 6432
+
+# Whether to enable prometheus exporter or not.
+enable_prometheus_exporter = true
+
+# Port at which prometheus exporter listens on.
+prometheus_exporter_port = 9930
+
+# How long to wait before aborting a server connection (ms).
+connect_timeout = 5000
+
+# How long an idle connection with a server is left open (ms).
+idle_timeout = 30000
+
+# How much time to give the health check query to return with a result (ms).
+healthcheck_timeout = 1000
+
+# How long to keep connection available for immediate re-use, without running a healthcheck query on it
+healthcheck_delay = 30000
+
+# How much time to give clients during shutdown before forcibly killing client connections (ms).
+shutdown_timeout = 60000
+
+# For how long to ban a server if it fails a health check (seconds).
+ban_time = 60 # seconds
+
+# If we should log client connections
+log_client_connections = false
+
+# If we should log client disconnections
+log_client_disconnections = false
+
+# Reload config automatically if it changes.
+autoreload = false
+
+# Number of worker threads the Runtime will use (4 by default).
+worker_threads = 5
+
+# TLS
+# tls_certificate = "server.cert"
+# tls_private_key = "server.key"
+
+# Credentials to access the virtual administrative database (pgbouncer or pgcat)
+# Connecting to that database allows running commands like `SHOW POOLS`, `SHOW DATABASES`, etc..
+admin_username = "admin_user"
+admin_password = "admin_pass"
+
+auth_query = "SELECT 1"
+auth_query_user = "testuser"
+
+# pool
+# configs are structured as pool.<pool_name>
+# the pool_name is what clients use as database name when connecting
+# For the example below a client can connect using "postgres://sharding_user:sharding_user@pgcat_host:pgcat_port/sharded_db"
+[pools.sharded_db]
+# Pool mode (see PgBouncer docs for more).
+# session: one server connection per connected client
+# transaction: one server connection per client transaction
+pool_mode = "transaction"
+
+# If the client doesn't specify, route traffic to
+# this role by default.
+#
+# any: round-robin between primary and replicas,
+# replica: round-robin between replicas only without touching the primary,
+# primary: all queries go to the primary unless otherwise specified.
+default_role = "any"
+
+# Query parser. If enabled, we'll attempt to parse
+# every incoming query to determine if it's a read or a write.
+# If it's a read query, we'll direct it to a replica. Otherwise, if it's a write,
+# we'll direct it to the primary.
+query_parser_enabled = true
+
+# If the query parser is enabled and this setting is enabled, the primary will be part of the pool of databases used for
+# load balancing of read queries. Otherwise, the primary will only be used for write
+# queries. The primary can always be explicitly selected with our custom protocol.
+primary_reads_enabled = true
+
+# So what if you wanted to implement a different hashing function,
+# or you've already built one and you want this pooler to use it?
+#
+# Current options:
+#
+# pg_bigint_hash: PARTITION BY HASH (Postgres hashing function)
+# sha1: A hashing function based on SHA1
+#
+sharding_function = "pg_bigint_hash"
+
+# Automatically parse this from queries and route queries to the right shard!
+automatic_sharding_key = "id"
+
+# Idle timeout can be overwritten in the pool
+idle_timeout = 40000
+
+# Credentials for users that may connect to this cluster
+[pools.sharded_db.users.0]
+username = "sharding_user"
+# Maximum number of server connections that can be established for this user
+# The maximum number of connection from a single Pgcat process to any database in the cluster
+# is the sum of pool_size across all users.
+pool_size = 9
+
+# Maximum query duration. Dangerous, but protects against DBs that died in a non-obvious way.
+statement_timeout = 0
+
+[pools.sharded_db.users.1]
+username = "other_user"
+password = "other_user"
+pool_size = 21
+statement_timeout = 15000
+
+# Shard 0
+[pools.sharded_db.shards.0]
+# [ host, port, role ]
+servers = [
+    [ "127.0.0.1", 5432, "primary" ],
+    [ "localhost", 5432, "replica" ]
+]
+# Database name (e.g. "postgres")
+database = "shard0"