Skip to content

auth: prototype mutual TLS client authentication - #38420

Draft
jubrad wants to merge 2 commits into
MaterializeInc:mainfrom
jubrad:mtls-poc
Draft

auth: prototype mutual TLS client authentication#38420
jubrad wants to merge 2 commits into
MaterializeInc:mainfrom
jubrad:mtls-poc

Conversation

@jubrad

@jubrad jubrad commented Aug 23, 2026

Copy link
Copy Markdown
Member

Prototype for the design in #38419. Includes that PR's doc commit, so review
just the second commit (auth: prototype mutual TLS client authentication).

Draft: this is a working prototype meant to validate the design, particularly
the balancerd path. Off by default, so no behaviour change for deployments
that do not opt in.

What it does

An operator configures trust anchors in SQL and requires that external SQL
connections present a certificate chaining to one:

ALTER SYSTEM SET mtls_client_ca = '-----BEGIN CERTIFICATE-----...';
ALTER SYSTEM SET mtls_mode = 'require';

Enforcement sits in mz_pgwire::protocol::run immediately before the
authenticator dispatches, so a client without an acceptable certificate is
turned away with 28000 before any password prompt, SASL challenge, or token
exchange.

How it gets through balancerd

balancerd terminates TLS, so it is the only party that can prove the client
holds its leaf's private key, but it has no tenant configuration to judge the
issuer with. It requests certificates with a permissive verify callback, so
OpenSSL still validates CertificateVerify while the trust decision is
deferred, and forwards the chain in a new mz_client_cert startup parameter
(base64 concatenated PEM, leaf first, 64 KiB cap). balancerd never learns any
tenant's certificate authority and stays stateless.

environmentd honours a forwarded chain only from a peer whose own certificate
chains to --tls-proxy-ca, so the internal leg gains an identity
(--internal-tls-cert/--internal-tls-key). With no proxy authority
configured, forwarded certificates are ignored entirely. A client that supplies
mz_client_cert itself is rejected, as with mz_forwarded_for.

Connecting directly collapses this: the peer certificate is the client
certificate, so self-managed deployments get mTLS with no proxy involved.

Two things the tests changed about the design

  • Trust anchors need X509_V_FLAG_PARTIAL_CHAIN. Without it, an operator
    who pins an intermediate as their sole anchor gets "unable to get local issuer
    certificate" for the exact authority they named.
  • A missing Common Name must not equal an empty username. pgwire defaults an
    absent user parameter to "", so a SPIFFE-shaped certificate (identity in a
    URI SAN, CN empty) would otherwise have satisfied a common-name binding.

Common Names are also decoded from raw bytes rather than OpenSSL's as_utf8,
which truncates at an interior NUL and would let admin\0evil.example compare
equal to admin.

Unrelated fix included

balancerd's startup-parameter rejection paths called FramedConn::send
without flushing, so a client rejected for supplying mz_connection_uuid or
mz_forwarded_for saw a bare connection close instead of the error. This
predates the change but the new mz_client_cert check sits on the same path.
Happy to split it out if preferred.

Tests

Added src/environmentd/tests/mtls.rs for the direct path (each mode, runtime
mode and anchor changes, the CN binding, intermediate chains, the internal-user
exemption); unit tests in mz_authenticator::client_cert for the policy
(untrusted and expired chains, pinned intermediates, multi-anchor bundles,
malformed anchors, trust-store cache invalidation, proxy-authority scoping);
wire-format tests in mz_pgwire_common::client_cert; and three tests in
src/balancerd/tests/server.rs for the forwarded path, covering anchor rotation
mid-flight, a balancer with no proxy identity having its assertion ignored, and
a client forging mz_client_cert being refused.

Note that test_balancer fails locally on macOS with -67609, which reproduces
on a clean tree and is the known Darwin test-CA TLS issue, not a regression.

Not included

The HTTP, WebSocket, and webhook paths are in scope for the feature but not yet
wired. Phases 2 and 3 from the design (edge filtering, CREATE CERTIFICATE AUTHORITY) are specified but not built.

Release notes

This release will add opt-in mutual TLS client authentication for SQL
connections, configured with the mtls_client_ca, mtls_mode, and
mtls_identity_binding system parameters. It is disabled by default.

jubrad and others added 2 commits August 22, 2026 22:49
Proposes mutual TLS as an admission gate for external SQL connections, as an
alternative to IP-based network policies. Egress IPs are unstable behind NAT
gateways and cloud NAT pools, an IP is not an identity, and an allowlist has no
rotation or revocation story.

The obstacle specific to Materialize is balancerd, which terminates TLS and
opens a separate pgwire connection to environmentd, so the process that sees the
client certificate has no tenant configuration and the process with the
configuration never sees the certificate.

The design separates proof of possession from trust evaluation. The handshake
proves the client holds the leaf's private key and only the terminating proxy
can obtain that proof, but judging the issuer is a pure function of a chain and
a set of trust anchors. balancerd therefore forwards the chain and environmentd
judges it, so balancerd never learns any tenant's certificate authority and
stays stateless. environmentd honours a forwarded chain only from a peer that
authenticates against a configured proxy authority.

Also records a three phase roadmap: system parameters with environmentd
enforcing, then edge filtering at balancerd over a cacheable API, then
CREATE CERTIFICATE AUTHORITY for per-authority scoping. Alternatives cover the
catalog object syntax, edge filtering, TLS passthrough, and folding certificates
into network policies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds mutual TLS as an admission gate for external pgwire connections, enforced
before any credential exchange. Off by default; no behaviour change for
deployments that do not opt in.

Proof of possession and trust evaluation are separated, which is what makes this
work through balancerd. balancerd requests client certificates with a permissive
verify callback, so OpenSSL still validates CertificateVerify (possession is
proven) while the trust decision is deferred to the party that holds the tenant's
anchors. The chain is forwarded to environmentd in a new mz_client_cert startup
parameter as base64 concatenated PEM, leaf first. balancerd never learns any
tenant's certificate authority and stays stateless.

environmentd honours a forwarded chain only from a peer whose own certificate
chains to --tls-proxy-ca, so the balancerd to environmentd leg gains a client
identity (--internal-tls-cert/--internal-tls-key). Without a proxy authority
configured, forwarded certificates are ignored entirely. A client that supplies
mz_client_cert itself is rejected, as with mz_forwarded_for.

Configuration is three system parameters, following the OIDC precedent:
mtls_client_ca (PEM bundle of trust anchors), mtls_mode (disable, allow,
require), and mtls_identity_binding (none, common-name). Anchors and mode are
read from the live ConfigSet rather than via get_system_vars, so the check costs
no coordinator round trip on the connection path.

Two details worth noting. Trust anchors are built with
X509_V_FLAG_PARTIAL_CHAIN, so pinning an intermediate as the sole anchor works
rather than failing with "unable to get local issuer certificate". A leaf with no
Common Name fails a common-name binding outright, because pgwire defaults an
absent user parameter to the empty string and a SPIFFE-shaped certificate would
otherwise satisfy the binding.

Also fixes a pre-existing balancerd bug this work surfaced: the startup parameter
rejection paths called FramedConn::send without flushing, so a client rejected
for supplying mz_connection_uuid or mz_forwarded_for saw a bare connection close
instead of the error.

Adds unit tests for the policy in mz-authenticator, integration tests for the
direct path in src/environmentd/tests/mtls.rs, and tests for the forwarded path
in src/balancerd/tests/server.rs covering anchor rotation, an unauthenticated
proxy being disbelieved, and a forged mz_client_cert being refused.

The HTTP, WebSocket, and webhook paths are in scope for the feature but not yet
wired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant