auth: prototype mutual TLS client authentication - #38420
Draft
jubrad wants to merge 2 commits into
Draft
Conversation
Proposes mutual TLS as an admission gate for external SQL connections, as an alternative to IP-based network policies. Egress IPs are unstable behind NAT gateways and cloud NAT pools, an IP is not an identity, and an allowlist has no rotation or revocation story. The obstacle specific to Materialize is balancerd, which terminates TLS and opens a separate pgwire connection to environmentd, so the process that sees the client certificate has no tenant configuration and the process with the configuration never sees the certificate. The design separates proof of possession from trust evaluation. The handshake proves the client holds the leaf's private key and only the terminating proxy can obtain that proof, but judging the issuer is a pure function of a chain and a set of trust anchors. balancerd therefore forwards the chain and environmentd judges it, so balancerd never learns any tenant's certificate authority and stays stateless. environmentd honours a forwarded chain only from a peer that authenticates against a configured proxy authority. Also records a three phase roadmap: system parameters with environmentd enforcing, then edge filtering at balancerd over a cacheable API, then CREATE CERTIFICATE AUTHORITY for per-authority scoping. Alternatives cover the catalog object syntax, edge filtering, TLS passthrough, and folding certificates into network policies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds mutual TLS as an admission gate for external pgwire connections, enforced before any credential exchange. Off by default; no behaviour change for deployments that do not opt in. Proof of possession and trust evaluation are separated, which is what makes this work through balancerd. balancerd requests client certificates with a permissive verify callback, so OpenSSL still validates CertificateVerify (possession is proven) while the trust decision is deferred to the party that holds the tenant's anchors. The chain is forwarded to environmentd in a new mz_client_cert startup parameter as base64 concatenated PEM, leaf first. balancerd never learns any tenant's certificate authority and stays stateless. environmentd honours a forwarded chain only from a peer whose own certificate chains to --tls-proxy-ca, so the balancerd to environmentd leg gains a client identity (--internal-tls-cert/--internal-tls-key). Without a proxy authority configured, forwarded certificates are ignored entirely. A client that supplies mz_client_cert itself is rejected, as with mz_forwarded_for. Configuration is three system parameters, following the OIDC precedent: mtls_client_ca (PEM bundle of trust anchors), mtls_mode (disable, allow, require), and mtls_identity_binding (none, common-name). Anchors and mode are read from the live ConfigSet rather than via get_system_vars, so the check costs no coordinator round trip on the connection path. Two details worth noting. Trust anchors are built with X509_V_FLAG_PARTIAL_CHAIN, so pinning an intermediate as the sole anchor works rather than failing with "unable to get local issuer certificate". A leaf with no Common Name fails a common-name binding outright, because pgwire defaults an absent user parameter to the empty string and a SPIFFE-shaped certificate would otherwise satisfy the binding. Also fixes a pre-existing balancerd bug this work surfaced: the startup parameter rejection paths called FramedConn::send without flushing, so a client rejected for supplying mz_connection_uuid or mz_forwarded_for saw a bare connection close instead of the error. Adds unit tests for the policy in mz-authenticator, integration tests for the direct path in src/environmentd/tests/mtls.rs, and tests for the forwarded path in src/balancerd/tests/server.rs covering anchor rotation, an unauthenticated proxy being disbelieved, and a forged mz_client_cert being refused. The HTTP, WebSocket, and webhook paths are in scope for the feature but not yet wired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype for the design in #38419. Includes that PR's doc commit, so review
just the second commit (
auth: prototype mutual TLS client authentication).Draft: this is a working prototype meant to validate the design, particularly
the
balancerdpath. Off by default, so no behaviour change for deploymentsthat do not opt in.
What it does
An operator configures trust anchors in SQL and requires that external SQL
connections present a certificate chaining to one:
Enforcement sits in
mz_pgwire::protocol::runimmediately before theauthenticator dispatches, so a client without an acceptable certificate is
turned away with
28000before any password prompt, SASL challenge, or tokenexchange.
How it gets through balancerd
balancerdterminates TLS, so it is the only party that can prove the clientholds its leaf's private key, but it has no tenant configuration to judge the
issuer with. It requests certificates with a permissive verify callback, so
OpenSSL still validates
CertificateVerifywhile the trust decision isdeferred, and forwards the chain in a new
mz_client_certstartup parameter(base64 concatenated PEM, leaf first, 64 KiB cap).
balancerdnever learns anytenant's certificate authority and stays stateless.
environmentdhonours a forwarded chain only from a peer whose own certificatechains to
--tls-proxy-ca, so the internal leg gains an identity(
--internal-tls-cert/--internal-tls-key). With no proxy authorityconfigured, forwarded certificates are ignored entirely. A client that supplies
mz_client_certitself is rejected, as withmz_forwarded_for.Connecting directly collapses this: the peer certificate is the client
certificate, so self-managed deployments get mTLS with no proxy involved.
Two things the tests changed about the design
X509_V_FLAG_PARTIAL_CHAIN. Without it, an operatorwho pins an intermediate as their sole anchor gets "unable to get local issuer
certificate" for the exact authority they named.
absent
userparameter to"", so a SPIFFE-shaped certificate (identity in aURI SAN, CN empty) would otherwise have satisfied a
common-namebinding.Common Names are also decoded from raw bytes rather than OpenSSL's
as_utf8,which truncates at an interior NUL and would let
admin\0evil.examplecompareequal to
admin.Unrelated fix included
balancerd's startup-parameter rejection paths calledFramedConn::sendwithout flushing, so a client rejected for supplying
mz_connection_uuidormz_forwarded_forsaw a bare connection close instead of the error. Thispredates the change but the new
mz_client_certcheck sits on the same path.Happy to split it out if preferred.
Tests
Added
src/environmentd/tests/mtls.rsfor the direct path (each mode, runtimemode and anchor changes, the CN binding, intermediate chains, the internal-user
exemption); unit tests in
mz_authenticator::client_certfor the policy(untrusted and expired chains, pinned intermediates, multi-anchor bundles,
malformed anchors, trust-store cache invalidation, proxy-authority scoping);
wire-format tests in
mz_pgwire_common::client_cert; and three tests insrc/balancerd/tests/server.rsfor the forwarded path, covering anchor rotationmid-flight, a balancer with no proxy identity having its assertion ignored, and
a client forging
mz_client_certbeing refused.Note that
test_balancerfails locally on macOS with-67609, which reproduceson a clean tree and is the known Darwin test-CA TLS issue, not a regression.
Not included
The HTTP, WebSocket, and webhook paths are in scope for the feature but not yet
wired. Phases 2 and 3 from the design (edge filtering,
CREATE CERTIFICATE AUTHORITY) are specified but not built.Release notes
This release will add opt-in mutual TLS client authentication for SQL
connections, configured with the
mtls_client_ca,mtls_mode, andmtls_identity_bindingsystem parameters. It is disabled by default.