GEP: Client Certificate Verification for Gateway Listeners

[jpeach]

What would you like to be added:

The ability for a HTTPS (or TLS generally) endpoint to require that the client present a certificate that can be validated according to some configurable policy.

Why is this needed:

As an application developer, I want to restrict access to my application to a certain audience of clients. The audience is defined by one or more of

• a collection of specific TLS certificates (maybe by hash)
• a collection of subject names in certificates
• a collection of certificates issued by a specific (unique) CA

I want the infrastructure to guarantee that I only receive client traffic that originates from this audience.

/kind user-story

[bowei]

Is this mTLS? Can you add to the issue title, as that is a commonly known term around the K8S community.

[youngnick]

It is mutual TLS in the sense that the client and the server perform mutual TLS, but 'mTLS' also has some connotations about speed and automation of certificate rotation that we probably don't want to bring in here.

I say this because when we had an issue to add 'mTLS' to Contour, people started asking straight away if Contour was now a service mesh. Which it is not. But the term is overloaded.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[hbagdi]

/remove-lifecycle rotten

#268 should close this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[hbagdi]

/remove-lifecycle stale
/lifecycle frozen

[alanchrt]

Adding my comments from Slack regarding some configuration options that would be nice:

either deny or don't deny requests from clients who don't present valid certs (useful when you want to show a "logged out state" based on application logic in your upstream)

an option to pass through the subject/sans from the certificate to your upstream (the identity of the client), maybe in a header? not sure if there's a standard there. this could probably be the default.. not sure if it would matter if you still passed through the client identity and the upstream didn't care.

this could let you configure things like "i'll take requests in my upstream even if they don't present a valid cert, but if they do present a valid cert, i want to know that the identity was alan so i can decide his access level. and if there's no cert, i want to show them logged out state."

The ingress-nginx annotations seem like a really good reference point for which configuration options might be nice, as well as the names and contents of headers to pass back to the upstream:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-certificate-authentication

I think this one largely gets at some of the use cases I was describing above:

nginx.ingress.kubernetes.io/auth-tls-verify-client: optional_no_ca

[youngnick]

It's also important to remember that, if the following are true:

• you have a single proxy routing multiple SNIs
• the multiple SNI routes can have different security profiles (as in, some can have client certificates and some can't)
• You are using a protocol that has a higher-level routing construct (like HTTP's Host header)

Then you can connect to insecure.foo.com without a client cert, and set your Host header to secure.foo.com, and end up at the secure site, with no client cert.

This is roughly the same thing as domain fronting.

[alanchrt]

I wonder if sane defaults plus an "I know I'm doing a dangerous thing" option could help with domain fronting.

That is, if your listener is configured in Terminate mode, client certificate authentication can be configured with no special options. In addition to verifying the certificate, requests by default won't be routed upstream unless the SNI host matches the HTTP Host.

But, if you're in Passthrough mode, client authentication is only allowed with disableHostVerify: true. The docs can indicate that any upstream routers will need to be sure not to route based on Host with the expectation that it is the actually hostname that the client authenticated with. The disableHostVerify option could also be available for Terminate mode if there was a use case that needed it.

[mmalone]

@alanchrt you're talking about two modes: one that enforces that Host matches SNI and the other that doesn't, right? We're not terminating TLS vs. passing it through for a backend to terminate? If that's correct, I like the idea but would probably look for different terminology ;).

I'm not convinced the non-enforcing mode needs to exist at all. What's the use case for upstreams actually needing to route based on Host header? I'd consider setting the Host header for the upstream request to force it to match SNI. The host header is ignored all the time (e.g., most non-proxy web applications will ignore the host header). In a case like this, I can't think of a way it would ever break non-malicious clients.

I don't have as much experience with http/2 so I have no comment on whether this might affect those clients.

[howardjohn]

I don't have as much experience with http/2 so I have no comment on whether this might affect those clients.

I think per the HTTP2 spec servers are expected to return a 421 here in some scenarios - this allows the browser to automatically retry when its optimizations cause it to not match
envoyproxy/envoy#6767 and istio/istio#13589 have a lot of info here

[alanchrt]

@mmalone Actually, I wasn't trying to suggest Terminate and Passthrough as names for host verification, but rather trying to describe some possible default behaviors for those (existing) TLS mode options, which are defined in the TLS config.

Enforcement of SNI host and HTTP Host match wouldn't be possible in a Passthrough scenario, which is why I was suggesting you'd have to explicitly disable the enforcement to be able to enable client auth. Then, people know what they're getting themselves into.

As an example of an upstream routing based on Host, imagine it wasn't even a router usptream, but rather a monolith web app that's multi-tenant. It could serve customer-a.mycompany.com and customer-b.mycompany.com.

This is totally contrived, but imagine they decided they wanted to have cient cert auth on Gateway to prevent Customer A from even being able to load Customer B's app at all. They already have all the wiring to terminate TLS and manage certs and keys for their upstream app (for the customer domains), so they don't want to mess with that. They just verify the client cert, then passthrough TLS traffic to the app upstream. But, the app uses the HTTP Host to determine which customer's app to display.

Again, contrived, but I could see scenarios where someone might want it. In the above scenario, they could set disableHostVerify: true, and they could check the SNI host (passed in a header) to make sure it matches the HTTP Host in the upstream app itself.

[shaneutt]

@arkodg wanted to check in on this one?

[arkodg]

thanks for the reminder @shaneutt, still plan on working on it

[shaneutt]

Sounds good, let us know if there's ways we can help facilitate 🖖

[arkodg]

/reopen

[k8s-ci-robot]

@arkodg: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[robscott]

/kind gep

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[youngnick]

/remove-lifecycle stale
/lifecycle frozen

Since the GEP is still in motion.

[howardjohn]

@arkodg are there conformance tests planned?

[arkodg]

thanks for the reminder, let me create a issue to track it

[shaneutt]

Doesn't seem like we've had motion on this recently?

/remove-lifecycle frozen

@arkodg just wanted to check in, did you have any thoughts or a summary update for this one?

[arkodg]

next steps here would be

1. have some implementations, implement this
2. build out conformance tests
3. make sure tests pass for at least 3 implementations
4. move to standard

we're still on 1., afaik, no one has implemented this yet

[shaneutt]

Arko if I'm understanding correctly you're actively/planning-on moving this one forward:

/priority important-longterm

Can you please help us to better understand a rough idea about what you imagine you'll be able to deliver and roughly when you'd like to try for that? Not looking for a promise just trying to get a feel for your capacity and what you're thinking, so we can look at organizing this along the upcoming milestones.