tlsContext interferes with filterChainMatch and TLS inspector

[vadimeisenbergibm]

The scenario: an application sends TLS traffic to https://edition.cnn.com, with SNI. The traffic is intercepted by a sidecar Envoy, which sends the traffic to a gateway Envoy, which, in turn, sends the traffic to edition.cnn.com. The gateway Envoy reports the SNI.

application (SNI = edition.cnn.com) -> sidecar proxy (SNI = edition.cnn.com)-> gateway proxy (SNI = edition.cnn.com) -> edition.cnn.com

Without mTLS between the proxies, the configuration is straightforward: the sidecar proxy has a filter chain with a filter chain match by server_names: edition.cnn.com, a TCP proxy filter that forwards the traffic to the gateway. The gateway has a filter chain with a filter chain match by server_names: edition.cnn.com, with a TCP proxy filter that forwards the traffic to edition.cnn.com. The gateway has also a filter that calls read_callbacks_->connection().requestedServerName() and prints the SNI equal to edition.cnn.com. Everything works as expected.

Now suppose we want to apply mTLS between the sidecar proxy and the gateway proxy (so we will have an mTLS "tunnel" between the proxies, with the original TLS with the original SNI inside the tunnel). The mTLS could be required for verifying the identity of the application (by verifying the identity of the sidecar proxy).

So the picture becomes as follows:
application (SNI = edition.cnn.com) -> sidecar proxy (SNI = gateway.cluster.local)-> gateway proxy (SNI = edition.cnn.com) -> edition.cnn.com

Note that now mTLS interferes with the filterChainMatch and SNI reporting of the gateway proxy. The gateway proxy will have to specify server_names: gateway.cluster.local and it will report SNI as gateway.cluster.local.

As far as I understand, mTLS between the proxies is transparent with regard to the other protocols, e.g. HTTP. The desired operation would be: Envoy will perform TLS termination to terminate mTLS between the proxies, and then will perform the SNI matching by the original SNI and reporting of the original SNI (in this case edition.cnn.com).

[vadimeisenbergibm]

cc @mattklein123 @PiotrSikora @rshriram

[mattklein123]

@PiotrSikora will know the details, but I'm not sure this is possible without using/inventing some type of TLS extension to forward the original SNI.

[PiotrSikora]

Traffic inspection / filter chain matching works on the traffic it sees (i.e. outer layer of your double-encrypted tunnel, i.e. mTLS), but you're trying to match on the inner layer, which is encrypted and thus invisible to traffic inspector. At the stage of matching, we don't even know if the filters will want to (or even can) decrypt the traffic, so there is no access to values from inner layer.

But, is there any reason why you can't keep SNI as edition.cnn.com for the outer layer?

[vadimeisenbergibm]

@PiotrSikora I am confused by the fact that HTTP routing and reporting works successfully on the inner layer (HTTP inside the mTLS tunnel). Envoy is able to perform TLS termination, and after that to perform routing by the decrypted Host header and to report the decrypted headers. Why the same cannot be performed for TLS inside mTLS tunnel? How TLS inside mTLS tunnel is different from HTTP inside mTLS tunnel, from Envoy's point of view?

But, is there any reason why you can't keep SNI as edition.cnn.com for the outer layer?

Because the sidecar proxy may not know the SNI. Consider a case it is configured to forward the traffic of *.cnn.com, or *.com or * to the gateway. The sidecar proxy is configured with the corresponding filterChainMatch and it manages to forward the traffic to the gateway proxy. However, the sidecar proxy does not know the original SNI, so it cannot send the original SNI to the gateway proxy as part of the mTLS tunnel.

[elevran]

@PiotrSikora it is true that the payload is encrypted and thus invisible to the traffic inspector. However, the SNI extension part of the client handshake is sent in clear text and is therefore available.
The external (mTLS) tunnel carries the destination service name in the SNI (e.g., egress-gw.cluster.local). I don't know if we can manipulate SNI on the sender side to have other values. The edition.cnn.com is part of the payload.

It seems that the goal is to extract it from the first SSL record in the mTLS connection payload (i.e., a traffic inspection filter) and set it as a request or connection attribute. Once that value is extracted the filter's work is done. The attribute ("inner_server_name") can be sent to the mixer for checks and possibly used to determine the upstream cluster for connection routing.

[PiotrSikora]

@PiotrSikora I am confused by the fact that HTTP routing and reporting works successfully on the inner layer (HTTP inside the mTLS tunnel). Envoy is able to perform TLS termination, and after that to perform routing by the decrypted Host header and to report the decrypted headers. Why the same cannot be performed for TLS inside mTLS tunnel? How TLS inside mTLS tunnel is different from HTTP inside mTLS tunnel, from Envoy's point of view?

Filter chain selection happens before TLS termination, without access to the data inside the mTLS tunnel.

HTTP routing happens inside HTTP connection manager, after the traffic is already decrypted.

You could create TLS filter that does routing / cluster selection based on the TLS ClientHello message from the inner layer, after the traffic is already decrypted, but that would need to happen after filter chain is selected, because you need to know which certificate to use to complete the TLS handshake and decrypt the traffic. This is definitely doable, but requires some work.

@PiotrSikora it is true that the payload is encrypted and thus invisible to the traffic inspector. However, the SNI extension part of the client handshake is sent in clear text and is therefore available.

SNI in the outer layer (i.e. mTLS tunnel), yes, but not the SNI in the inner layer (i.e. workload-to-workload TLS inside mTLS tunnel), which this issue is about.

It seems that the goal is to extract it from the first SSL record in the mTLS connection payload (i.e., a traffic inspection filter) and set it as a request or connection attribute. Once that value is extracted the filter's work is done. The attribute ("inner_server_name") can be sent to the mixer for checks and possibly used to determine the upstream cluster for connection routing.

Note that filter chain selection, cluster routing and information extraction for the logging purposes are all very different things, and what you're talking about is unrelated to filter chain selection.

[vadimeisenbergibm]

You could create TLS filter that does routing / cluster selection based on the TLS ClientHello message from the inner layer, after the traffic is already decrypted, but that would need to happen after filter chain is selected, because you need to know which certificate to use to complete the TLS handshake and decrypt the traffic. This is definitely doable, but requires some work.

OK, let's focus on this. I think what happened here is that filter chain selection for the mTLS tunnel is mixed with TLS routing of the original traffic. I think we have two options to solve it:

Option 1: Is it possible to perform filter chain selection for mTLS tunnel, then mTLS termination, and then to perform the second round of filter chain selection based on the original traffic? This will work the same with the filter chain matches with or without mTLS tunnel. You create filter chain matches for both server_names: gateway.cluster.local and server_names: edition.cnn.com. The first one performs TLS termination, the second one does not perform it. If the mTLS is used with SNI gateway.cluster.local, the first filter chain is selected based on the mTLS SNI, TLS termination is performed, then in the second round of filter chain selection, the second filter chain is selected based on the inner SNI, so the routing based on the original SNI is performed and the original SNI is reported. If mTLS is not used, the second filter chain is selected already in the first selection round, no TLS termination is performed, the routing and reporting are performed based on the original SNI.

Option 2: to create envoy.tls_proxy filter based on the TLS ClientHello message from the inner layer. It will perform routing based on the inner SNI and will report the inner SNI.

Which option would be more clear for the users and more consistent? Which option would be easier to implement?

[rshriram]

I think you should create a new filter in both http and tcp. The filter needs to read the incoming server_name field and stick this into the RequestInfo object. Then in the cluster, you need to check if the user has explicitly set the SNI field in the cluster. if yes, you use the user provided value. If no, you look at the RequestInfo object to see if any filter has set the SNI value, and use it.

This will allow you to propagate the incoming SNI value from app as outbound sni value in the sidecar to gateway..

[vadimeisenbergibm]

@rshriram So you propose option 3: the sidecar proxy will extract the SNI and forward it as the SNI of the mTLS tunnel.

@PiotrSikora @mattklein123 So which option among the options in #4076 (comment), #4076 (comment) is better? Which one is easier to implement?

[elevran]

@vadimeisenbergibm based on my understanding the purpose of egress service is to protect against cases where the sending proxy has been compromised, meaning we don't want to trust the SNI it sent but rather extract the client SNI from the inner TLS (since that's what we'll use as the target).

@PiotrSikora:

SNI in the outer layer (i.e. mTLS tunnel), yes, but not the SNI in the inner layer (i.e. workload-to-workload TLS inside mTLS tunnel), which this issue is about.

I may not be correctly understanding your comment. The inner SNI is encrypted using the mTLS. I was suggesting to extract the inner SNI in the receiving proxy, after it decrypts the mTLS connection. At that point the inner SNI is clear text again and can be inspected.

[vadimeisenbergibm]

@elevran Right, the option 3 would not help us to prevent cheating (sending a fake SNI) by a compromised sidecar proxy.

[rshriram]

How is this improving security? is someone MITMing and changing the SNI alone? that can't happen in a k8s network as there is no L2. Is someone masquerading as the source pod? That can't happen if you do RBAC at the egress gateway that validates against the source pod's SAN.

Doing the double/triple mtls inspection stuff is something that cannot be easily programmed through Pilot as it treats ingress and egress gateways in the same way. This will then end up manifesting as a multitude of user configurations (classify gateway type or define a special TLS mode, etc. etc.). The gateway has no idea whether the payload inside a mTLS connection is already encrypted or not or its protocol etc. So, simply reading the SNI string verbatim is a security hole in itself as the protocol is not typed [we don't know what it is in advance].

[vadimeisenbergibm]

The scenario is as follows. Suppose you have the following policy in Istio:

1. PodA may access *.foxnews.com
2. PodB may access *.cnn.com

An attacker is able to break into the PodA and wants to access edition.cnn.com, which is forbidden by the example policy.

Let's assume that the attacker is able to kill/bypass the sidecar proxy, steal its certificates (stealing certificates is mentioned here - https://istio.io/docs/concepts/security/#secure-naming) and to run its own, malicious version of the sidecar proxy with the certificates of the PodA. Now the attacker will issue a request to edition.cnn.com (with the original SNI equal edition.cnn.com). The malicious sidecar proxy will open mTLS connection to the gateway, with the outer SNI equal www.foxnews.com. The gateway's proxy will send to the Istio Mixer the identity of the pod PodA and the outer SNI www.foxnews.com. The Mixer check will succeed. Then the gateway will perform mTLS termination, and will forward the request by the original SNI to edition.cnn.com (see istio/istio.io#1984).

To prevent such attack we need to ensure that the gateway sends the original, inner SNI to the mixer, not the outer one. The security is enforced in the following way - you forward the traffic to the same server that you report to the mixer. This way the traffic will indeed be forwarded to the server that was reported to and allowed by the Mixer.

[elevran]

@rshriram do you think it could, perhaps, be cleaner if we had a clear separation between tunnel setup phase (where envoys can exchange any data needed - e.g., to support mTLS, forward original client and server L3 information, etc.) and payload exchange after the tunnel is fully set up?

This would work well for TCP proxying where the envoy-envoy connection is "per session", and may be skipped for when multiplexing multiple connections unto the same envoy connection. SNI routing is more like TCP proxying than HTTP/H2.