Certificates for double proxy (mTLS)

[avinashupadhya99]

Title: Certificates for double proxy (mTLS)

Description:

The envoy docs has an example of running double proxy using Envoy for postgres. However, the docs does not cover which certificate is to be used where. The guide generates 2 certificates with the same key. The docker-compose file in the repo refers to the certificates in a different name, making it not clear on which certificate to use on which machine(container).

A section to clarify that in the guide would be helpful.

I am following the same guide and have generated certificates for AWS EC2's internal DNS name. However I get the error TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED on the client side and TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN on the server side.

Server Envoy TLS config section -

transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          require_client_certificate: true
          common_tls_context:
            validation_context:
              trusted_ca:
                filename: /home/ubuntu/certs/ca.crt
            tls_certificates:
              - certificate_chain: { filename: "/home/ubuntu/certs/ip-10-0-101-153.ap-south-1.compute.internal.crt" } # ip-10-0-101-153 is the IP of the server machine
                private_key: { filename: "/home/ubuntu/certs/ap-south-1.compute.internal.key" }

Client Envoy TLS config section -

transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: /home/ubuntu/certs/ip-10-0-101-74.ap-south-1.compute.internal.crt # ip-10-0-101-74 is the IP of the client machine
              private_key:
                filename: /home/ubuntu/certs/ap-south-1.compute.internal.key
                #tls_params:
                #tls_minimum_protocol_version: TLSv1_2
                #tls_maximum_protocol_version: TLSv1_2
            validation_context:
              match_subject_alt_names:
                - exact: ip-10-0-101-74.ap-south-1.compute.amazonaws.com
              trusted_ca:
                filename: /home/ubuntu/certs/ca.crt

[optional Relevant Links:]
https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/double-proxy

[alyssawilk]

cc @phlax @ggreenway

[phlax]

re the docs - i would be more than happy to review a PR to clarify how the certs are used

to give some context - the Dockerfile where the certs are loaded is here https://github.com/envoyproxy/envoy/blob/main/examples/shared/envoy/Dockerfile#L49

it may be helpful to add some comment in the compose file about looking there

for how the certs are used you need to look in the envoy frontend/backend yaml files

the sandbox is pretty focused on explaining how to setup and use the certs so would be good if we can make this clearer

[avinashupadhya99]

I followed the guide and setup the certs accordingly. It looks like Envoy is not terminating the TLS connection and is forwarding it to the actual backend, resulting in

[2023-03-22 13:50:21.790][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:748] [C9] TCP:onUpstreamEvent(), requestedServerName: 
[2023-03-22 13:50:21.792][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:233] [C9] remote address:10.0.101.118:37814,TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2023-03-22 13:50:21.792][15][debug][connection] [source/common/network/connection_impl.cc:250] [C9] closing socket: 0
[2023-03-22 13:50:21.792][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:233] [C9] remote address:10.0.101.118:37814,TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2023-03-22 13:50:21.792][15][debug][connection] [source/common/network/connection_impl.cc:139] [C10] closing data_to_write=0 type=0
[2023-03-22 13:50:21.792][15][debug][connection] [source/common/network/connection_impl.cc:250] [C10] closing socket: 1

Here's the TLS part of my envoy config for the server -

transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          require_client_certificate: true
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: /certs/servercert.pem
              private_key:
                filename: /certs/serverkey.pem
            tls_params:
              tls_minimum_protocol_version: TLSv1_2
              tls_maximum_protocol_version: TLSv1_3
            validation_context:
              match_typed_subject_alt_names:
              - san_type: DNS
                matcher:
                  exact: ip-10-0-101-118.ap-south-1.compute.internal
              trusted_ca:
                filename: /certs/cacert.pem

[phlax]

hmm - you may be right - this needs to set TLS termination in the "frontend" envoy

that would break using postgres i think (not 100% sure) so it would also require changing what was used as a backend

definitely open to a PR to update this - otherwise i will look further when i have time