CONNECT-UDP example configuration error

[Bfarkiani]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: CONNECT-UDP example configuration error
Description:
I am trying to test the configuration provided in https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades#connect-udp-support. I am also using quiche masque_client https://github.com/google/quiche/tree/main/quiche/quic/masque. Also I am using envoy:dev latest container. The client side proxy configuration is as below:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/server.crt
                private_key:
                  filename: /etc/envoy/server.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog                
          
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: cluster_0
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.170.128
                port_value: 10000
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: /etc/envoy/server.crt
              private_key:
                filename: /etc/envoy/server.key

Server side proxy (192.168.170.128) that terminates connect-udp and forward request to google is as follows:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/server.crt
                private_key:
                  filename: /etc/envoy/server.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog             
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: service_google
                  upgrade_configs:
                  - upgrade_type: CONNECT-UDP
                    connect_config:
                      {}
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: service_google
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_google
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: www.google.com
                port_value: 443

And the command for masque client is
./masque_client --disable_certificate_verification=true 127.0.0.1:10000 https://www.google.com

It fails. The output of client envoy is

[2024-04-25 03:31:08.165][1][info][main] [source/server/server.cc:971] starting main dispatch loop
[2024-04-25 03:32:38.204][15][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:974] No hostname indicated in SNI
[2024-04-25 03:32:38.215][15][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname:
58755145347776:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:
[2024-04-25T03:32:38.211Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 503 UF 1311 236 4 - "-" "-" "c3a36610-997b-40ab-a2a4-a7d46f698f00" "www.google.com:443" "192.168.170.128:10000"

The server envoy output is

[2024-04-25 03:32:38.210][20][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:974] No hostname indicated in SNI

The client and server envoy proxies both use the same self-signed certificate, and the configuration is the same as provided in the website.
If I enable auto_sni in client side :

  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        upstream_http_protocol_options:
          auto_sni: true
          
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.170.128
                port_value: 10000
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          common_tls_context:
            validation_context:
              trust_chain_verification: ACCEPT_UNTRUSTED

I still get the following error

[2024-04-25 03:56:25.547][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:974] No hostname indicated in SNI
[2024-04-25 03:56:25.563][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: www.google.com
22063240393008:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:
[2024-04-25T03:56:25.554Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 503 UF 1311 236 9 - "-" "-" "3e1dbc32-dba5-48fa-8069-2065bfecefec" "www.google.com:443" "192.168.170.128:10000"

I also changed server proxy to the follow and nothing changed:

  clusters:
  - name: service_google
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        upstream_http_protocol_options:
          auto_sni: true
        explicit_http_config: 
          http2_protocol_options: {}
    
    load_assignment:
      cluster_name: service_google
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: www.google.com
                port_value: 443

Thank you for checking this.

[wbpcode]

cc @DavidSchinazi cc @RyanTheOptimist

[DavidSchinazi]

@jeongseokson could you take a look at this one please?

[jeongseokson]

Hi @Bfarkiani, thank you for reporting the issue. Does sending a CONNECT-UDP request directly to the server-proxy work?

I will try to test the forwarding and terminating proxy two-hop setup myself and see if there's any issue in the current example configs.

[Bfarkiani]

Hi @jeongseokson
Thank you for your attention. I sent the request to the server. The server uses the same configuration as above (the same as the envoy docs). Envoy server prints the following message

[2024-04-29T20:15:37.750Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 200 - 3875 68976 383 - "-" "-" "c70459b6-454d-45e1-9629-55f97aae6c7f" "www.google.com:443" "142.250.190.36:443"

So, it seems that sending directly to the server works fine.

[jeongseokson]

Thanks for the update. I will test the two proxy setup myself when time permits to see if I can reproduce the issue.

[Bfarkiani]

Dear @jeongseokson
Have you had any chance to review this matter?

[jeongseokson]

Hi @Bfarkiani, I started looking into this issue last week. I will follow up soon after gathering more information reproducing the issue myself.

[jeongseokson]

I reproduced the setup and tried the same. The issue is due to certificate verification on the forwarding proxy side when establishing a QUIC connection with the upstream terminating proxy. This happens regardless of CONNECT-UDP when a self-signed certificate is used in the terminating proxy.

#17700 is a relevant discussion around this issue. I am not sure if we provided a way to disable upstream certificate checks when Envoy is working as a QUIC client. @danzh2010 Could you provide some insight on this?

[danzh2010]

There is no way to bypass Leaf certificate doesn't match hostname: www.google.com. You can use a self-signed cert on the proxy server to both downstream and upstream. But your masque client should be configured to expect your proxy server's own cert instead of a google.com cert. auto_sni will pick the host name from your request url, so it will be google.com. If you explicit config upstream_tls_context.sni in your masque client config to the SAN in your own cert, you should be able to bypass this error.

[jeongseokson]

Thanks for the reply Dan. But the masque_client has --disable_certificate_verification=true, and the Leaf certificate doesn't match hostname: error is from the Envoy (forwading proxy), not from the masque_client. Is there a way to bypass it from the Envoy when it works as a QUIC client (as a forwarding proxy)?

[danzh2010]

Thanks for the reply Dan. But the masque_client has --disable_certificate_verification=true, and the Leaf certificate doesn't match hostname: error is from the Envoy (forwading proxy), not from the masque_client.

Sorry I'm confused about this comment .

Is below log observed on the client envoy? If so it means it failed handshake with masque client as a server (given tls_server_handshaker).

[2024-04-25 03:56:25.547][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:974] No hostname indicated in SNI
[2024-04-25 03:56:25.563][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: www.google.com
22063240393008:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:
[2024-04-25T03:56:25.554Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 503 UF 1311 236 9 - "-" "-" "3e1dbc32-dba5-48fa-8069-2065bfecefec" "www.google.com:443" "192.168.170.128:10000"

And this is downstream config. What confuses me is that you said auto_sni which is in upstream config changes the logging. From [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: 58755145347776:error:1000007d:SSL to [2024-04-25 03:56:25.563][17][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_handshaker.cc:256] Cert chain verification failed: Leaf certificate doesn't match hostname: www.google.com 22063240393008:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:external/boringssl/src/ssl/handshake.cc:393:

Is there a way to bypass it from the Envoy when it works as a QUIC client (as a forwarding proxy)?

No other than via the sni config knob.

[jeongseokson]

@Bfarkiani Do you have anything to add regarding the auto_sni? I think auto_sni doesn't make sense in this setup because the target hostname (google.com) is not the hostname of the terminating proxy.

@danzh2010 My understanding is that it's the TLS handshake failure between the forwarding proxy (client Envoy) and the terminating proxy (server Envoy), not between the masque client and the forwarding proxy. Is there a reason you believe it's still a handshake failure between the masque_client and the forwarding envoy?

To try to solve the issue, I set the sni value in UpstreamTlsContext config of the forwarding proxy to match the self signed certificate's CN of the terminating proxy, but I still got the same Leaf certificate doesn't match hostname: error from the forwarding proxy side.

[jeongseokson]

I figured out that sni should match SAN, not CA of the certificate to pass the leaf hostname check test of the Envoy. The test certificates provided have lyft.com as a SAN. To make the TLS handshake between the proxies work, you should set sni: "lyft.com" under upstream_tls_context:.

Please try this @Bfarkiani. If you created your own self-signed certificate, you should specify the SAN there to match the one in the config. I checked other http3 example configs of Envoy and they have sni field set to avoid this issue I guess. I will update the config soon.

Unfortunately, the CONNECT-UDP request still doesn't go through even after the TLS handshake is successfully done. This one looks like an actual issue related to the HTTP Datagram implementation. I will take a look into it as well, but please try to do it yourself just to make sure we encounter the same issue @Bfarkiani.

[Bfarkiani]

Dear @jeongseokson
Sorry for my late response. I tried the following configurations after applying your change and also creating a self-signed certificate with SAN of example.com:

client:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/example.com.crt
                private_key:
                  filename: /etc/envoy/example.com.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog                       
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: cluster_0
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        upstream_http_protocol_options:
          auto_sni: true   ####<---- I added this but no help
          
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.170.128
                port_value: 10000
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          # SAN of the certs/servercert.pem used for local testing. Update this if using a different certificate.
          sni: example.com

Server:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/example.com.crt
                private_key:
                  filename: /etc/envoy/example.com.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog                       
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: service_google
                  upgrade_configs:
                  - upgrade_type: CONNECT-UDP
                    connect_config:
                      {}
          http_filters:
          - name: envoy.filters.http.dynamic_forward_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                name: dynamic_forward_proxy_cache_config
                dns_lookup_family: V4_ONLY          
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: service_google
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_google
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: google.com
                port_value: 443
                 

My certificate details:

The command I am using for masque client is

./masque_client --disable_certificate_verification=true 127.0.0.1:10000 https://www.google.com

What server envoy prints is

And what happens for packets:

Seems that there is a connection. Although masque client shows nothing.

As you see also server envoy prints "No hostname indicated in SNI" even with auto_sni.

[jeongseokson]

@Bfarkiani, I think you shouldn't use auto_sni in this case. It sets the SNI value same as the host/authority, which isn't correct in this forwarding proxy scenario.
The only change I made was to add sni in the upstream tls context of the forwading proxy. I added it to match the SAN in the certificate of the terminating proxy, as you can see in the PR: https://github.com/envoyproxy/envoy/pull/34359/files

[Bfarkiani]

@jeongseokson
I changed my client to chromium masque https://www.chromium.org/quic/playing-with-quic/ and tried the same configuration without auto_sni to connect to a local website. Still Failure with this error at client.

[jeongseokson]

@Bfarkiani I think we've reached the same actual error in the existing forwarding proxy mode I mentioned in a previous comment. Fixing this would probably require some modification to the existing HTTP Datagram handler unfortunately. I will create a separate issue for tracking this as this isn't a configuration issue.

[Bfarkiani]

@jeongseokson
Thank you.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[DavidSchinazi]

@RyanTheOptimist could you reopen this issue and mark no-stale-bot please? I don't think it's fully resolved yet

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[jeongseokson]

Just want to clarify that the issue raised in this post (as the title says) is about the incorrect example config (No SNI specified), which has been resolved. We uncovered the other issue in the CONNECT-UDP forwarding mode in the process, and I created #34836 for that. I think we can mark this issue resolved and use #34836 to track the issue separately, which I am currently working on.

[DavidSchinazi]

That makes sense, thanks for the details. OK let's close this issue once #34836 is reopened