Catch Segmentation Fault using UpstreamStartTls in network filter

[ayyatsenko]

Hello, all!
I'm trying to use UpstreamStartTls in my network filter and I catch a segfault ( when switch to tls ) :

[2021-06-24 12:48:50.560][12412][critical][backtrace] [./source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[2021-06-24 12:48:50.560][12412][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2021-06-24 12:48:50.560][12412][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 1aa65ac/1.19.0-dev/Modified/DEBUG/BoringSSL
[2021-06-24 12:48:50.627][12412][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x5630ddd4fba9]
[2021-06-24 12:48:50.627][12412][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7f6e517fd980]
[2021-06-24 12:48:50.675][12412][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::ConnectionPool::ActiveClient::onEvent() [0x5630dcf945b8]
[2021-06-24 12:48:50.723][12412][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::Tcp::ActiveTcpClient::onEvent() [0x5630dcf6eee6]
[2021-06-24 12:48:50.770][12412][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Network::ConnectionImplBase::raiseConnectionEvent() [0x5630dda01d97]
[2021-06-24 12:48:50.817][12412][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Network::ConnectionImpl::raiseEvent() [0x5630dd9eb8ad]
[2021-06-24 12:48:50.864][12412][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Extensions::TransportSockets::Tls::SslSocket::onSuccess() [0x5630dda1ba80]
[2021-06-24 12:48:50.911][12412][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Extensions::TransportSockets::Tls::SslHandshakerImpl::doHandshake() [0x5630ddae7dc1]
[2021-06-24 12:48:50.958][12412][critical][backtrace] [./source/server/backtrace.h:96] #8: Envoy::Extensions::TransportSockets::Tls::SslSocket::doHandshake() [0x5630dda1a445]
[2021-06-24 12:48:51.005][12412][critical][backtrace] [./source/server/backtrace.h:96] #9: Envoy::Extensions::TransportSockets::Tls::SslSocket::doWrite() [0x5630dda1bfbf]
[2021-06-24 12:48:51.054][12412][critical][backtrace] [./source/server/backtrace.h:96] #10: Envoy::Extensions::TransportSockets::StartTls::StartTlsSocket::doWrite() [0x5630da571266]
[2021-06-24 12:48:51.103][12412][critical][backtrace] [./source/server/backtrace.h:96] #11: Envoy::Network::ConnectionImpl::onWriteReady() [0x5630dd9f552e]
[2021-06-24 12:48:51.157][12412][critical][backtrace] [./source/server/backtrace.h:96] #12: Envoy::Network::ConnectionImpl::onFileEvent() [0x5630dd9f3cd3]
[2021-06-24 12:48:51.204][12412][critical][backtrace] [./source/server/backtrace.h:96] #13: Envoy::Network::ConnectionImpl::ConnectionImpl()::$_6::operator()() [0x5630dd9fd54e]
[2021-06-24 12:48:51.253][12412][critical][backtrace] [./source/server/backtrace.h:96] #14: std::_Function_handler<>::_M_invoke() [0x5630dd9fd411]
[2021-06-24 12:48:51.304][12412][critical][backtrace] [./source/server/backtrace.h:96] #15: std::function<>::operator()() [0x5630dc9f3604]
[2021-06-24 12:48:51.350][12412][critical][backtrace] [./source/server/backtrace.h:96] #16: Envoy::Event::DispatcherImpl::createFileEvent()::$_5::operator()() [0x5630dc9eacbf]
[2021-06-24 12:48:51.397][12412][critical][backtrace] [./source/server/backtrace.h:96] #17: std::_Function_handler<>::_M_invoke() [0x5630dc9eab01]
[2021-06-24 12:48:51.397][12412][critical][backtrace] [./source/server/backtrace.h:96] #18: std::function<>::operator()() [0x5630dc9f3604]
[2021-06-24 12:48:51.445][12412][critical][backtrace] [./source/server/backtrace.h:96] #19: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x5630dc9fbb63]
[2021-06-24 12:48:51.491][12412][critical][backtrace] [./source/server/backtrace.h:96] #20: Envoy::Event::FileEventImpl::assignEvents()::$_1::operator()() [0x5630dc9fc0f4]
[2021-06-24 12:48:51.553][12412][critical][backtrace] [./source/server/backtrace.h:96] #21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x5630dc9fbc89]
[2021-06-24 12:48:51.604][12412][critical][backtrace] [./source/server/backtrace.h:96] #22: event_persist_closure [0x5630ddd2d76b]
[2021-06-24 12:48:51.653][12412][critical][backtrace] [./source/server/backtrace.h:96] #23: event_process_active_single_queue [0x5630ddd2cde2]
[2021-06-24 12:48:51.702][12412][critical][backtrace] [./source/server/backtrace.h:96] #24: event_process_active [0x5630ddd27728]
[2021-06-24 12:48:51.761][12412][critical][backtrace] [./source/server/backtrace.h:96] #25: event_base_loop [0x5630ddd2662c]
[2021-06-24 12:48:51.814][12412][critical][backtrace] [./source/server/backtrace.h:96] #26: Envoy::Event::LibeventScheduler::run() [0x5630ddce8cff]
[2021-06-24 12:48:51.867][12412][critical][backtrace] [./source/server/backtrace.h:96] #27: Envoy::Event::DispatcherImpl::run() [0x5630dc9e5962]
[2021-06-24 12:48:51.914][12412][critical][backtrace] [./source/server/backtrace.h:96] #28: Envoy::Server::WorkerImpl::threadRoutine() [0x5630dc9d02bb]
[2021-06-24 12:48:51.960][12412][critical][backtrace] [./source/server/backtrace.h:96] #29: Envoy::Server::WorkerImpl::start()::$_5::operator()() [0x5630dc9d20e0]
[2021-06-24 12:48:52.010][12412][critical][backtrace] [./source/server/backtrace.h:96] #30: std::_Function_handler<>::_M_invoke() [0x5630dc9d1efd]
[2021-06-24 12:48:52.057][12412][critical][backtrace] [./source/server/backtrace.h:96] #31: std::function<>::operator()() [0x5630d8dbfdf5]
[2021-06-24 12:48:52.106][12412][critical][backtrace] [./source/server/backtrace.h:96] #32: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x5630de19951d]
[2021-06-24 12:48:52.157][12412][critical][backtrace] [./source/server/backtrace.h:96] #33: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x5630de1994f5]
[2021-06-24 12:48:52.157][12412][critical][backtrace] [./source/server/backtrace.h:96] #34: start_thread [0x7f6e517f26db]

It seems that the second Network::ConnectionEvent::Connected event is generated when handshake complete and it causes a segfault on source/common/conn_pool/conn_pool_base.cc:457 , before that the first event was generated when plaintext connection was established.
Is the UpstreamStartTls functionality ready ( It's not mentioned in current.rst ) ? It's a bug or my mistake?
Thanks in advance.

Base issue #15443

[lizan]

What's your config? You need a network filter that support upstream STARTTLS to use with the transport socket.

cc @bryce-anderson

[ayyatsenko]

I used the following config:

stats_config:
  stats_matcher:
    reject_all: true
admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 15001
static_resources:
  listeners:     
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 5432
    filter_chains:
      filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: postgres
          cluster: postgres_ssl
  clusters:
  - name: postgres_ssl
    connect_timeout: 0.25s
    type: STATIC
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: postgres
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 15432
    filters:
    - name: envoy.filters.network.postgres_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.postgres_proxy.v3alpha.PostgresProxy
        stat_prefix: postgres
    transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.starttls.v3.UpstreamStartTlsConfig
          cleartext_socket_config: {}
          tls_socket_config:
            common_tls_context:
              tls_certificates:
                certificate_chain:
                  filename: client.crt
                private_key:
                  filename: client.key
              validation_context:
                trusted_ca:
                  filename: ca.crt

PostgresProxy is a my modification of filter of the same name from default codebase. I registered it as the upstream network filter using macros :

REGISTER_FACTORY(PostgresConfigFactoryMy, Server::Configuration::NamedUpstreamNetworkFilterConfigFactory);

and I modifyed onData() and onWrite() calls to work with UpstreamStartTls transport socket.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.