envoy bug failure: pending_streams_.size() <= connecting_stream_capacity

[zhishi]

Title: after upgraded envoy from 1.31.5 to 1.33.3, envoy start showing envoy bug failure: pending_streams_.size() <= connecting_stream_capacity_

Description:
after upgraded envoy from 1.31.5 to 1.33.3, envoy start showing this error log:

"2025-05-20T15:48:02.839950000","39","conn_pool_base.cc:341","error","envoy bug failure: pending_streams_.size() <= connecting_stream_capacity_ || connecting_stream_capacity_ > old_capacity || (result == ConnectionResult::NoConnectionRateLimited || result == ConnectionResult::FailedToCreateConnection). Details: Failed to create expected connection: ConnPoolImplBase 0x278f3ec7e200, ready_clients_.size(): 0, busy_clients_.size(): 2, connecting_clients_.size(): 0, connecting_stream_capacity_: 0, num_active_streams_: 106, pending_streams_.size(): 49 per upstream preconnect ratio: 1","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.839990000","39","assert.h:38","error","stacktrace for envoy bug","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.839999000","39","assert.h:43","error","#0 Envoy::Http::HttpConnPoolImplBase::newStream() [0x5ec1984f9c86]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840004000","39","assert.h:43","error","#1 Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream() [0x5ec1986f86bd]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840009000","39","assert.h:43","error","#2 Envoy::Router::UpstreamRequest::acceptHeadersFromRouter() [0x5ec1987159c5]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840013000","39","assert.h:43","error","#3 Envoy::Router::Filter::decodeHeaders() [0x5ec1987004d1]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840017000","39","assert.h:43","error","#4 Envoy::Http::FilterManager::decodeHeaders() [0x5ec19872a6b9]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840022000","39","assert.h:43","error","#5 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x5ec19869ef7a]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840026000","39","assert.h:43","error","#6 Envoy::Http::Http2::ConnectionImpl::onHeaders() [0x5ec1986d1097]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840030000","39","assert.h:43","error","#7 Envoy::Http::Http2::ConnectionImpl::Http2Visitor::OnEndHeadersForStream() [0x5ec1986d5cd0]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840034000","39","assert.h:43","error","#8 http2::adapter::callbacks::OnFrameReceived() [0x5ec1989399d8]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840038000","39","assert.h:43","error","#9 session_after_header_block_received [0x5ec1989536ce]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840042000","39","assert.h:43","error","#10 nghttp2_session_mem_recv2 [0x5ec19895264a]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840046000","39","assert.h:43","error","#11 http2::adapter::NgHttp2Adapter::ProcessBytes() [0x5ec1989373f0]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840050000","39","assert.h:43","error","#12 Envoy::Http::Http2::ConnectionImpl::dispatch() [0x5ec1986cf07d]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840054000","39","assert.h:43","error","#13 Envoy::Http::Http2::ServerConnectionImpl::dispatch() [0x5ec1986d989b]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840058000","39","assert.h:43","error","#14 Envoy::Http::ConnectionManagerImpl::onData() [0x5ec1986988a7]","KubeApiProxyEnvoy"
"2025-05-20T15:48:02.840062000","39","assert.h:43","error","#15 Envoy::Network::FilterManagerImpl::onContinueReading() [0x5ec1988bf35e]","KubeApiProxyEnvoy"

the error log is: pending_streams_.size() <= connecting_stream_capacity_ || connecting_stream_capacity_ > old_capacity || (result == ConnectionResult::NoConnectionRateLimited || result == ConnectionResult::FailedToCreateConnection). Details: Failed to create expected connection: ConnPoolImplBase 0x278f3ec7e200, ready_clients_.size(): 0, busy_clients_.size(): 2, connecting_clients_.size(): 0, connecting_stream_capacity_: 0, num_active_streams_: 106, pending_streams_.size(): 49 per upstream preconnect ratio: 1"

before upgrading we never see such error log in envoy 1.31, I'm not sure if this is caused by this change .
and we have this config in our upstream cluster:

      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          use_downstream_protocol_config:
            http2_protocol_options:
              max_concurrent_streams: 100
              initial_stream_window_size: 65536
              initial_connection_window_size: 1048576
              connection_keepalive:
                interval: 30s
                timeout: 5s

Repro steps: since this happened in our production cluster with heavy traffic, I'm not sure if it's possible to reproduce on local environment.

[agrawroh]

cc @ggreenway Looks similar to #39238

[ggreenway]

It's likely that this is fixed by 8c9f3ce, which is the latest commit on branch releases/v1.33. Can you test with this commit and see if it still reproduces?

[zhishi]

we deployed the 1.33 with latest commits this week, looks the "envoy bug failure" is not showing anymore. however, there is still some UR errors: "503","upstream_reset_before_response_started{remote_refused_stream_reset}"

which we expected to be fixed by 1.33. I'll open a new bug for that issue. this one can be closed.