Expected behavior when HttpClientFactory in Azure App Service attempts to use reclaimed port?

[aarondel4]

I am using HttpClientFactory in a MVC .NET 6 Core app which runs in an Azure App Service. I have been seeing these errors intermittently and now almost all outbound requests fail.

I have been reading about some of the various behaviors in Azure on port handling, such as cooldown timers and idle timers for reclaiming ports. I've also seen anecdotes from developers who have engaged MS support around these issues and were told that ports can be reclaimed by Azure. Of course this could be related to SNAT exhaustion, but I've observed it in test environments where I'm the only user, so I assume sometimes it's an idle port being reclaimed(which is a documented behavior in Azure).

So in the context of pooled http clients(or handlers) within HttpClientFactory, what would be the expected behavior in that scenario where an allocated port is no longer valid? Is it coded to handle a scenario where attempting to reuse a pooled client handler may no longer have a valid socket and gracefully recreate the handler? I.e. when I call .CreateClient, my understand is the handler the client is associated with is retrieved from a pool of handlers. As far as I can tell, no validation is performed. The failure occurs only when attempting to make the request. It also does not seem to treat the request failure as an indicator to remove the pooled handler from the pool. Can someone confirm this is correct? If a SocketException occurs during a request, is it possible the handler can still be returned for future CreateClient requests? It seems like over time, I am accumulating a pool of invalid handlers in my HttpClientFactory. It appears the factory doesn't know or care about the individual request exceptions, and will happily keep the handler in its pool for future use.

"You're experiencing SNAT exhaustion" Possible aggravating factor in production, but as mentioned I've observed same behavior when I'm the only user and there's only a couple outbound requests every few minutes. In either case, the root of my question here is to do due diligence in making sure the pragmatic approach is correct in the context of an environment that can reclaim ports, before recommending additional spend on NAT gateway.

Code base that has worked flawlessly for years now fails frequently in Azure, and I don't want to jump straight to recommending a spend on infrastructure. At a glance, I don't think the recommended HttpClientFactory's pooling approach is very compatible with an environment that can reclaim ports and invalidate pooled handlers, mainly because as far as I can tell, it won't remove a handler whenever a request fails on SocketException. So it seems like it can potentially accumulated a pool of bad handlers. Is there anything I can do to adjust its configuration to handle this scenario more gracefully?

Example of the request error. Note, if/when these requests fail, it is always after exactly 20 seconds, which I've seen discussed as being expected in situations where the port is no longer valid.

 ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Validator.Services.VendorClient.VendorClient.Get[T](String urlPath) in D:\a\1\s\Validator.Services\VendorClient\VendorClient.cs:line 136

Client registration:

services.AddHttpClient("VendorHttpClient", c =>
{
    c.BaseAddress = new Uri(VendorConfig.VendorBaseUrl.TrimEnd('/') + "/");
    c.DefaultRequestHeaders.Add("User-Agent", "Validator");    
    var encodedApiKey = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{VendorConfig.VendorApiKey}"));
    c.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", encodedApiKey);
    // c.DefaultRequestHeaders.ConnectionClose = true; 
});

Usage:

    public class VendorClient : IVendorClient
    {
        private readonly VendorApiConfiguration config;
        private readonly IHttpClientFactory httpClientFactory;
        private readonly IValidatorLogger logger;

        public WelsClient(VendorApiConfiguration config, IHttpClientFactory httpClientFactory, IValidatorLogger logger)
        {
            this.config = config;
            this.httpClientFactory = httpClientFactory;
            this.logger = logger;
        }
        
        private async Task<T> Get<T>(string urlPath)
        {
            urlPath = urlPath.TrimStart('/');
            HttpResponseMessage response = await httpClientFactory.CreateClient("VendorHttpClient").GetAsync(urlPath);
            await AssertValidResponse(response);
            return await response.Content.ReadFromJsonAsync<T>();
        }  
    }