Skip to content

HttpClientResiliencePredicates.IsHttpConnectionTimeout will always false after aot publish #7188

@Misaka-L

Description

@Misaka-L

Description

In the latest version of Microsoft.Extensions.Http.Resilience, HttpClientResiliencePredicates.IsHttpConnectionTimeout assume that a OperationCanceledException which Source is System.Private.CoreLib and InnerException is TimeoutException is a connection timeout exception.

internal static bool IsHttpConnectionTimeout(in Outcome<HttpResponseMessage> outcome, in CancellationToken cancellationToken)
=> !cancellationToken.IsCancellationRequested
&& outcome.Exception is OperationCanceledException { Source: "System.Private.CoreLib", InnerException: TimeoutException };

It usually works well, but after aot publish. The Exception.Source will always return <unknown> (If metadata for current method cannot be established).

https://github.com/dotnet/runtime/blob/138d5fe8f32e954a3a834be6360a834688130d4f/src/libraries/System.Private.CoreLib/src/System/Exception.cs#L94-L100

public virtual string? Source
{
    [UnconditionalSuppressMessage("ReflectionAnalysis", "IL2026:RequiresUnreferencedCode",
        Justification = "The API will return <unknown> if the metadata for current method cannot be established.")]
    get => _source ??= HasBeenThrown ? (TargetSite?.Module.Assembly.GetName().Name ?? "<unknown>") : null;
    set => _source = value;
}

Which case any resilience pipeline using HttpRetryStrategyOptions with default ShouldHandle will never retry if connection timeout error occured after aot publish.

Reproduction Steps

Using follow program:

#!/usr/local/share/dotnet/dotnet run

#:package Microsoft.Extensions.Http.Resilience@10.1.0
#:package Microsoft.Extensions.Logging.Console@10.0.0

using Microsoft.Extensions.Http.Resilience;
using Microsoft.Extensions.Logging;
using Polly;

using ILoggerFactory factory = LoggerFactory.Create(builder => builder.AddConsole());
ILogger logger = factory.CreateLogger("Program");

var socketHttpHandler = new SocketsHttpHandler
{
    ConnectTimeout = TimeSpan.FromMilliseconds(1) // to trigger connect timeout
};
var retryPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 1,
        // just fail fast
        Delay = TimeSpan.FromMilliseconds(1),
        BackoffType = DelayBackoffType.Linear
    })
    .ConfigureTelemetry(factory)
    .Build();

var client = new HttpClient(new ResilienceHandler(retryPipeline)
{
    InnerHandler = socketHttpHandler
});

try
{
    await client.GetAsync("https://example.com");
}
catch (Exception ex)
{
    logger.LogInformation("Exception.Source: {ExceptionSource}", ex.Source); // Should be `System.Private.CoreLib`, but `<unknown>` after aot publish
}

Just dotnet run

dotnet run .\retry-fail-aot.cs
warn: Polly[3]
      Execution attempt. Source: '(null)/(null)/Retry', Operation Key: '', Result: 'The operation was canceled.', Handled: 'True', Attempt: '0', Execution Time: 30.3317ms
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.TimeoutException: A connection could not be established within the configured ConnectTimeout.
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.CreateConnectTimeoutException(OperationCanceledException oce)
         ......
         at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext()
warn: Polly[0]
      Resilience event occurred. EventName: 'OnRetry', Source: '(null)/(null)/Retry', Operation Key: '', Result: 'The operation was canceled.'
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.TimeoutException: A connection could not be established within the configured ConnectTimeout.
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.CreateConnectTimeoutException(OperationCanceledException oce)
         ......
         at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext()
fail: Polly[3]
      Execution attempt. Source: '(null)/(null)/Retry', Operation Key: '', Result: 'The operation was canceled.', Handled: 'True', Attempt: '1', Execution Time: 17.4763ms
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.TimeoutException: A connection could not be established within the configured ConnectTimeout.
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.CreateConnectTimeoutException(OperationCanceledException oce)
         ......
         at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext()
info: Program[0]
      Exception.Source: System.Private.CoreLib

After aot publish

.\retry-fail-aot.exe
info: Polly[3]
      Execution attempt. Source: '(null)/(null)/Retry', Operation Key: '', Result: 'The operation was canceled.', Handled: 'False', Attempt: '0', Execution Time: 21.8876ms
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.TimeoutException: A connection could not be established within the configured ConnectTimeout.
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.CreateConnectTimeoutException(OperationCanceledException) + 0xb3
         ......
         at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext() + 0x91
info: Program[0]
      Exception.Source: <unknown>

Resilience event occurred. EventName: 'OnRetry' was missing from program after aot publish. And Exception.Source is <unknown>.

Expected behavior

HttpClientResiliencePredicates.IsHttpConnectionTimeout should return true if exception is connect timeout exception after aot publish.

Actual behavior

HttpClientResiliencePredicates.IsHttpConnectionTimeout always return false after aot publish.

Regression?

10.0.0 have same issue.

Known Workarounds

Write a own HttpClientResiliencePredicates and replace IsHttpConnectionTimeout() to following version:

internal static bool IsHttpConnectionTimeout(in Outcome<HttpResponseMessage> outcome, in CancellationToken cancellationToken)
    => !cancellationToken.IsCancellationRequested
       && outcome.Exception is OperationCanceledException { InnerException: TimeoutException };
new HttpRetryStrategyOptions()
{
    ShouldHandle = args => new ValueTask<bool>(CustomHttpClientResiliencePredicates.IsTransient(args.Outcome, args.Context.CancellationToken))
}

Configuration

.NET SDK:
 Version:           10.0.101
 Commit:            fad253f51b
 Workload version:  10.0.100-manifests.c57ac48b
 MSBuild version:   18.0.6+fad253f51

 OS Name:     Windows
 OS Version:  10.0.26100
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\10.0.101\

No .net workloads was installed.

Host:
  Version:      10.0.1
  Architecture: x64
  Commit:       fad253f51b

.NET SDKs installed:
  9.0.305 [C:\Program Files\dotnet\sdk]
  10.0.101 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 10.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Other information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-resiliencebugThis issue describes a behavior which is not expected - a bug.untriaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions