.NET: [Flaky test] SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync — HTTP 400 conflict despite in-test retry #5847

Summary

Foundry.Hosting.IntegrationTests.SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync is failing intermittently in the dotnet-foundry-hosted-it job with an HTTP 400 conflict from the hosted-agents service even though the test already implements a 5-attempt retry for this exact failure class.

This is not a flaky test in the unstable-by-default sense — it is the test catching a real eventual-consistency/identity-binding behavior in the hosted-agents service that the existing in-test mitigation can no longer absorb. Filing so it can be addressed permanently.

Failure signature

System.ClientModel.ClientResultException : HTTP 400 (ServiceError: conflict)

The resource already exists or was modified concurrently. Please retry. [Request ID: 979e52b45be320cfa0e5adf2ff314cc0]

{
  ""code"": ""conflict"",
  ""message"": ""The resource already exists or was modified concurrently. Please retry. ..."",
  ""type"": ""invalid_request_error"",
  ...
}
   at OpenAI.Responses.ResponsesClient.CreateResponseAsync(...)
   at Azure.AI.Extensions.OpenAI.ProjectResponsesClient.CreateResponseAsync(CreateResponseOptions, CancellationToken)
   at Foundry.Hosting.IntegrationTests.SessionFilesHostedAgentTests.UploadedFile_IsReadByHostedAgentAsync()
     in dotnet/tests/Foundry.Hosting.IntegrationTests/SessionFilesHostedAgentTests.cs:141

Test wall time: 1m 02s 367ms — consistent with all 5 retry attempts being exhausted.

What the test already does

Step 4 of the test wraps responses.CreateResponseAsync(readOptions) in a bounded retry loop that catches exactly this error class:

const int MaxAttempts = 5;
for (int attempt = 1; attempt <= MaxAttempts; attempt++)
{
    try
    {
        rawResponse = await responses.CreateResponseAsync(readOptions);
        break;
    }
    catch (ClientResultException ex) when (
        ex.Status == 400 &&
        ex.Message.Contains(""modified concurrently"", StringComparison.OrdinalIgnoreCase) &&
        attempt < MaxAttempts)
    {
        await Task.Delay(TimeSpan.FromSeconds(2 * attempt));
    }
}

(dotnet/tests/Foundry.Hosting.IntegrationTests/SessionFilesHostedAgentTests.cs:135-151)

The service is staying in conflict state for the full backoff window (2 + 4 + 6 + 8 = 20s plus call latency), so either:

1. The session/conversation revision is taking longer than 20s to settle after the AgentSessionFiles upload, or
2. The retry's identity is being re-bound on every attempt and never converges on the same session container.

Suspected root cause

The test comment already calls this out:

The platform mutates session/conversation revision when AgentSessionFiles uploads land, so an immediate /responses follow-up races and 400's with ""modified concurrently. Please retry."" — the response message literally tells us to retry. Bounded retry handles it.

Hypothesis: the platform-side write-fence after UploadSessionFileAsync is wider than the test's backoff envelope, or the conversation_id route is being recomputed by the service and momentarily disagrees with the upload's session id.

Options to consider

1. Stronger client-side mitigation: increase MaxAttempts, lengthen backoff, or switch to exponential with jitter (e.g. up to 60s total).
2. Service-side guarantee: request that the hosted-agents service either (a) serialize the upload commit before acknowledging, or (b) return a retry-after hint instead of a 400 conflict.
3. Quarantine with [Fact(Skip=""..."")] short-term while This repo is missing a LICENSE file #1 or Adding Microsoft SECURITY.MD #2 is implemented.

Owner

Assigning to myself (PR #5702 area). Will handle so this stops blocking other devs.

Related

Skip PR for the unrelated DevUI flake: #5846 / issue #5845.