.NET: [Bug]: InvalidDataException when resuming from Checkpoint with new AIAgent.Id values

Description

I'm working on a Web API application and I want to create a Workflow using an Agent with Handoff, as I need to handle a fairly complex scenario. Specifically, I need to call tools and AI context providers that depend on the request (for example, because they need access to a DbContext). For this reason, I'm trying to register my agents as scoped.

To implement this scenario, I encountered a problem that I was able to reproduce with this simplified version of the code:

#!/usr/bin/env dotnet

#:sdk Microsoft.NET.Sdk.Web

#:property TargetFramework=net10.0
#:property Nullable=enable
#:property ImplicitUsings=enable
#:property NoWarn=$(NoWarn);MEAI001;MAAI001;OPENAI001
#:property PublishAot=false

#:package Azure.AI.OpenAI@2.1.0
#:package Azure.Identity@1.19.0
#:package Microsoft.Agents.AI.Hosting@1.0.0-preview.260311.1
#:package Microsoft.Agents.AI.OpenAI@1.0.0-rc4
#:package Microsoft.AspNetCore.OpenApi@10.0.4
#:package Swashbuckle.AspNetCore.SwaggerUI@10.1.5

using System.ClientModel;
using System.Collections.Concurrent;
using System.Text.Json;
using Azure.AI.OpenAI;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Hosting;
using Microsoft.Agents.AI.Workflows;
using Microsoft.Extensions.AI;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddChatClient(_ =>
{
    var endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT") ?? throw new InvalidOperationException("AZURE_OPENAI_ENDPOINT is not set.");
    var deploymentName = Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT_NAME") ?? "gpt-4.1";

    var client = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential());
    return client.GetChatClient(deploymentName).AsIChatClient();
});

builder.Services.AddAIAgent("Default", (services, key) =>
{
    var chatClient = services.GetRequiredService<IChatClient>();
    var loggerFactory = services.GetRequiredService<ILoggerFactory>();

    var chatHistoryProvider = new InMemoryChatHistoryProvider(new()
    {
        ChatReducer = new MessageCountingChatReducer(20),
        ReducerTriggerEvent = InMemoryChatHistoryProviderOptions.ChatReducerTriggerEvent.AfterMessageAdded,
        StorageInputRequestMessageFilter = messages =>
        {
            return messages.Where(m => m.GetAgentRequestMessageSourceType() != AgentRequestMessageSourceType.AIContextProvider
              && m.GetAgentRequestMessageSourceType() != AgentRequestMessageSourceType.ChatHistory);
        }
    });

    var triageAgent = chatClient.AsAIAgent(new()
    {
        Name = "triage_agent",
        Description = "Routes messages to the appropriate specialist agent",
        ChatOptions = new()
        {
            Instructions = "You determine which agent to use based on the user's homework question.",
        },
        ChatHistoryProvider = chatHistoryProvider
    },
    loggerFactory: loggerFactory,
    services: services);

    var historyTutor = chatClient.AsAIAgent(new()
    {
        Name = "history_tutor",
        Description = "Specialist agent for historical questions",
        ChatOptions = new()
        {
            Instructions = "You provide assistance with historical queries. Explain important events and context clearly. Only respond about history.",
        }
    },
    loggerFactory: loggerFactory,
    services: services);

    var mathTutor = chatClient.AsAIAgent(new()
    {
        Name = "math_tutor",
        Description = "Specialist agent for math questions",
        ChatOptions = new()
        {
            Instructions = "You provide help with math problems. Explain your reasoning at each step and include examples. Only respond about math.",
        }
    },
    loggerFactory: loggerFactory,
    services: services);

    var workflow = AgentWorkflowBuilder.CreateHandoffBuilderWith(triageAgent)
        .WithHandoffs(triageAgent, [mathTutor, historyTutor])
        .WithHandoffs([mathTutor, historyTutor], triageAgent)
        .Build().AsAIAgent(id: key, name: key);

    return workflow;

}, ServiceLifetime.Scoped)
.WithSessionStore((services, key) =>
{
    return new CustomAgentSessionStore();
}, ServiceLifetime.Singleton);

builder.Services.AddOpenApi();

var app = builder.Build();

// Configure the HTTP request pipeline.
app.UseHttpsRedirection();

app.MapOpenApi();
app.UseSwaggerUI(options =>
{
    options.SwaggerEndpoint("/openapi/v1.json", app.Environment.ApplicationName);
});

app.MapPost("/api/chat", async (ChatRequest request, [FromKeyedServices("Default")] AIAgent agent, [FromKeyedServices("Default")] AgentSessionStore store) =>
{
    var conversationId = request.ConversationId ?? Guid.NewGuid().ToString("N");
    var session = await store.GetSessionAsync(agent, conversationId);

    var response = await agent.RunAsync(request.Message, session);

    await store.SaveSessionAsync(agent, conversationId, session);

    return TypedResults.Ok(new ChatResponse(conversationId, response.Text));
})
.Produces<ChatResponse>();

app.Run();

public record class ChatRequest(string? ConversationId, string Message);

public record class ChatResponse(string ConversationId, string Response);

public sealed class CustomAgentSessionStore : AgentSessionStore
{
    private readonly ConcurrentDictionary<string, JsonElement> sessions = new();

    public override async ValueTask SaveSessionAsync(AIAgent agent, string conversationId, AgentSession session, CancellationToken cancellationToken = default)
    {
        var key = GetKey(conversationId, agent.Id);
        sessions[key] = await agent.SerializeSessionAsync(session, cancellationToken: cancellationToken);
    }

    public override async ValueTask<AgentSession> GetSessionAsync(AIAgent agent, string conversationId, CancellationToken cancellationToken = default)
    {
        var key = GetKey(conversationId, agent.Id);
        JsonElement? threadContent = sessions.TryGetValue(key, out var existingThread) ? existingThread : null;

        return threadContent switch
        {
            null => await agent.CreateSessionAsync(cancellationToken),
            _ => await agent.DeserializeSessionAsync(threadContent.Value, cancellationToken: cancellationToken),
        };
    }

    private static string GetKey(string conversationId, string agentId)
        => conversationId;
}

Note that in this example I haven't defined any tools or context providers that require a scoped context, but they are present in the real application. I'm also using a custom AgentSessionStore that relies on a conversationId provided by the user (this works for my use case, and I'm confident it is unique for each user). Of course, in the real application the data storage logic is more complex.

The first time I invoke the /api/chat endpoint, it works as expected, but on the second invocation I get this exception:

System.IO.InvalidDataException: The specified checkpoint is not compatible with the workflow associated with this runner.
         at Microsoft.Agents.AI.Workflows.InProc.InProcessRunner.RestoreCheckpointAsync(CheckpointInfo checkpointInfo, CancellationToken cancellationToken)
         at Microsoft.Agents.AI.Workflows.InProc.InProcessRunner.ResumeStreamAsync(ExecutionMode mode, CheckpointInfo fromCheckpoint, CancellationToken cancellationToken)
         at Microsoft.Agents.AI.Workflows.InProc.InProcessExecutionEnvironment.ResumeStreamingAsync(Workflow workflow, CheckpointInfo fromCheckpoint, CancellationToken cancellationToken)
         at Microsoft.Agents.AI.Workflows.WorkflowSession.CreateOrResumeRunAsync(List`1 messages, CancellationToken cancellationToken)
         at Microsoft.Agents.AI.Workflows.WorkflowSession.InvokeStageAsync(CancellationToken cancellationToken)+MoveNext()
         at Microsoft.Agents.AI.Workflows.WorkflowSession.InvokeStageAsync(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Microsoft.Agents.AI.Workflows.WorkflowHostAgent.RunCoreAsync(IEnumerable`1 messages, AgentSession session, AgentRunOptions options, CancellationToken cancellationToken)
         at Microsoft.Agents.AI.Workflows.WorkflowHostAgent.RunCoreAsync(IEnumerable`1 messages, AgentSession session, AgentRunOptions options, CancellationToken cancellationToken)

I'm wondering if this is somehow related to the issue #3620.

Package Versions

Microsoft.Agents.AI.Hosting: 1.0.0-preview.260311.1, Microsoft.Agents.AI.OpenAI: 1.0.0-rc4

.NET Version

.NET 10

The compatibility check is generally about workflow structure and types on the executors. If the structure changed, the checkpoint is considered incompatible.

Jacob Alber (@lokitoth)

And so, Jacob Alber (@lokitoth), in my scenario how can I solve the issue?

Marco Minerva (@marcominerva)

Marco Minerva (@marcominerva) Still need to investigate precisely why the verifier thinks the structure changed. I was just adding some context for Peter Ibekwe (@peibekwe)

Marco Minerva (@marcominerva)

Hey Marco Minerva (@marcominerva) - you wouldn't happen to have a copy of the serialized checkpoint that this is happening to, would you?

Jacob Alber (@lokitoth)

Hi Jacob Alber (@lokitoth), I don't have it right now, but I can reliably reproduce the issue even with the latest versions of the Agent Framework packages. How can I obtain this checkpoint?

If you set things up to use the FileSystemJsonCheckpointManager:

// Set up the CheckpointManager
var maybeJsonSerializationOptions = haveCustomJsonTypes ? <something> : null;
var checkpointStore = new FileSystemJsonCheckpointStore(<directory>)
var checkpointManager = CheckpointManager.CreateJson(checkpointStore, maybeJsonSerializationOptions)

// In your .AsAgentCall
.AsAIAgent(id: key, name: key, InProcessExecution.Concurrent.WithCheckpointing(checkpointManager));

When the checkpointing happens, it will write the checkpoint to that folder, along with an index. If you do not specify this, it will use InMemory checkpointing and rely on that to generate the JSON-serializable state that gets put in the session state store.

The other thing to consider is that now that AgentSession has the ability to store more arbitrary data in a generalized form, it might be a good idea to move the state data from being a custom AgentSession to living in the StateBag -- though this would be a breaking change.

Here you can find the checkpoint: Checkpoint.zip

Marco Minerva (@marcominerva)

Found the issue, I think. When you are resuming the workflow after having reinstantiated the AIAgent instances, they have different Guids for IDs and thus the workflow thinks you are not resuming into the same one.

Here's what's going on:

private sealed class IdFixingAgent(string id, AIAgent innerAgent) : DelegatingAIAgent(innerAgent)
{
    protected override string? IdCore => id;

    /* If you actually use this, you will probably need to overload the Run/RunAsync methods to
       overwrite the AgentId on the outgoing AgentResponse/AgentResponseUpdate */
}

[Theory]
[InlineData(true)]
[InlineData(false)]
internal async Task Checkpoint_Resume_WithReinstantiatedAgents_DoesNotFail_Async(bool useFixedId)
{
    InProcessExecutionEnvironment ee = ExecutionEnvironment.InProcess_OffThread.ToWorkflowExecutionEnvironment();

    Workflow workflow = CreateWorkflow();

    CheckpointManager checkpointManager = CheckpointManager.CreateJson(new InMemoryJsonStore());
    ee = ee.WithCheckpointing(checkpointManager);

    Run firstRun = await ee.RunAsync(workflow, "Hello", sessionId: "SessionId");
    SuperStepCompletedEvent lastStep = firstRun.OutgoingEvents.OfType<SuperStepCompletedEvent>().Last();

    // Verify and extract the last checkpoint.
    CheckpointInfo checkpoint = lastStep.CompletionInfo.Should().NotBeNull()
                                    .And.Subject.As<SuperStepCompletionInfo>()
                                        .Checkpoint.Should().NotBeNull()
                                    .And.Subject.As<CheckpointInfo>();

    workflow = CreateWorkflow(); // Reinstantiate the workflow to ensure agents are reinstantiated as well.
    StreamingRun resumed = await ee.ResumeStreamingAsync(workflow, checkpoint); // throws when `useFixedId != true` due to agent ID mismatch

    await resumed.TrySendMessageAsync("Hello again");
    await resumed.TrySendMessageAsync(new TurnToken());

    await resumed.RunToCompletionAsync(evt => default);

    Workflow CreateWorkflow()
    {
        MockChatClient chatClient = new(
            (messages, options) => new ChatResponse([new ChatMessage(ChatRole.Assistant, "Hello")]));

        AIAgent innerAgent = chatClient.AsAIAgent();
        AIAgent agent = useFixedId
                      ? new IdFixingAgent("FixedId", innerAgent)
                      : innerAgent;

        WorkflowBuilder builder = new(agent);

        return builder.WithOutputFrom(agent)
                      .Build();
    }
}

Marco Minerva (@marcominerva): In the short term you can use something like IdFixingAgent to work around this issue. I need to figure out what the right solution here is.

Marco Minerva (@marcominerva)

Hi Marco Minerva (@marcominerva), thanks for the detailed repro; it made the root cause easier to pin down.

Here's what's happening:
Your inner agents (triage_agent, math_tutor, history_tutor) set only a Name, not an Id. When an agent has no explicit Id, it gets a new random id every time it's constructed. Because you register the workflow as Scoped, each request rebuilds these agents with fresh ids, which changes the executor identities inside the workflow. The checkpoint stored on the first request records the original executor identities, so on the second request the reconstructed workflow no longer matches it; and resume fails. This is the checkpoint doing an intentional safety check, not data corruption.

The Fix:
Give every inner agent a stable, deterministic Id (a fixed logical role - not the conversationId /user id). Your agent Names are already constant, which is important too: when a Name is set, the executor identity includes it, so it must stay stable across reconstructions.

var triageAgent = chatClient.AsAIAgent(new()
{
    Id = "triage-agent",      // stable across scopes/requests
    Name = "triage_agent",
    Description = "...",
    ChatOptions = new() { Instructions = "..." },
    ChatHistoryProvider = chatHistoryProvider
}, loggerFactory: loggerFactory, services: services);

// Do the same for the others:
//   Id = "math-tutor"  (Name = "math_tutor")
//   Id = "history-tutor" (Name = "history_tutor")

With stable ids, the workflow rebuilds to the same executor identities each request and the checkpoint resume correctly; scoped registration is totally fine.

We've also added regression tests and sample guidance for exactly this scenario in #7415 (and are updating the docs to make the stable-id requirement clearer), so this is easier to discover going forward.

Happy to take a closer look if you still see issues after pinning the ids.

Peter Ibekwe (@peibekwe)

Thank you Peter Ibekwe (@peibekwe). Using a fixed ID solved the issue.

Navigation Menu

Repository navigation

.NET: [Bug]: InvalidDataException when resuming from Checkpoint with new AIAgent.Id values #4793

Description

Package Versions

.NET Version

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

.NET: [Bug]: InvalidDataException when resuming from Checkpoint with new AIAgent.Id values #4793

Description

Description

Package Versions

.NET Version

Activity

lokitoth commented on Mar 25, 2026

marcominerva commented on Mar 25, 2026

lokitoth commented on Mar 25, 2026

1 remaining item

lokitoth commented on May 26, 2026

marcominerva commented on May 26, 2026

lokitoth commented on May 26, 2026

marcominerva commented on May 26, 2026

lokitoth commented on May 28, 2026

peibekwe commented on Aug 3, 2026

marcominerva commented on Aug 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions