.NET feedback: Structured output feels more cumbersome in AF + Have issues with Enums in result object

[rwjdk]

(Based on how it is done in the sample: Agent_Step05_StructuredOutput)

After playing around with structured Output in AF, it feels like a step down from MEAI and SK

1. MEAI feels like it has the best solution, as you can do something like this:

string question = "What is the capital of Denmark and what is its population?";
ChatResponse<City> structuredOutputResponse = await chatClient.GetResponseAsync<City>(question);
City result = structuredOutputResponse.Result;

//(AF do not have a similar agent.RunAsync<City>(question)) ☹️

2. SK was a bit more cumbersome than MEAI with the need to set the typeof() and manual deserialization

   ChatCompletionAgent agent = new()
   {
       Kernel = kernel,
       Instructions = "You are a City Expert",
       Arguments = new KernelArguments(new AzureOpenAIPromptExecutionSettings
       {
           ResponseFormat = typeof(City)
       })
   };
   
   ...
   
   await foreach (AgentResponseItem<ChatMessageContent> response in agent.InvokeAsync(message))
   {
       string json = response.Message.Content!;
       City? city = JsonSerializer.Deserialize<City>(json, new JsonSerializerOptions
       {
           Converters = { new JsonStringEnumConverter() }
       });
   }

   3. AF, however, feels even more cumbersome than SK

   ChatOptions = new()
   {
       ResponseFormat = ChatResponseFormat.ForJsonSchema<City>()
   }

   ...And if the Structured Output object has Enums, you need the following JSON Serializer options for it to work in AF

   JsonSerializerOptions jsonSerializerOptions = new()
   {
       TypeInfoResolver = new DefaultJsonTypeInfoResolver(),
       PropertyNameCaseInsensitive = true,
       Converters = { new JsonStringEnumConverter() }
   };

   (It even seems to generate the incorrect schema if Enums are inside nested objects in the Structured output object, giving back non-existing enum values as the schema does not generate the enum valid values)

   ---

   It would be cool if AF could have the MEAI approach...

   ---

   For context, here is the City and Underlying Enum

   public class City
   {
       public required string NameOfCapital { get; set; }
       public required string Population { get; set; }
       public required Size SizeOfCity { get; set; }
   }
   
   public enum Size
   {
       RandomValue1, //values AI can't guess randomly
       RandomValue2 //values AI can't guess randomly
   }

[SergeyMenshykh]

Adding more context for SO in AF:

• https://github.com/microsoft/agent-framework/blob/main/docs/decisions/0001-agent-run-response.md#structuredoutputs
• https://github.com/microsoft/agent-framework/blob/main/docs/decisions/0001-agent-run-response.md#structuredoutputs-decision

CC: @stephentoub, @westey-m

[SergeyMenshykh]

@rwjdk, thanks for your feedback. To proceed, we would like to better understand the scenarios where you plan to use structural output.

Are these single-agent scenarios where you have only one agent sending prompts to and receiving structural output (SO) from? If so, which AF agent type do you plan to use for them, and what would you expect to happen if an agent does not support SO but is asked to return one via agent.RunAsync<City>(question)?

Or, are your scenarios multi-agent ones, utilizing one of the AF workflows?

[rwjdk-test]

Hi @SergeyMenshykh : rwjdk here but from a secondary account as there are some issues accessing this site for those of use that was in the private preview (hope it will be resolved soon)

As for your question, my usecase is primarily Single Agent usage using AzureOpenAI ChatClient.CreateAIAgent.

In my book if an underlying LLM does not support SO, then the call should just throw a raw LLM exception (just like a model would if you set ReasoningEffort on a non-reasoning Model (there will always be combinations of something that are not supported; you learn that and know what works and what don't)... I would of cause mention/warn in the XML Summaries that not all models support SO

Hope that helps... Else feel free to ask further

And thank for a new awesome framework so far btw. Have been exploring it all day 💪👍😊

[rosieks]

I have the case that I generate test exercise. I want to get them as JSON. I tried to test workflows and have editor as in the blog post that you made. Unfortunately as a result I get a response that is combination of two calls to LLM. It would be nice if the framework handle it and return the final result as an deserialized object.