Update to OpenAI 2.6.0 (#6996)

* Update to OpenAI 2.6.0

In addition to updating the dependency, it now enables us to do additional things:
- Propagate ChatOptions/EmbeddingGeneratorOptions.ModelID to allow overriding the model per-request.
- Propagate AIContent results of tool invocations using non-string tool results.
- Add support for container file annotations.
- Fixed handling of GetResponse{Streaming}Async to use RequestOptions
- Remove hacky serialization code around ResponseCreationOptions.

A previous PR added HostedFileContent.Name/MediaType, but we missed adding HasTopLevelMediaType (which both DataContent and UriContent have). I had a need for that here, so included it.

* Add more tests based on code coverage gaps

* Fix handling of role in AsChatMessages

Author: stephentoub

eng/packages/General.props

@@ -20,7 +20,7 @@
     <PackageVersion Include="Microsoft.ML.Tokenizers" Version="$(MicrosoftMLTokenizersVersion)" />
     <PackageVersion Include="Newtonsoft.Json" Version="13.0.3" />
     <PackageVersion Include="OllamaSharp" Version="5.1.9" />
-    <PackageVersion Include="OpenAI" Version="2.5.0" />
+    <PackageVersion Include="OpenAI" Version="2.6.0" />
     <PackageVersion Include="Polly" Version="8.4.2" />
     <PackageVersion Include="Polly.Core" Version="8.4.2" />
     <PackageVersion Include="Polly.Extensions" Version="8.4.2" />

src/Libraries/Microsoft.Extensions.AI.Abstractions/CHANGELOG.md

@@ -5,6 +5,7 @@
 - Updated `AIFunctionFactory` to respect `[DisplayName(...)]` on functions as a way to override the function name.
 - Updated `AIFunctionFactory` to respect `[DefaultValue(...)]` on function parameters as a way to specify default values.
 - Added `CodeInterpreterToolCallContent`/`CodeInterpreterToolResultContent` for representing code interpreter tool calls and results.
+- Added `Name`, `MediaType`, and `HasTopLevelMediaType` to `HostedFileContent`.
 - Fixed the serialization/deserialization of variables typed as `UserInputRequestContent`/`UserInputResponseContent`.
 
 ## 9.10.1

src/Libraries/Microsoft.Extensions.AI.Abstractions/Contents/HostedFileContent.cs

@@ -50,6 +50,24 @@ public string? MediaType
     /// <summary>Gets or sets an optional name associated with the file.</summary>
     public string? Name { get; set; }
 
+    /// <summary>
+    /// Determines whether the <see cref="MediaType"/>'s top-level type matches the specified <paramref name="topLevelType"/>.
+    /// </summary>
+    /// <param name="topLevelType">The type to compare against <see cref="MediaType"/>.</param>
+    /// <returns><see langword="true"/> if the type portion of <see cref="MediaType"/> matches the specified value; otherwise, false.</returns>
+    /// <remarks>
+    /// <para>
+    /// A media type is primarily composed of two parts, a "type" and a "subtype", separated by a slash ("/").
+    /// The type portion is also referred to as the "top-level type"; for example,
+    /// "image/png" has a top-level type of "image". <see cref="HasTopLevelMediaType"/> compares
+    /// the specified <paramref name="topLevelType"/> against the type portion of <see cref="MediaType"/>.
+    /// </para>
+    /// <para>
+    /// If <see cref="MediaType"/> is <see langword="null"/>, this method returns <see langword="false"/>.
+    /// </para>
+    /// </remarks>
+    public bool HasTopLevelMediaType(string topLevelType) => MediaType is not null && DataUriParser.HasTopLevelMediaType(MediaType, topLevelType);
+
     /// <summary>Gets a string representing this instance to display in the debugger.</summary>
     [DebuggerBrowsable(DebuggerBrowsableState.Never)]
     private string DebuggerDisplay

src/Libraries/Microsoft.Extensions.AI.Abstractions/Microsoft.Extensions.AI.Abstractions.json

@@ -2006,6 +2006,10 @@
         {
           "Member": "Microsoft.Extensions.AI.HostedFileContent.HostedFileContent(string fileId);",
           "Stage": "Stable"
+        },
+        {
+          "Member": "bool Microsoft.Extensions.AI.HostedFileContent.HasTopLevelMediaType(string topLevelType);",
+          "Stage": "Stable"
         }
       ],
       "Properties": [

src/Libraries/Microsoft.Extensions.AI.OpenAI/CHANGELOG.md

@@ -2,20 +2,23 @@
 
 ## NOT YET RELEASED
 
-- Updated the `IChatClient` for the OpenAI Responses API to allow either conversation or response ID for `ChatOptions.ConversationId`.
+- Updated to depend on OpenAI 2.6.0.
+- Updated the OpenAI Responses `IChatClient` to allow either conversation or response ID for `ChatOptions.ConversationId`.
+- Updated the OpenAI Responses `IChatClient` to support `AIFunction`s that return `AIContent` like `DataContent`.
+- Updated the OpenAI Chat Completion `IChatClient`, the Responses `IChatClient`, and the `IEmbeddingGenerator` to support per-request `ModelId` overrides.
 - Added an `AITool` to `ResponseTool` conversion utility.
 - Updated to accommodate the additions in `Microsoft.Extensions.AI.Abstractions`.
 
 ## 9.10.1-preview.1.25521.4
 
-- Updated the `IChatClient` for the OpenAI Responses API to support connectors with `HostedMcpServerTool`.
-- Fixed the `IChatClient` for the OpenAI Responses API to roundtrip a `ResponseItem` stored in an `AIContent` in a `ChatRole.User` message.
+- Updated the OpenAI Responses `IChatClient` to support connectors with `HostedMcpServerTool`.
+- Fixed the OpenAI Responses `IChatClient` to roundtrip a `ResponseItem` stored in an `AIContent` in a `ChatRole.User` message.
 - Updated to accommodate the additions in `Microsoft.Extensions.AI.Abstractions`.
 
 ## 9.10.0-preview.1.25513.3
 
-- Fixed issue with `IChatClient` for the OpenAI Assistants API where a chat history including unrelated function calls would cause an exception.
-- Fixed issue with `IChatClient` for the OpenAI Assistants API sending a tool in `ChatOptions.Tools` that had the same name as a function configured with the Assistant would cause an exception.
+- Fixed issue with the OpenAI Assistants `IChatClient` where a chat history including unrelated function calls would cause an exception.
+- Fixed issue with the OpenAI Assistants `IChatClient` sending a tool in `ChatOptions.Tools` that had the same name as a function configured with the Assistant would cause an exception.
 - Updated to accommodate the additions in `Microsoft.Extensions.AI.Abstractions`.
 
 ## 9.9.1-preview.1.25474.6
@@ -38,7 +41,7 @@
 
 - Updated to depend on OpenAI 2.3.0.
 - Added more conversion helpers for converting bidirectionally between Microsoft.Extensions.AI messages and OpenAI messages.
-- Fixed handling of multiple response messages in the Responses `IChatClient`.
+- Fixed handling of multiple response messages in the OpenAI Responses `IChatClient`.
 - Updated to accommodate the additions in `Microsoft.Extensions.AI.Abstractions`.
 
 ## 9.7.1-preview.1.25365.4

src/Libraries/Microsoft.Extensions.AI.OpenAI/Microsoft.Extensions.AI.OpenAI.csproj

@@ -16,7 +16,7 @@
   <PropertyGroup>
     <TargetFrameworks>$(TargetFrameworks);netstandard2.0</TargetFrameworks>
     <NoWarn>$(NoWarn);CA1063</NoWarn>
-    <NoWarn>$(NoWarn);OPENAI001;OPENAI002;MEAI001</NoWarn>
+    <NoWarn>$(NoWarn);OPENAI001;OPENAI002;MEAI001;SCME0001</NoWarn>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <DisableNETStandardCompatErrors>true</DisableNETStandardCompatErrors>
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
@@ -26,8 +26,6 @@
     <InjectCompilerFeatureRequiredOnLegacy>true</InjectCompilerFeatureRequiredOnLegacy>
     <InjectExperimentalAttributeOnLegacy>true</InjectExperimentalAttributeOnLegacy>
     <InjectRequiredMemberOnLegacy>true</InjectRequiredMemberOnLegacy>
-    <InjectSharedEmptyCollections>true</InjectSharedEmptyCollections>
-    <InjectSharedServerSentEvents>true</InjectSharedServerSentEvents>
     <InjectStringHashOnLegacy>true</InjectStringHashOnLegacy>
   </PropertyGroup>
 

src/Libraries/Microsoft.Extensions.AI.OpenAI/MicrosoftExtensionsAIChatExtensions.cs

@@ -183,6 +183,7 @@ public static async IAsyncEnumerable<StreamingChatCompletionUpdate> AsOpenAIStre
             switch (message)
             {
                 case AssistantChatMessage acm:
+                    resultMessage.Role = ChatRole.Assistant;
                     resultMessage.AuthorName = acm.ParticipantName;
                     OpenAIChatClient.ConvertContentParts(acm.Content, resultMessage.Contents);
                     foreach (var toolCall in acm.ToolCalls)
@@ -195,21 +196,25 @@ public static async IAsyncEnumerable<StreamingChatCompletionUpdate> AsOpenAIStre
                     break;
 
                 case UserChatMessage ucm:
+                    resultMessage.Role = ChatRole.User;
                     resultMessage.AuthorName = ucm.ParticipantName;
                     OpenAIChatClient.ConvertContentParts(ucm.Content, resultMessage.Contents);
                     break;
 
                 case DeveloperChatMessage dcm:
+                    resultMessage.Role = ChatRole.System;
                     resultMessage.AuthorName = dcm.ParticipantName;
                     OpenAIChatClient.ConvertContentParts(dcm.Content, resultMessage.Contents);
                     break;
 
                 case SystemChatMessage scm:
+                    resultMessage.Role = ChatRole.System;
                     resultMessage.AuthorName = scm.ParticipantName;
                     OpenAIChatClient.ConvertContentParts(scm.Content, resultMessage.Contents);
                     break;
 
                 case ToolChatMessage tcm:
+                    resultMessage.Role = ChatRole.Tool;
                     resultMessage.Contents.Add(new FunctionResultContent(tcm.ToolCallId, ToToolResult(tcm.Content))
                     {
                         RawRepresentation = tcm,

src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIChatClient.cs

@@ -554,12 +554,12 @@ private ChatCompletionOptions ToOpenAIOptions(ChatOptions? options)
     {
         if (options is null)
         {
-            return new ChatCompletionOptions();
+            return new();
         }
 
         if (options.RawRepresentationFactory?.Invoke(this) is not ChatCompletionOptions result)
         {
-            result = new ChatCompletionOptions();
+            result = new();
         }
 
         result.FrequencyPenalty ??= options.FrequencyPenalty;
@@ -568,6 +568,7 @@ private ChatCompletionOptions ToOpenAIOptions(ChatOptions? options)
         result.PresencePenalty ??= options.PresencePenalty;
         result.Temperature ??= options.Temperature;
         result.Seed ??= options.Seed;
+        OpenAIClientExtensions.PatchModelIfNotSet(ref result.Patch, options.ModelId);
 
         if (options.StopSequences is { Count: > 0 } stopSequences)
         {

src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIClientExtensions.cs

@@ -2,6 +2,7 @@
 // The .NET Foundation licenses this file to you under the MIT license.
 
 using System;
+using System.ClientModel.Primitives;
 using System.Collections.Generic;
 using System.Diagnostics.CodeAnalysis;
 using System.Text;
@@ -228,6 +229,19 @@ internal static string ImageUriToMediaType(Uri uri)
             "image/*";
     }
 
+    /// <summary>Sets $.model in <paramref name="patch"/> to <paramref name="modelId"/> if not already set.</summary>
+    internal static void PatchModelIfNotSet(ref JsonPatch patch, string? modelId)
+    {
+        if (modelId is not null)
+        {
+            _ = patch.TryGetValue("$.model"u8, out string? existingModel);
+            if (existingModel is null)
+            {
+                patch.Set("$.model"u8, modelId);
+            }
+        }
+    }
+
     /// <summary>Used to create the JSON payload for an OpenAI tool description.</summary>
     internal sealed class ToolJson
     {

src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIEmbeddingGenerator.cs

@@ -107,10 +107,12 @@ private OpenAI.Embeddings.EmbeddingGenerationOptions ToOpenAIOptions(EmbeddingGe
     {
         if (options?.RawRepresentationFactory?.Invoke(this) is not OpenAI.Embeddings.EmbeddingGenerationOptions result)
         {
-            result = new OpenAI.Embeddings.EmbeddingGenerationOptions();
+            result = new();
         }
 
         result.Dimensions ??= options?.Dimensions ?? _dimensions;
+        OpenAIClientExtensions.PatchModelIfNotSet(ref result.Patch, options?.ModelId);
+
         return result;
     }
 }