Invalid Value: 'file'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', and 'audio'

[jozkee]

Service

Azure OpenAI

Describe the bug

I'm trying to use Files API to upload an image and then ask the model to describe it but I'm getting an error.

Unhandled exception. System.ClientModel.ClientResultException: HTTP 400 (invalid_request_error: invalid_value)
Parameter: messages[0].content[1].type

Invalid Value: 'file'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', and 'audio'.
   at OpenAI.ClientPipelineExtensions.ProcessMessage(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at OpenAI.Chat.ChatClient.CompleteChat(BinaryContent content, RequestOptions options)
   at OpenAI.Chat.ChatClient.CompleteChat(IEnumerable`1 messages, ChatCompletionOptions options, CancellationToken cancellationToken)
   at Azure.AI.OpenAI.Chat.AzureChatClient.CompleteChat(IEnumerable`1 messages, ChatCompletionOptions options, CancellationToken cancellationToken)
   at OpenAI.Chat.ChatClient.CompleteChat(ChatMessage[] messages)
   at Program.Main(String[] args) in C:\Users\dacantu\source\repos\ConsoleApp37.OpenAI\ConsoleApp.OpenAI\Program.cs:line 30
   at Program.<Main>(String[] args)

I want to achieve the same result I get from passing the image as base64 (commented-out line in code snippet). Is my expectation wrong?

For context I want to avoid buffering large contents into memory when interacting with the model as described in dotnet/extensions#5819.

---

I was also unable to use UploadFileAsync with FileUploadPurpose.UserData, I get Invalid value for purpose.. As per the OpenAI docs, user_data seems more appropriate:

You can upload these files to the Files API with any purpose, but we recommend using the user_data purpose for files you plan to use as model inputs.

Steps to reproduce

using System.ClientModel;
using Azure.AI.OpenAI;
using Azure.Identity;
using OpenAI;
using OpenAI.Chat;
using OpenAI.Files;

class Program
{
    static async Task Main(string[] args)
    {
        // Initialize the OpenAI client
        OpenAIClient openAIClient = new AzureOpenAIClient(
            new Uri("https://my-endpoint.openai.azure.com/"),
            new DefaultAzureCredential());

        ChatClient chatClient = openAIClient.GetChatClient("gpt-4o-mini");

        // Upload file and then reference it by ID.
        OpenAIFileClient fileClient = openAIClient.GetOpenAIFileClient();
        using Stream fileStream = File.OpenRead(@"C:\Users\dacantu\Desktop\pancake_bunny.jpg");
        ClientResult<OpenAIFile> fileResult = await fileClient.UploadFileAsync(fileStream, "bunny.jpg", FileUploadPurpose.Assistants);

        var message = new UserChatMessage([
            ChatMessageContentPart.CreateTextPart("Describe this image."),
            ChatMessageContentPart.CreateFilePart(fileResult.Value.Id),
            //ChatMessageContentPart.CreateImagePart(new BinaryData(File.ReadAllBytes(@"C:\Users\dacantu\Desktop\pancake_bunny.jpg")), "image/jpeg"),
        ]);

        ChatCompletion completion = chatClient.CompleteChat(message);
        foreach (ChatMessageContentPart? contentPart in completion.Content)
        {
            Console.Write(contentPart.Text);
        }
    }
}

Code snippets

OS

Windows

.NET version

.NET 9.0

Library version

2.2.0-beta.4, both OpenAI and Azure.AI.OpenAI

[joseharriaga]

Thank you for reaching out, @jozkee ! File inputs currently only work with PDF files. You can find more information here:
🔗 https://platform.openai.com/docs/guides/pdf-files

[jozkee]

Are there plans to support images and/or arbitrary file formats?

[joseharriaga]

Unfortunately, I do not know the answer to that since it is related to a limitation in the REST API and not the .NET library. You might be able to get more help here:
🔗 https://help.openai.com/en/
🔗 https://community.openai.com/

[jozkee]

@joseharriaga thanks. I tried uploading this PDF and I get the same error.

[joseharriaga]

I just noticed that you are using Azure OpenAI. File inputs are not yet available in the Azure OpenAI service.

[AMORALESFRONTSTEP]

I'm also facing this error through Azure OpenAI service. Do we know when file inputs will be available there?

[Justarrrrr]

I encountered the same problem. How can I send pictures to GPT through the API?

[robinluo12]

this works for me, you can have a try:

var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMinutes(5);

var kernel = Kernel.CreateBuilder()
    .AddAzureOpenAIChatCompletion(
        deploymentName: chatCompletionModelDeploymentName,
        endpoint: azureOpenAiEndpoint,
        credentials: new DefaultAzureCredential(new DefaultAzureCredentialOptions { TenantId = "******" }),
        httpClient: httpClient)
    .Build();

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

byte[] imageBytes = File.ReadAllBytes("C:\\*****\\*****.jpg");

history.Add(new()
{
    Role = AuthorRole.User,
    Items =
    [
        new TextContent { Text = prompt },
        new ImageContent(imageBytes, "image/jpeg")
    ]
});


var settings = new OpenAIPromptExecutionSettings()
{
    Temperature = 1
};

var result = await chatCompletionService.GetChatMessageContentAsync(history, settings, kernel);
Console.WriteLine(result?.Content);