Tool calling with Gemini + structured_output

[ant31]

Please read this first

• Have you read the docs?Agents SDK docs
• Have you searched for related issues? Others may have faced similar issues.

Describe the bug

When using tool calling combined with output_type on a Agent using Gemini as model it fails with:

BadRequestError: Error code: 400 - [{'error': {'code': 400, 'message': "For controlled generation of only function calls (forced function calling), please set 'tool_config.function_calling_config.mode' field to ANY instead of populating 'response_mime_type' and 'response_schema' fields. For more details, see: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#tool-config", 'status': 'INVALID_ARGUMENT'}}]

When

• there's no tools it works
• output_type is str it works
• when both are set it fails

Debug information

• Agents SDK version: v0.0.4 and main
• Python 3.12

Repro steps

class MyContext:
    def __init__(self, user_name: str) -> None:
        self.user_name = user_name

@function_tool
async def get_user_name(ctx: RunContextWrapper[MyContext]) -> str:
    print(f"User id: {ctx.context.user_name}")
    return ctx.context.user_name

class GreetAnswer(BaseModel):
  user_name: str
  answers: list[str]



hiagent = Agent(
    name="HiAgent",
    instructions="Greet the user by it's name",
    model=OpenAIChatCompletionsModel(model="gemini-1.5-pro",
                                     openai_client=openai_aclient("gemini")),
    output_type=GreetAnswer,
    tools=[get_user_name]
)

res = (await Runner.run(hiagent, context=MyContext(user_name="ant31"), input="Hi")).final_output

Expected behavior

I expect to works the same than with openai models.

[rm-openai]

Looks like Gemini doesn't support structured outputs and tool calling together. I used this curl to test it:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GEMINI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "contents": {
    "role": "user",
    "parts": {
      "text": "Which theaters in Mountain View show Barbie movie?"
    }
  },
  "tools": [
    {
      "function_declarations": [
        {
          "name": "find_movies",
          "description": "find movie titles currently playing in theaters based on any description, genre, title words, etc.",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "description": {
                "type": "string",
                "description": "Any kind of description including category or genre, title words, attributes, etc."
              }
            },
            "required": [
              "description"
            ]
          }
        },
        {
          "name": "find_theaters",
          "description": "find theaters based on location and optionally movie title which is currently playing in theaters",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "movie": {
                "type": "string",
                "description": "Any movie title"
              }
            },
            "required": [
              "location"
            ]
          }
        },
        {
          "name": "get_showtimes",
          "description": "Find the start times for movies playing in a specific theater",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA or a zip code e.g. 95616"
              },
              "movie": {
                "type": "string",
                "description": "Any movie title"
              },
              "theater": {
                "type": "string",
                "description": "Name of the theater"
              },
              "date": {
                "type": "string",
                "description": "Date for requested showtime"
              }
            },
            "required": [
              "location",
              "movie",
              "theater",
              "date"
            ]
          }
        }
      ]
    }
  ],
  "generationConfig": {
    "response_mime_type": "application/json",
    "response_schema": {
      "type": "ARRAY",
      "items": {
        "type": "OBJECT",
        "properties": {
          "theater_name": {
            "type": "string",
            "description": "Name of the theater showing the movie"
          },
          "showtimes": {
            "type": "ARRAY",
            "items": {
              "type": "string"
            },
            "description": "List of showtimes for the movie"
          }
        },
        "required": [
          "theater_name",
          "showtimes"
        ]
      }
    }
  }
}'

Response:

{
  "error": {
    "code": 400,
    "message": "For controlled generation of only function calls (forced function calling), please set 'tool_config.function_calling_config.mode' field to ANY instead of populating 'response_mime_type' and 'response_schema' fields. For more details, see: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#tool-config",
    "status": "INVALID_ARGUMENT"
  }
}

[fghory]

Issue is still there...

[rm-openai]

@fghory the issue is with Gemini. Would recommend reaching out to them via support.