Skip to content

Fix bug where empty messages might be created in the Anthropic model #1027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 27, 2025

Conversation

oscar-broman
Copy link
Contributor

This PR fixes an issue where empty messages are created.

The error happens when the function _map_message runs with this input:

[
    ModelRequest(parts=[SystemPromptPart(content='<my system prompt>', dynamic_ref=None, part_kind='system-prompt')], kind='request'),
    ModelResponse(parts=[TextPart(content='<model response>', part_kind='text')], model_name=None, timestamp=datetime.datetime(2025, 3, 2, 7, 10, 12, 560121, tzinfo=datetime.timezone.utc), kind='response'),
    ModelRequest(parts=[UserPromptPart(content="<user message>", timestamp=datetime.datetime(2025, 3, 2, 7, 10, 12, 566706, tzinfo=datetime.timezone.utc), part_kind='user-prompt')], kind='request'),
]

The following error is eventually returned from the Anthropic API:

{'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0: all messages must have non-empty content except for the optional final assistant message'}

@Kludex
Copy link
Member

Kludex commented Mar 2, 2025

Can you provide an MRE, please?

@oscar-broman
Copy link
Contributor Author

from anthropic import AsyncAnthropic
from openai import AsyncOpenAI
from pydantic_ai import Agent
from pydantic_ai.messages import ModelRequest, SystemPromptPart, TextPart, ModelResponse, UserPromptPart
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.openai import OpenAIModel

async def test_anthropic_issue() -> None:
    openai_key = ""
    anthropic_key = ""

    models = [
        ("OpenAI", OpenAIModel(model_name="gpt-4o-mini", openai_client=AsyncOpenAI(api_key=openai_key))),
        ("Anthropic", AnthropicModel(model_name="claude-3-7-sonnet-latest", anthropic_client=AsyncAnthropic(api_key=anthropic_key))),
    ]


    for name, model in models:
        print(f"\n\nModel: {name}")

        agent = Agent(
            model=model,
        )

        messages = [
            ModelRequest(parts=[SystemPromptPart("Respond with the user's last message in upper-case")]),
            # ModelRequest(parts=[UserPromptPart(content="Hello")]), <-- Issue does not happen if uncommented
            ModelResponse(parts=[TextPart(content="HELLO")]),
        ]

        await agent.run(
            user_prompt="Test",
            message_history=messages,
            result_type=str,
        )

@alexmojaki
Copy link
Contributor

Please convert the MRE into a test

@oscar-broman
Copy link
Contributor Author

I've tried to do this without any other changes in the codebase -- it's quite hard. The function is an internal function not accessible without some indirections.

One way would be to create a live test, but this seems a bit extreme perhaps.

@alexmojaki What would you suggest I do?

@alexmojaki alexmojaki requested a review from Kludex March 4, 2025 08:26
@alexmojaki
Copy link
Contributor

OK then I think it's fine unless @Kludex has some ideas.

@Kludex
Copy link
Member

Kludex commented Mar 4, 2025

I'll create a test with VCR.

@DouweM DouweM marked this pull request as draft April 30, 2025 21:28
@mike-luabase
Copy link
Contributor

@Kludex I'm hitting this error on most of our attempts to use Claude 3.7

@Kludex Kludex marked this pull request as ready for review May 27, 2025 15:48
@Kludex Kludex enabled auto-merge (squash) May 27, 2025 15:48
@Kludex Kludex merged commit 3b12530 into pydantic:main May 27, 2025
16 checks passed
Kludex added a commit that referenced this pull request May 30, 2025
Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants