bug: Semantic Kernel Adapter for Anthropic models with tool calls

[ekzhu]

import asyncio
import os
from dotenv import load_dotenv

load_dotenv()
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.ui import Console
from autogen_core.models import ModelFamily, UserMessage
from autogen_ext.models.semantic_kernel import SKChatCompletionAdapter
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.anthropic import AnthropicChatCompletion, AnthropicChatPromptExecutionSettings
from semantic_kernel.memory.null_memory import NullMemory


async def get_weather(city: str) -> str:
    """Get the weather for a city."""
    return f"The weather in {city} is 75 degrees."


async def main() -> None:
    sk_client = AnthropicChatCompletion(
        # ai_model_id="claude-3-5-sonnet-20241022",
        ai_model_id="claude-3-haiku-20240307",
        api_key=os.environ["ANTHROPIC_API_KEY"],
        service_id="my-service-id",  # Optional; for targeting specific services within Semantic Kernel
    )
    settings = AnthropicChatPromptExecutionSettings(
        temperature=0.2,
    )

    model_client = SKChatCompletionAdapter(
        sk_client,
        kernel=Kernel(memory=NullMemory()),
        prompt_settings=settings,
        model_info={
            "function_calling": True,
            "json_output": True,
            "vision": True,
            "family": ModelFamily.CLAUDE_3_5_SONNET,
        },
    )

    # Call the model directly.
    response = await model_client.create([UserMessage(content="What is the capital of France?", source="test")])
    print(response)

    # Create an assistant agent with the model client.
    assistant = AssistantAgent(
        "assistant", model_client=model_client, system_message="You are a helpful assistant.", tools=[get_weather],
        reflect_on_tool_use=True,
        model_client_stream=False,
    )
    # Call the assistant with a task.
    await Console(assistant.run_stream(task="What is the weather in Paris and London?"))



if __name__ == "__main__":
    asyncio.run(main())

This is the example, I am trying.

error trace:

finish_reason='stop' content='The capital of France is Paris.' usage=RequestUsage(prompt_tokens=0, completion_tokens=0) cached=False logprobs=None thought=None
---------- user ----------
What is the weather in Paris and London?
---------- assistant ----------
[FunctionCall(id='toolu_017CvSHZsqnf8repjmYZZysM', arguments='{"city": "Paris"}', name='get_weather')]
---------- assistant ----------
[FunctionExecutionResult(content='The weather in Paris is 75 degrees.', call_id='toolu_017CvSHZsqnf8repjmYZZysM', is_error=False)]
Unsupported item type in Tool message while formatting chat history for Anthropic: <class 'semantic_kernel.contents.text_content.TextContent'>
Traceback (most recent call last):
  File ".venv-3.13/lib/python3.13/site-packages/semantic_kernel/connectors/ai/anthropic/services/anthropic_chat_completion.py", line 328, in _send_chat_request
    response = await self.async_client.messages.create(**settings.prepare_settings_dict())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/anthropic/resources/messages/messages.py", line 2054, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
    ...<24 lines>...
    )
    ^
  File ".venv-3.13/lib/python3.13/site-packages/anthropic/_base_client.py", line 1855, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/anthropic/_base_client.py", line 1549, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File ".venv-3.13/lib/python3.13/site-packages/anthropic/_base_client.py", line 1650, in _request
    raise self._make_status_error_from_response(err.response) from None
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.2: all messages must have non-empty content except for the optional final assistant message'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1570, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/anthropicTest.py", line 59, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "src/anthropicTest.py", line 54, in main
    await Console(assistant.run_stream(task="What is the weather in Paris and London?"))
  File ".venv-3.13/lib/python3.13/site-packages/autogen_agentchat/ui/_console.py", line 117, in Console
    async for message in stream:
    ...<74 lines>...
                    total_usage.prompt_tokens += message.models_usage.prompt_tokens
  File ".venv-3.13/lib/python3.13/site-packages/autogen_agentchat/agents/_base_chat_agent.py", line 176, in run_stream
    async for message in self.on_messages_stream(input_messages, cancellation_token):
    ...<9 lines>...
            output_messages.append(message)
  File ".venv-3.13/lib/python3.13/site-packages/autogen_agentchat/agents/_assistant_agent.py", line 512, in on_messages_stream
    reflection_model_result = await self._model_client.create(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        llm_messages, cancellation_token=cancellation_token
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File ".venv-3.13/lib/python3.13/site-packages/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py", line 424, in create
    result = await self._sk_client.get_chat_message_contents(chat_history, settings=settings, kernel=kernel)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/semantic_kernel/connectors/ai/chat_completion_client_base.py", line 134, in get_chat_message_contents
    return await self._inner_get_chat_message_contents(chat_history, settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/semantic_kernel/utils/telemetry/model_diagnostics/decorators.py", line 112, in wrapper_decorator
    return await completion_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/semantic_kernel/connectors/ai/anthropic/services/anthropic_chat_completion.py", line 167, in _inner_get_chat_message_contents
    return await self._send_chat_request(settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv-3.13/lib/python3.13/site-packages/semantic_kernel/connectors/ai/anthropic/services/anthropic_chat_completion.py", line 330, in _send_chat_request
    raise ServiceResponseException(
    ...<2 lines>...
    ) from ex
semantic_kernel.exceptions.service_exceptions.ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.anthropic.services.anthropic_chat_completion.AnthropicChatCompletion'> service failed to complete the request", BadRequestError("Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.2: all messages must have non-empty content except for the optional final assistant message'}}"))
python-BaseException
Exception ignored in: <function _DeleteDummyThreadOnDel.__del__ at 0x104df16c0>
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydevd_bundle/pydevd_pep_669_tracing.py", line 635, in py_raise_callback
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/threading.py", line 1435, in current_thread
TypeError: 'NoneType' object is not subscriptable

Process finished with exit code 1
``` 

Originally posted by @hariharan1st in #2164

[ekzhu]

@lspinheiro this may be fixed by including the thought field in AssistantMessage introduced in #5500, which only fixed for OpenAI client. We may need to add this to Semantic kernel model adapter

[lspinheiro]

Looking into it.

[lspinheiro]

Some update on this. I do have some updates to push on how the messages are processed, but it does end up in another exception because anthropic requires the tool definitions to be sent on every request that includes tool call or result, but they are not submitted in the AssistantAgent reflection call. See the discussion here: BerriAI/litellm#5388 and here BerriAI/litellm#5747

And our code for reflection:

autogen/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py

Line 988 in 173e649

async for chunk in model_client.create_stream(llm_messages):

This results in the anthropic api exception semantic_kernel.exceptions.service_exceptions.ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.anthropic.services.anthropic_chat_completion.AnthropicChatCompletion'> service failed to complete the request", BadRequestError("Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Requests which include tool_useortool_result blocks must define tools.'}}")).

The only way to fix this is by adding the tools to the create message during reflection, but I believe they are left out on purpose to avoid having the model try to call the tools again. What I have tried and seems to help is to add the tools and also add a message on top of the existing messages asking for the reflection. Something like:

        if self._reflect_on_tool_use:
            # Generate another inference result based on the tool call and result.
            llm_messages = self._get_compatible_context(
                self._system_messages + await self._model_context.get_messages()
            )
            # Add a message asking the LLM to reflect on the tool call results
            llm_messages.append(
                UserMessage(
                    content="Explain the tool call results in a concise manner.",
                    source="user"
                )
            )
            reflection_model_result: CreateResult | None = None
            if self._model_client_stream:
                # Stream the model client.
                async for chunk in self._model_client.create_stream(
                    llm_messages, cancellation_token=cancellation_token
                ):
                    if isinstance(chunk, CreateResult):
                        reflection_model_result = chunk
                    elif isinstance(chunk, str):
                        yield ModelClientStreamingChunkEvent(content=chunk, source=self.name)
                    else:
                        raise RuntimeError(f"Invalid chunk type: {type(chunk)}")
                assert isinstance(reflection_model_result, CreateResult)
            else:
                reflection_model_result = await self._model_client.create(
                    llm_messages, cancellation_token=cancellation_token, tools=self._tools + self._handoff_tools
                )
                breakpoint()

But it does have the risk of not working depending on which model is being used. Thoughts? I'm curious if the same error happens in the PR @victordibia is working on.

[ekzhu]

Thanks for the investigation. The native Anthropic client has no problem handling this:

autogen/python/packages/autogen-ext/tests/models/test_anthropic_model_client.py

Lines 152 to 153 in 98aeee0

	after_tool_result = await client.create(messages=messages)

This test peforms the reflect step without passing in tools, and it passes.

[ekzhu]

Any kind of model-specific speciali handling we should use mode_info.family to help us determine.

[lspinheiro]

It seems that for the anthropic client we are adding the tools under the hood by adding the last_used_tools. It is a bit of a hack but we could do something similar for SK, I just don't know what would be the side effects or maybe we should do it more explicitly in the assistant agent by using the model family. What do you think?

# python\packages\autogen-ext\src\autogen_ext\models\anthropic\_anthropic_client.py
def create(
...
        # Add system message if present
        if system_message is not None:
            request_args["system"] = system_message

        has_tool_results = any(isinstance(msg, FunctionExecutionResultMessage) for msg in messages)

        # Store and add tools if present
        if len(tools) > 0:
            converted_tools = convert_tools(tools)
            self._last_used_tools = converted_tools
            request_args["tools"] = converted_tools
        elif has_tool_results:
            # anthropic requires tools to be present even if there is any tool use
            request_args["tools"] = self._last_used_tools

[victordibia]

Yes, we keep track of last used tools and then append that to subsequent queries.
We might need more testing to fully identify side effects (I am not sure if the model ends up getting distracted by the ever-present set of tools in the in the request).

I am also not sure about how much model client specific stuff we want to have in assistant_agent.

There is also the issue with reflect_on_tool_use where the anthropic model seems to respond with an additional tool call instead of a string content. Might need more investigation there.

[ekzhu]

@lspinheiro @victordibia I think it is possible in the future Anthropic will support tool_choice=None just like OpenAI, or support not requiring the tools field when there is tool usage in the message history.

Before that happens, in AssistantAgent we can use model_info of the model client to determine if the model belongs to the Claude family. In that case, we raise an exception when reflect_on_tool_use=True, in the constructor.

So basically an AssistantAgent won't support reflection on tool usage when the model is Claude.

Create a new issue for this task: #5731