agent that both proposes and executes tools

[sonichi]

I like the concept of Autogen, I like to use a couple of features of it. But I currently just need a simple tool executor.

I notice that Autogen heavily works on the concept of generating code and executing it. Does it not has an simple way of executing code directly after it has been "selected"? - So basically same like OpenAI Function calling? Does it all use function calling?

I ask that, because for my use case it's sometimes enough to just select the correct tool. No need of extra checks etc.

Originally posted by @WebsheetPlugin in #2208

Suggestion: Create a reply function `propose_and_execute_tools_nested_reply" which uses a nested chat between an AssistantAgent and a UserProxyAgent with human_input_mode="NEVER".

cc @qingyun-wu

[shippy]

Doesn't this work already? I frequently use register_function with the same agent assigned to both caller and executor; the latter is a little misleadingly named, because such agent doesn't even need a code execution configuration to use said tool.

[GeorgSatyros]

It works in an unorthodox way. In your case the same agent would be selected to speak twice, given that it is the only one with access to the tool in question. So assuming the default groupchat flexibility in agent selection, it works. The moment you start dictating agent order, though, it will fail. I have implemented a workaround to this issue with Society Of Mind agents (a single agent that is composed of multiple under the hood). One SoM agent is called that under the hood calls a tool caller and then a tool executor, returning the result. I could make a PR with it, but I feel a more integrated solution to this would be preferable as SoM is experimental.

[sonichi]

@shippy it works in group chat while it takes two messages in the group chat for tool proposal and tool execution. @WebsheetPlugin probably wants to encapsulate tool proposal and execution inside one agent's inner conversation and only returns a single message of the result to the outer chat.

@GeorgSatyros thanks for sharing your experience. SoM Agent is experimental, while nested chat is in the core library. I suggest using nested chat to implement a new agent with the same functionality: https://microsoft.github.io/autogen/docs/tutorial/conversation-patterns#nested-chats
It should be very easy to use a single register_nested_chat to implement it. Would you like to give it a try?

[shippy]

Odd: pretty sure all my use cases have been in multi-agent chats with defined transitions between agents, where the tool-executing agent wasn't allowed to speak twice (I think - I'll double-check)

[GeorgSatyros]

Odd: pretty sure all my use cases have been in multi-agent chats with defined transitions between agents, where the tool-executing agent wasn't allowed to speak twice (I think - I'll double-check)

@shippy I could have definitely elaborated more above, so let me fix that!
Basically, assuming you are using the speaker_selection_method parameter in groupchats and not overriding/shadowing the speaker_selection method itself, then the system should work.
But the core of the problem is this: The system will actually, under the hood, not follow your defined transitions. If you want agent_A->tool_agent_B->agent_C, the actual order will be agent_A->tool_agent_B->tool_agent_B->agent_C. This could be a non-issue for many projects that have looser requirements on agent order, but for other projects it will be a problem. It is also a relatively unintuitive and inexplicit pattern, as is evident on how one may disallow agents speaking twice through the allow_repeat_speaker flag and the system will ignore that when it comes to tool execution. Here's the relevant code snippet for the above as reference: snippet

@sonichi Sure, I was looking for an excuse to dive deeper into nested chats anyway! Is there any planned support for SoM going forward or will the "agent composed of agents" niche be fulfilled by nested chats? Asking because I was considering contributing to SoM and that effort may be better spent on the core component instead.

[ChristianWeyer]

@shippy it works in group chat while it takes two messages in the group chat for tool proposal and tool execution. @WebsheetPlugin probably wants to encapsulate tool proposal and execution inside one agent's inner conversation and only returns a single message of the result to the outer chat.

@GeorgSatyros thanks for sharing your experience. SoM Agent is experimental, while nested chat is in the core library. I suggest using nested chat to implement a new agent with the same functionality: https://microsoft.github.io/autogen/docs/tutorial/conversation-patterns#nested-chats It should be very easy to use a single register_nested_chat to implement it. Would you like to give it a try?

Would be great to have a sample for this 👍🏼.

[ChristianWeyer]

@shippy it works in group chat while it takes two messages in the group chat for tool proposal and tool execution. @WebsheetPlugin probably wants to encapsulate tool proposal and execution inside one agent's inner conversation and only returns a single message of the result to the outer chat.
@GeorgSatyros thanks for sharing your experience. SoM Agent is experimental, while nested chat is in the core library. I suggest using nested chat to implement a new agent with the same functionality: https://microsoft.github.io/autogen/docs/tutorial/conversation-patterns#nested-chats It should be very easy to use a single register_nested_chat to implement it. Would you like to give it a try?

Would be great to have a sample for this 👍🏼.

I guess we already have it ☺️
https://microsoft.github.io/autogen/docs/notebooks/agentchat_nested_chats_chess

[sonichi]

@GeorgSatyros it'll be great if you could make a PR to reimplement the SoM agent using nested chat. It'll be easier to maintain. The current SoM agent can retire after feature parity.

[GeorgSatyros]

@sonichi I agree, that would be a more graceful solution than deprecation. Will be opening a PR with a solution as soon as my schedule allows!

[WebsheetPlugin]

Odd: pretty sure all my use cases have been in multi-agent chats with defined transitions between agents, where the tool-executing agent wasn't allowed to speak twice (I think - I'll double-check)

@shippy I could have definitely elaborated more above, so let me fix that! Basically, assuming you are using the speaker_selection_method parameter in group chats and not overriding/shadowing the speaker_selection method itself, then the system should work. But the core of the problem is this: The system will actually, under the hood, not follow your defined transitions. If you want agent_A->tool_agent_B->agent_C, the actual order will be agent_A->tool_agent_B->tool_agent_B->agent_C. This could be a non-issue for many projects that have looser requirements on agent order, but for other projects, it will be a problem. It is also a relatively unintuitive and inexplicit pattern, as is evident in how one may disallow agents speaking twice through the allow_repeat_speaker flag and the system will ignore that when it comes to tool execution. Here's the relevant code snippet for the above as reference: snippet

@sonichi Sure, I was looking for an excuse to dive deeper into nested chats anyway! Is there any planned support for SoM going forward or will the "agent composed of agents" niche be fulfilled by nested chats? Asking because I was considering contributing to SoM and that effort may be better spent on the core component instead.

Yes, I use speaker selection to decide the order of Agents.
And for me, it seemed unintuitive to pass a round here just to execute the tool by the same agent again. As I had some logic to already "select" the next agent in order. So my solution was that if the last message is tool selection just allow the same agent again, which had the tool attached.

But again, for me, all this seemed unintuitive. And it's still not clear to me why another agent should execute the tool. Or why it should not be executed if selected.

Is there a case where you want to select a tool by Agent A and execute it by Agent B or C or not execute it at all?

[sonichi]

Odd: pretty sure all my use cases have been in multi-agent chats with defined transitions between agents, where the tool-executing agent wasn't allowed to speak twice (I think - I'll double-check)

@shippy I could have definitely elaborated more above, so let me fix that! Basically, assuming you are using the speaker_selection_method parameter in group chats and not overriding/shadowing the speaker_selection method itself, then the system should work. But the core of the problem is this: The system will actually, under the hood, not follow your defined transitions. If you want agent_A->tool_agent_B->agent_C, the actual order will be agent_A->tool_agent_B->tool_agent_B->agent_C. This could be a non-issue for many projects that have looser requirements on agent order, but for other projects, it will be a problem. It is also a relatively unintuitive and inexplicit pattern, as is evident in how one may disallow agents speaking twice through the allow_repeat_speaker flag and the system will ignore that when it comes to tool execution. Here's the relevant code snippet for the above as reference: snippet
@sonichi Sure, I was looking for an excuse to dive deeper into nested chats anyway! Is there any planned support for SoM going forward or will the "agent composed of agents" niche be fulfilled by nested chats? Asking because I was considering contributing to SoM and that effort may be better spent on the core component instead.

Yes, I use speaker selection to decide the order of Agents. And for me, it seemed unintuitive to pass a round here just to execute the tool by the same agent again. As I had some logic to already "select" the next agent in order. So my solution was that if the last message is tool selection just allow the same agent again, which had the tool attached.

But again, for me, all this seemed unintuitive. And it's still not clear to me why another agent should execute the tool. Or why it should not be executed if selected.

Is there a case where you want to select a tool by Agent A and execute it by Agent B or C or not execute it at all?

For example, it makes it possible for Agent B to perform extra conversations with other agents or humans before executing.

[WebsheetPlugin]

Ahhh, now I got it. I am acutaly counting each token twice :) so this would be not my use case, but finaly I understand it. Makes sense.

Just an observation. I feel that Autogen is pivoted for use cases where many tokens are beings used, and it's kinda leaving out more simple use cases like the one mentioned above.