Allow Tool Calls that use returnDirect to return the result object without serialisation

[mands]

Currently, when doing tool calls, the response is a converted to JSON so it can be sent back to the model to process the result. This works for the normal tool calling flow, but is overly restrictive when using the return-direct functionality.

In some of my tool calls, I use returnDirect and create complex objects that are not easily serialisable. It would be great if we could return the actual object when "direct returning" the result of a tool call.

Expected Behavior

When performing a tool call with return direct, the result object is stored within the ToolResponse item contained within ToolResponseMessage.

e.g.

public record ToolResponse(
  String id, String name, String responseData, @Nullable Object responseObj
) {}

Would be happy to just use an Object and let the user cast as needed, or some generic way to implement this.

Current Behavior

Tool calls that return-direct must return an object that can be serialised to JSON, even if not practical.

Context

I've hit this within a chatbot I'm building that needs to integrate with a third-party. It's possible to create intermediate objects to delay creating the un-serialisable objects until I have control of the code-flow, but it's a lot of ceremony.

I've taken to using AtomicRefs within ToolContext in the meantime, but obviously it's not ideal!

[tzolov]

Hey @mands,
You can customize the ToolCallResultConverter and hopefully make it return what you want.
Would this work for you?

[ThomasVitale]

Returning Java objects directly from a tool call is not part of the current APIs. Each tool call results in a String that can be processed via a ToolCallResultConverter, as mentioned by @tzolov. But the output of the conversion logic would still need to be a String.

public interface ToolCallback {

	String call(String toolInput);

}

[mands]

Yep exactly, would great to see this as an addition to the tool-calling API going forwards, post 1.0.0. There is definitely a need for it and would simplify (my) code considerably!