Description
Bug description
When attempting to use a tools along with streaming, I am using a java.util.function.Supplier as my FunctionCallback (vs a java.util.function.Function); I am using BedrockProxyChatModel as my ChatModel.
The initial bedrock LLM response correctly selects my Supplier as the tool choice, however to tool is never called by Spring AI. This appears to only happen with Suppliers; java.util.function.Function works as expected.
While debugging, the issue a manifested in the isToolCall of the org.springframework.ai.chat.model.AbstractToolCallSupport class when after the following debug message.
DEBUG 34272 --- [tyEventLoop-2-2] o.s.a.b.converse.BedrockProxyChatModel : Received converse stream output:MessageStopEvent(StopReason=tool_use)
The chatResponse contains no generations, therefor the tool is not executed.
protected boolean isToolCall(ChatResponse chatResponse, Set<String> toolCallFinishReasons) {
Assert.isTrue(!CollectionUtils.isEmpty(toolCallFinishReasons), "Tool call finish reasons cannot be empty!");
if (chatResponse == null) {
return false;
}
var generations = chatResponse.getResults();
if (CollectionUtils.isEmpty(generations)) {
return false;
}
return generations.stream().anyMatch(g -> isToolCall(g, toolCallFinishReasons));
}
Environment
This was encountered using spring-ai M4 along with Java 21.
Steps to reproduce
The following is an excerpt from a trivial SpringBoot application that can reproduce the problem when Spring AI is configured use the Bedrock/Converse chat models. It includes a very simple System.out to view the output.
@Bean
public CommandLineRunner commandLineRunner(ChatClient chatClient) {
return (args) -> {
FunctionCallback.Builder builder =
FunctionCallback.builder();
FunctionCallback functionCallBack = builder.description("Get the weather")
.function("weather", new WeatherService())
.build();
ChatClientRequestSpec spec = chatClient.prompt();
spec.functions(functionCallBack).user("What's the weather like").stream().chatResponse()
.doOnNext(resp -> {
if (resp.getResult() != null)
System.out.println(resp.getResult().getOutput().getContent());
}).collectList().block();
};
}
public static class WeatherService implements Supplier<WeatherService.Response> {
public record Response(double temp) {}
public Response get() {
return new Response(30.0);
}
}
Expected behavior
I expect an output something similar to "The current weather is 30 degrees".
Minimal Complete Reproducible example
See the steps to reproduce for a code sample.