Description
Currently, providers that expose a tool role for messages (OpenAI, Gemini, Ollama) do not support images in the tool result.
Conversely, Anthropic has no tool role and requires tool results to be included in a user message, which can contain images. Since multimodal support in Amazon Bedrock is mostly based on Anthropic, Bedrock supports this use case as well. For a Bedrock example, see deepset-ai/haystack-experimental#307 (comment).
Supporting this use case would require refactoring our ToolCallResult dataclass.
If more model providers begin to allow this pattern, we should investigate and evaluate this refactoring.
This doesn't currently seem like a high priority, because a simple workaround is to create a user message with the image returned by the tool.