feat(tools): add image caption fallback for FileReadTool when provide…

[kawayiYokami]

Motivation / 动机

EN: When the current chat provider does not support image input (modalities lacks image) and a vision/caption provider is configured via provider_settings.default_image_caption_provider_id, astrbot_file_read_tool previously returned a raw ImageContent that the chat model cannot consume. The intention is to automatically describe the image as text (image-to-text fallback) when a multimodal model is available.

中文： 当主对话模型不支持图片输入（modalities 未包含 image）且已在 provider_settings.default_image_caption_provider_id 配置了多模态/图转文模型时，astrbot_file_read_tool 原先会直接返回 ImageContent，导致主模型无法消费。期望行为是：自动调用已配置的多模态模型将图片转述为文本（图转文兜底），仅在同时满足“主模型不支持图片 + 已配置图转文模型”时触发。

Modifications / 改动点

EN:

• astrbot/core/tools/computer_tools/fs.py
  • Import mcp.types for CallToolResult / ImageContent detection.
  • Add _provider_supports_image(context) — resolves the active chat provider via ProviderManager.get_using_provider_async (fallback to get_using_provider), treats None / empty / non-list modalities as supporting image (backward-compatible, consistent with tool_loop_agent_runner).
  • Add _caption_image_fallback(context, image_ref) — reads provider_settings.default_image_caption_provider_id and image_caption_prompt, resolves the caption provider via get_provider_by_id, calls text_chat(prompt, image_urls=[image_ref]), returns [Image description]: {caption} or a clear error when not configured / unavailable / empty.
  • Update FileReadTool.call — after read_file_tool_result, if the result is an ImageContent and the active provider lacks image, builds a data:image/jpeg;base64,... URI from the already-compressed payload (so sandbox runtimes work without host file access) and delegates to the caption fallback; otherwise returns the original result. Text/PDF/docx paths are unaffected.

中文：

• astrbot/core/tools/computer_tools/fs.py

  • 引入 mcp.types 用于识别 CallToolResult / ImageContent。
  • 新增 _provider_supports_image(context) —— 通过 ProviderManager.get_using_provider_async（回退到 get_using_provider）解析当前会话的主对话模型，将 modalities 为 None / 空数组 / 非列表视为支持图片（向后兼容，与 tool_loop_agent_runner 一致）。
  • 新增 _caption_image_fallback(context, image_ref) —— 读取 provider_settings.default_image_caption_provider_id 和 image_caption_prompt，通过 get_provider_by_id 获取图转文模型，调用 text_chat(prompt, image_urls=[image_ref])，成功返回 [Image description]: {caption}，未配置/不可用/空返回时给出明确错误提示。
  • 修改 FileReadTool.call —— 在 read_file_tool_result 之后，若结果为 ImageContent 且主模型不支持 image，则从已压缩的图片数据构造 data:image/jpeg;base64,... URI（兼容 sandbox 容器内文件无法通过宿主机路径访问的场景）并走图转文兜底，否则原样返回；文本/PDF/docx 流程不受影响。

• This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

uv run ruff check astrbot/core/tools/computer_tools/fs.py  -> All checks passed
uv run ruff format --check                                  -> 1 file would be reformatted (fixed)
python -m py_compile astrbot/core/tools/computer_tools/fs.py -> OK

EN - Manual verification (7 cases, all passed):

• provider modalities=[text,image,tool_use] → returns ImageContent directly
• provider modalities=[text] + caption provider vision-1 configured → text_chat called with data:image/jpeg;base64,... + prompt, returns [Image description]: ...
• default_image_caption_provider_id="" → Error: ... no image caption provider is configured. Please set provider_settings.default_image_caption_provider_id.
• configured id not found → Error: ... is not available. Unable to read image file.
• modalities=[] or missing → treated as supporting image, returns ImageContent
• text file (hello.txt) → unaffected, returns plain text

中文 - 手动验证（7 项均通过）：

• modalities=[text,image,tool_use] → 直接返回 ImageContent
• modalities=[text] 且已配置 vision-1 → 以 data:image/jpeg;base64,... + 提示词调用 text_chat，返回 [Image description]: ...
• default_image_caption_provider_id="" → 提示未配置图转文模型
• 配置的 ID 不存在 → 提示模型不可用
• modalities=[] 或缺失 → 视为支持图片，直接返回 ImageContent
• 文本文件（hello.txt）→ 不受影响，原样返回文本

Existing suite against master (tests/test_computer_fs_tools.py -k "not test_sandbox_file_download_handles_windows_remote_filename"): 15 passed / 5 pre-existing Windows failures unchanged.

---

Checklist / 检查清单

• 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
  / 如果 PR 中有新加入的功能，已经通过 Issue / 邮件等方式和作者讨论过。

• 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
  / 我的更改经过了良好的测试，并已在上方提供了“验证步骤”和“运行截图”。

• 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
  / 我确保没有引入新依赖库，或者引入了新依赖库的同时将其添加到 requirements.txt 和 pyproject.toml 文件相应位置。

• 😮 My changes do not introduce malicious code.
  / 我的更改没有引入恶意代码。

Summary by Sourcery

Enable FileReadTool to provide text descriptions for images when the current agent model cannot consume image input.

New Features:

• Add automatic image-to-text fallback for file reads when the active model does not support image input and a caption provider is configured.

Bug Fixes:

• Prevent image file results from being unusable by text-only models while preserving the original image when captioning is unavailable.

Enhancements:

• Expose the selected provider's modalities through the agent run context so tools can use capabilities from the current turn.

Tests:

• Add coverage for image caption fallback, unavailable or empty caption providers, supported and unknown model capabilities, and modality propagation in the tool-loop runner.

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/tools/computer_tools/fs.py" line_range="347" />
<code_context>
+    return "image" in modalities
+
+
+async def _provider_supports_image(
+    context: ContextWrapper[AstrAgentContext],
+) -> bool:
</code_context>
<issue_to_address>
**Image handling uses wrong provider**

When the agent turn uses a provider different from the session manager’s currently resolved provider, such as after a fallback, an explicit provider selection, or a concurrent session switch, `_provider_supports_image` re-resolves the session provider instead of checking the provider selected for the agent turn, so `FileReadTool.call` can return raw `ImageContent` to a text-only provider or caption an image for a vision-capable one. The agent receives image content it cannot consume or loses raw image input.

Have `_provider_supports_image` inspect the provider selected for the agent turn rather than re-resolving the session’s current provider.

Also at `astrbot/core/tools/computer_tools/fs.py:365-368`, `astrbot/core/tools/computer_tools/fs.py:566`.
</issue_to_address>

### Comment 2
<location path="tests/test_computer_fs_tools.py" line_range="951-952" />
<code_context>
+    assert result == "[Image description]: a red rectangle"
+    caption_provider.text_chat.assert_awaited_once()
+    captioned = caption_provider.text_chat.await_args.kwargs["image_urls"]
+    assert captioned == [captioned[0]]
+    assert captioned[0].startswith("data:image/jpeg;base64,")
+
+
</code_context>
<issue_to_address>
**Empty image payload passes test**

When the fallback produces a URI with the expected prefix but no valid encoded image bytes, `test_file_read_tool_captions_image_when_provider_lacks_image_modality` passes with an empty or truncated base64 payload: line 951 only checks that `image_urls` has one entry, line 952 checks only its prefix, and the mocked `text_chat` returns its caption regardless of the image. The test therefore passes even though the caption provider receives no usable image.

Assert that the URI payload decodes to non-empty image bytes, for example by checking its JPEG signature.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and when the active provider lacks image support, image data read from local files is automatically sent to a separately configured caption provider, so an incorrect provider choice or routing decision could expose sensitive file contents externally. Reverting stops future transfers but cannot undo images that were already sent.

Blocking findings: astrbot/core/tools/computer_tools/fs.py:347, tests/test_computer_fs_tools.py:952

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook