Skip to content

旧式openai接口 /v1/chat/completions 是不是不支持 tool 中返回图片的base64多模态 #2313

Description

@DarkNoah

Problem

In the legacy OpenAI-compatible endpoint /v1/chat/completions, image content works correctly when it is sent in a normal user message.

However, if a tool message returns image content (for example structured array content containing image_url or file-based multimodal parts), the Codex path fails and the client receives:

empty_stream: upstream stream closed before first payload

Current behavior

  • user message with image input: works
  • tool message with image content: fails

This suggests the issue is in the translator path for tool-role content, not in general multimodal handling.

Expected behavior

Tool-role content should preserve multimodal payloads in the same way user-role content does, as long as the downstream Responses payload remains valid.

Suspected area

  • internal/translator/codex/openai/chat-completions/codex_openai_request.go

The translator likely needs to support structured tool content arrays and correctly map multimodal parts for function_call_output.output.

Required behavior

Please update the translator so that:

  • structured tool content arrays can preserve supported multimodal parts such as text, image_url, and valid file parts
  • input_image is emitted only when a real image source exists (url or file_id)
  • input_file is emitted only when a real file source exists (file_id, file_data, or file_url)
  • unsupported or incomplete parts are not silently dropped in a lossy way; they should fall back to raw text or serialized JSON
  • function_call_output.output should remain present even when the original tool content is not a string/array (for example null or other JSON values)

Reproduction hints

Observed symptom:

empty_stream: upstream stream closed before first payload

Repro pattern:

  1. send request to /v1/chat/completions
  2. let the model call a tool
  3. return tool content containing image data or multimodal structured content
  4. observe upstream stream closes before first payload

Regression coverage

Please add tests covering:

  • tool content with image_url
  • tool content with valid file parts
  • filename-only file parts should not emit invalid input_file
  • image parts without source should not emit invalid input_image
  • content: null
  • unknown tool content part types should not be silently dropped

Related context

  • Related PR attempt: #2349

Activity

  1. DarkNoah commented on Mar 25, 2026

    @DarkNoah
    Author
  2. DarkNoah commented on Apr 23, 2026

    @DarkNoah
    Author

    补充一下当前需要维护团队处理的目标:

    • 旧式 OpenAI 接口 /v1/chat/completions 下,tool-role 的多模态内容需要正确翻译到 Codex 请求链路
    • tool 返回 image_url / file 这类结构化内容时,不能再出现 empty_stream: upstream stream closed before first payload
    • 生成下游 payload 时,不能产生无有效来源的 input_image / input_file
    • 对未知或不完整的 part,应该保留为文本或序列化 JSON,而不是静默丢失

    相关尝试和上下文可参考:#2349

  3. added a commit that references this issue on May 3, 2026
    17be644
  4. added a commit that references this issue on May 9, 2026
    d43bec5
  5. added a commit that references this issue on May 27, 2026
    c38db8f
  6. added a commit that references this issue on Jun 3, 2026
    fc39114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions