Skip to content

[Feature] Let the vision-sidecar description model be any configured provider's vision model (e.g. Doubao/volcengine) instead of only OpenAI/Anthropic forward models #1937

Closed as not planned
Closed as not planned
[Feature] Let the vision-sidecar description model be any configured provider's vision model (e.g. Doubao/volcengine) instead of only OpenAI/Anthropic forward models#1937
Labels
enhancementNew feature or requesthelp wantedExtra attention is neededneeds-design설계 필요. Implementation waits on a written design.providerProvider adapters, OpenAI-compat presets, upstream API quirksroadmapAccepted long-term work item; needs its own project cycle

Description

@ShuaiHui
ShuaiHui
opened on Aug 17, 2026

Area

Provider adapter

What are you trying to accomplish?

I use several text-only routed models through the vision sidecar, which describes images before feeding them to the text-only model. Right now my only choice for the "describer" is an OpenAI-native or Anthropic model that runs through the ChatGPT forward backend. I'd like to be able to pick any configured provider's vision model — for instance a Doubao/volcengine vision model I already have configured in opencodex — as the describer.

What prevents this today?

The vision sidecar (visionSidecar.model) sends the describe request through forwardProvider (the ChatGPT forward backend) to /responses, so model is effectively limited to OpenAI-native (or Anthropic) model ids. A model hosted on a custom provider (cliproxy, volcengine/doubao, etc.) can't be selected because the describe path doesn't know how to reach that provider.

What should OpenCodex do?

Allow the vision-sidecar description model to be resolved against any configured provider, not just the forward/ChatGPT backend — for example, per-provider or a routed model id for the describer, so a Doubao/volcengine (or any other) vision-capable model can act as the image describer for text-only routed models. This mirrors how noVisionModels lets operators choose which models get the sidecar; the companion ask is letting them choose who does the describing.

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. github-actions
    added
    enhancementNew feature or request
    providerProvider adapters, OpenAI-compat presets, upstream API quirks
    on Aug 17, 2026

  2. Ingwannu commented on Aug 18, 2026

    @Ingwannu
    Ingwannu
    on Aug 18, 2026
    Owner

    This is a reasonable roadmap direction, but it is not only a provider selector. The describer call would need to re-enter normal routing without recursively triggering the vision sidecar, while preserving the selected provider's auth, cancellation, usage attribution, image limits, and failure policy.

    A safe implementation contract is:

    • configure an explicit routed provider/model describer, defaulting to the current forward path for compatibility;
    • require verified image input capability for that route and fail closed if it resolves to a text-only model;
    • set an internal no-recursion marker so the describer can never invoke the vision sidecar again;
    • keep the original user's image and credentials out of logs/persistence, and account the nested call separately;
    • add tests for disabled provider, missing credential, cancellation, recursion prevention, and custom-destination isolation.

    Marking this as accepted roadmap work rather than treating every configured model as automatically eligible.

  3. Ingwannu
    added
    roadmapAccepted long-term work item; needs its own project cycle
    on Aug 18, 2026

  4. lidge-jun commented on Aug 19, 2026

    @lidge-jun
    lidge-jun
    on Aug 19, 2026
    Owner

    리뷰 · 우선순위 22 / 80

    비전 사이드카 설명 모델을 OpenAI/Anthropic forward만이 아니라, 이미 설정한 Doubao/volcengine 같은 비전 모델로 고르고 싶다. 텍스트 전용 라우티드 모델 앞에 이미지를 설명하는 경로다. 로드맵/옵트인 프로바이더 요청이라 22다. 기본 native 설치를 깨지 않는다.

    지금 실행기는 두 개다. src/vision/describe.ts의 describeImage()는 forwardProvider.baseUrl/responses로 보낸다. src/vision/anthropic-describe.ts의 describeImageAnthropic()는 Anthropic Messages다. src/vision/index.ts의 VisionPlan.backend은 "openai" | "anthropic"뿐이다. resolveEffectiveVisionModel()도 이 두 백엔드의 기본 모델만 고른다.

    설명자 자격은 src/vision/eligibility.ts의 visionBackendForCandidate()다. native/provider === "openai"면 openai, 고른 Anthropic OAuth 행과 이름이 같으면 anthropic, 아니면 undefined다. 키 인증 Anthropic도 실행 불가라 고르지 않는다. Doubao/volcengine 행은 여기서 탈락한다. noVisionModels는 “누구를 위해 설명할지”이고, “누가 설명할지”가 아니다.

    사이드카를 임의 어댑터로 열려면 새 실행기가 필요하다. openai-chat/volcengine 경로를 있다고 지어 호출하지 않는다. 이미지 바이트와 프롬프트가 그 프로바이더 키로 나간다. 기본 off, 명시 모델 id가 안전하다.

    해결방안

    visionSidecar.model을 카탈로그의 라우티드 비전 행으로 해석하게 하려면, VisionPlan.backend 유니온과 visionBackendForCandidate()를 먼저 넓혀야 한다. 그 전에 Doubao용 describe 함수를 추측하지 않는다. 패치는 추측하지 않음. 당장은 설명자를 OpenAI native 또는 Anthropic OAuth 비전 모델로 두고, noVisionModels로 텍스트 전용 대상만 고른다. 새 백엔드를 넣을 때도 프롬프트/이미지를 로그에 남기지 않는다.

    이 댓글은 grok-bot이 작성했습니다

  5. lidge-jun
    added
    help wantedExtra attention is needed
    needs-design설계 필요. Implementation waits on a written design.
    on Aug 19, 2026

  6. lidge-jun commented on Aug 19, 2026

    @lidge-jun
    lidge-jun
    on Aug 19, 2026
    Owner

    설계 필요 + help wanted로 열어둠. JUN이 설계/구현할 이슈임. 누가 PR 내도 됨. 설명자를 아무 설정된 비전 모델로 고르는 쪽만. #2031이 대신 안 해줌.

    관련 묶음:

    • 같은 요청 이전분: #1276 (닫힘. 비전+웹서치 백엔드를 OpenAI/Anthropic 밖으로)
    • 소비자 쪽 (누가 사이드카를 타느냐): #1927 아직 열림, #2031은 머지됨 (ingw/fix-mimo-vision-1927)
    • 설명자 자격 필터 이미 들어간 것: #1326 #1327 #1328
    • 웹서치 형제: #1616 (Zen 웹서치 사이드카, 열림)
    • 시도 중인 PR: #1645 draft pr/vision-chat-sidecar (chat/Google 사이드카). 스플릿 이후라 리베이스보다 닫고 다시 자르는 편이 맞음
    • 브랜치: codex/external-vision, waw4303:pr/vision-chat-sidecar

    이 댓글은 grok-bot이 작성했습니다

  7. lidge-jun commented on Aug 20, 2026

    @lidge-jun
    lidge-jun
    on Aug 20, 2026
    Owner

    이 이슈는 #2188로 합침. 백엔드를 하나씩 더하는 대신, 전역 auth 슬롯 + Codex 피커 거름망 + (웹서치는 라이브 프로브)로 선택 규칙을 먼저 고친다. 후속 구현은 #2188을 보면 됨.

  8. lidge-jun
    closed this as not plannedon Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededneeds-design설계 필요. Implementation waits on a written design.providerProvider adapters, OpenAI-compat presets, upstream API quirksroadmapAccepted long-term work item; needs its own project cycle

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions