[Enhancement]: Determine whether an endpoint and/or model is vision capable via config

[danny-avila]

What features would you like to see added?

Related issues:

• [Bug]: MCP Image Generation Tools: Generated images sent to non-vision models causing errors #11413
• Image Upload Option Shown for Non-Vision Models #11333
• [Enhancement]: Custom endpoint config. to allow processing image outputs from MCP as attachments #10996
• [Bug]: Flux image generation tool compatibility with Claude 3.7 on AWS Bedrock #6255
• Enhancement: Link to image (image_url) for vision-capable endpoints rather than b64 #2774
• Enhancement: Select Vision Model from Client or Config file for Custom Endpoint #1634

More details

Most relevant:

• Enhancement: Custom Token Rates for Endpoints (including Custom) #1633

[JumpLink]

Hi @danny-avila,

We've been working on implementing vision capability detection and discovered it requires more changes than initially expected, with potential side effects.

Problem

Currently, certain providers (OpenRouter, XAI, DeepSeek) are handled specially, using string format instead of content array format. This prevents MCP tool artifacts (e.g., generated images) from being sent back to the LLM, even when the model supports vision.

Proposed Solution

Treat all OpenAI-compatible providers the same (like OpenAI itself) and determine compatibility via NON_OPENAI_PROVIDERS. Unknown providers are treated as OpenAI-compatible, which works for many custom endpoints (Scaleway, Together, Perplexity, etc.) and new providers without code changes.

1. Simplified Provider Detection

// packages/api/src/mcp/parsers.ts
const NON_OPENAI_PROVIDERS = new Set(['google', 'anthropic', 'bedrock', 'ollama']);

function usesContentArrayFormat(provider: t.Provider): boolean {
  // All providers NOT in NON_OPENAI_PROVIDERS 
  // are treated as OpenAI-compatible and use array format
  if (CONTENT_ARRAY_PROVIDERS.has(provider)) {
    return true;
  }
  
  // Removed: Special handling for openrouter, xai, deepseek
  // if (['openrouter', 'xai', 'deepseek'].includes(provider)) {
  //   return false;
  // }
  
  // All unknown providers = OpenAI-compatible = Array format
  if (!NON_OPENAI_PROVIDERS.has(provider)) {
    return true;
  }
  
  return false;
}

2. Vision Capability via Config

# librechat.yaml
modelSpecs:
  list:
    - name: "openai-gpt5-2"
      vision: true  # Explicitly enable vision
      preset:
        endpoint: "OpenRouter"
        model: "openai/gpt-5.2"
    - name: "deepseek-chat"
      vision: false  # Explicitly disable vision
      preset:
        endpoint: "OpenRouter"
        model: "deepseek/deepseek-chat"

3. Vision Filtering in Agents

// dev/agents/src/messages/core.ts
export function formatArtifactPayload(
  messages: BaseMessage[],
  isVisionModel: boolean = true
): void {
  // ...
  const artifactContent = isVisionModel
    ? msg.artifact.content
    : msg.artifact.content.filter(
        (item) => !isBase64ImageUrl(item)  // Only filters base64, not HTTP URLs
      );
  // ...
}

4. Fallback to Hardcoded List

// packages/data-provider/src/config.ts
export function validateVisionModel({ model, modelSpecs, ... }): boolean {
  // 1. Check modelSpecs.vision first (new method)
  if (modelSpecs?.list) {
    const matchingSpec = modelSpecs.list.find(/* ... */);
    if (matchingSpec?.vision !== undefined) {
      return matchingSpec.vision === true;
    }
  }
  
  // 2. Fallback to hardcoded list (deprecated, but kept for compatibility)
  return visionModels.concat(additionalModels).some(/* ... */);
}

Impact

• All OpenAI-compatible providers use content array format (including OpenRouter)
• Artifacts are processed correctly and sent to vision-capable models
• Base64 images are filtered when vision is disabled (prevents context-length-overflow errors)
• Models with vision enabled receive images (improves response quality)

Open Question

Should OpenRouter also be switched to content array format (removing the special handling), or is there a reason it should use string format?

We'd like feedback before finalizing the changes, as multiple areas are affected:

• packages/api/src/mcp/parsers.ts (content format logic)
• packages/api/src/agents/run.ts (vision passing)
• dev/agents/src/graphs/Graph.ts (artifact formatting)
• dev/agents/src/messages/core.ts (vision filtering)

What do you think about this approach?

[Jonah16-hub]

Confirming the same root cause with a different setup: self-hosted SGLang Diffusion (FLUX.2-klein-9B) via OpenAI Image Tools (not MCP) pointed at a local non-vision LLM (Gemma 4 / Ornith-1.0, served via vLLM/LiteLLM). Same symptom : image generates successfully (confirmed backend-side and via direct connectivity test from inside the LibreChat container), but never displays; agent reports a generic "no image data" error instead of the image content. This suggests the issue isn't MCP-specific. It affects any tool returning image content back to a non-vision model, including the built-in OpenAI Image Tools. +1 for vision?: boolean in modelSpecs as proposed by @JumpLink in #11333.

Thanks for all this beautiful project.

[Jonah16-hub]

Confirming the same root cause with a different setup: self-hosted SGLang Diffusion (FLUX.2-klein-9B) via OpenAI Image Tools (not MCP) pointed at a local non-vision LLM (Gemma 4 / Ornith-1.0, served via vLLM/LiteLLM). Same symptom : image generates successfully (confirmed backend-side and via direct connectivity test from inside the LibreChat container), but never displays; agent reports a generic "no image data" error instead of the image content. This suggests the issue isn't MCP-specific. It affects any tool returning image content back to a non-vision model, including the built-in OpenAI Image Tools. +1 for vision?: boolean in modelSpecs as proposed by @JumpLink in #11333.

Thanks for all this beautiful project.

The underlying model in my case (Gemma 4, Google, April 2026) is natively vision-capable and supports native tool calling The issue is specifically that LibreChat has no way to declare/detect vision capability for custom endpoints, so it falls back to non-vision behavior regardless of what the model can actually do.

[Bobettemcnugget]

Has there been any progress on this? Any way we can help?

Right now, using Lemonade as a provider with known vision capable models, I am unable to send an image. As I am contemplating a migration from OWUI, this is quite the roadblock. Is there something I'm missing or this truly related to this issue?