MultiLLM Proxy

A powerful proxy server that provides a unified interface for multiple LLM providers. This project simplifies the integration and management of various AI models by providing a single, consistent API endpoint while handling provider-specific requirements behind the scenes.

Features

🔒 Secure authentication system with user management and JWT tokens
🔑 Universal API key support with provider-specific key management
🔄 Automatic token-based rate limiting and request distribution
🌐 Support for multiple LLM providers:
- OpenAI (GPT models)
- Groq (ultra-fast inference)
- Together AI
- Google AI on Vertex AI
- Google Gemini API and hosted Gemma models
- Cerebras
- X.AI (formerly Twitter)
- Azure AI
- Scaleway
- Hyperbolic
- SambaNova
- OpenRouter
- OpenCode Go
- Xiaomi MiMo Token Plan
- NanoGPT
- NavyAI
- Codex Everywhere
- Kimi Code
- LinkAPI
- PaLM API
- Nineteen AI
- Chutes AI
🧭 Operator control plane with provider health, circuit state, route traces, and request exploration
🧩 Dashboard-managed auto:<model> priorities with safe rate-limit failover across providers
🔄 Four-state provider recovery with bounded parallel half-open probes
📊 Request, latency, response-class, and configured cost telemetry
🚀 Streaming support for compatible providers
🎭 Cloudflare-native roleplay sessions with adaptive Kimi/GLM routing and durable continuity memory
⚡ Configurable timeouts and retry mechanisms per provider
🔄 Automatic parameter handling and compatibility checks

The rationale for selectively adopting OmniRoute-style operational features without replacing MultiLLM's provider adapters is documented in the OmniRoute assessment.

Setup

Clone the repository:

git clone https://github.com/ALikesToCode/MultiLLM-Proxy.git
cd MultiLLM-Proxy

Create a virtual environment and install dependencies:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Copy the example environment file and configure your settings:

cp .env.example .env

Configure the required environment variables in .env:

Required Configuration:

# Server Configuration
SERVER_HOST=localhost
SERVER_PORT=1400

# Authentication (Required)
ADMIN_USERNAME=admin
ADMIN_API_KEY=your-universal-api-key
FLASK_SECRET_KEY=your-flask-secret-key
JWT_SECRET=your-jwt-secret-key

Provider-Specific Configuration:

# OpenAI
OPENAI_API_KEY=your-openai-api-key

# Cerebras
CEREBRAS_API_KEY=your-cerebras-api-key

# X.AI
XAI_API_KEY=your-xai-api-key

# Google AI
GOOGLE_APPLICATION_CREDENTIALS=path-to-your-google-credentials.json

# Groq (supports multiple keys for rate limiting)
GROQ_API_KEY_1=your-first-groq-api-key
GROQ_API_KEY_2=your-second-groq-api-key

# Together AI
TOGETHER_API_KEY=your-together-api-key

# Azure AI
AZURE_API_KEY=your-azure-api-key

# Scaleway
SCALEWAY_API_KEY=your-scaleway-api-key

# Hyperbolic
HYPERBOLIC_API_KEY=your-hyperbolic-api-key

# SambaNova
SAMBANOVA_API_KEY=your-sambanova-api-key

# OpenRouter
OPENROUTER_API_KEY=your-openrouter-api-key

# OpenCode Go
OPENCODE_GO_API_KEY=your-opencode-go-key
# Optional compatibility alias: OPENCODE_API_KEY=your-opencode-go-key

# Xiaomi MiMo Token Plan
MIMO_API_KEY=your-mimo-token-plan-api-key

# NanoGPT
NANOGPT_API_KEY=your-nanogpt-api-key
NANOGPT_API_KEY_1=your-second-nanogpt-api-key
NANOGPT_PREFERRED_KEY_INDEX=1
# Compatibility aliases: NANO_GPT_KEY and NANO_GPT_KEY_N

# NavyAI
NAVYAI_API_KEY=your-navyai-api-key

# Codex Everywhere (preferred key name)
CODEX_EASY_API_KEY=your-codex-everywhere-key
# Optional compatibility alias: CODEX_API_KEY=your-codex-everywhere-key

# Kimi Code
KIMI_CODE_API_KEY=your-kimi-code-key

# LinkAPI (preferred key name)
LINKAPI_KEY=your-linkapi-key

# Optional Cloudflare Worker fast-path override; Flask uses the global endpoint
LINKAPI_BASE_URL=https://api.linkapi.ai

# PaLM API
PALM_API_KEY=your-palm-api-key

# Nineteen AI
NINETEEN_API_KEY=your-nineteen-api-key

# Chutes AI
CHUTES_API_TOKEN=your-chutes-api-token

# Gemini API and hosted Gemma models
GEMINI_API_KEY=your-gemini-api-key

Resilience and operational telemetry

MultiLLM tracks normalized provider transports as closed, degraded, open, or half_open. The defaults admit two bounded half-open probes so two Codex instances can verify provider recovery without opening an unlimited retry flood. Fidelity-first raw transports remain retry- and circuit-free and are labeled passthrough; OpenCode is shown as mixed because its direct native paths are raw while normalized routes are managed. Configure the thresholds with the CIRCUIT_BREAKER_* settings in .env.example. Circuit state is process-local, resets on restart, and is not presented as shared across application replicas.

API responses expose the selected provider, model, route decision, circuit state, proxy latency, and—when configured—an estimated cost through X-MultiLLM-* headers. Pricing is operator-supplied:

MODEL_PRICING_USD_PER_MILLION='{"provider:model":{"input":1.00,"output":2.00},"provider:*":{"input":1.00,"output":2.00}}'

Exact model entries take precedence over provider wildcards and the global "*" wildcard. Estimates use the incoming prompt-token estimate and the caller's requested output limit; they are operational planning signals, not provider invoices. If no output limit is supplied, MultiLLM does not invent one. When no matching price is configured, the request remains visibly unpriced.

Run the server:

python app.py

The server will start at http://localhost:1400 (or your configured host/port).

Usage

Authentication

The proxy uses a secure authentication system with:

Session-based authentication for web dashboard
JWT token generation for API access
Universal API key system
Secure password hashing
CSRF protection

API Endpoints

After signing in, open /docs on the running proxy for the live setup guide. It combines runtime credential status, provider capabilities, native endpoint paths, saved auto: priorities, copyable chat/image requests, and every model known from built-in configuration, the last successful provider catalog refresh, or a saved route. Use /docs.json for the same credential-safe catalog as JSON.

Each provider is accessible through their respective endpoints:

# OpenAI-compatible endpoint
curl -X POST "http://localhost:1400/openai/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Groq endpoint
curl -X POST "http://localhost:1400/groq/openai/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b-8192",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Provider Endpoints

OpenCode Go supports both native client protocols through the proxy:

OpenAI-compatible base:   http://localhost:1400/opencode/v1
Anthropic-compatible base: http://localhost:1400/opencode
Models:                    http://localhost:1400/opencode/v1/models

See OpenCode Go integration for model/protocol mapping, authentication, subscription limits, and the backward-compatible legacy chat route.

Quick reference for all provider endpoint URLs:

# OpenAI
http://localhost:1400/openai/v1/chat/completions

# Groq
http://localhost:1400/groq/openai/v1/chat/completions

# Together AI
http://localhost:1400/together/v1/chat/completions

# Google AI (Gemini)
http://localhost:1400/googleai/predict

# Gemini API and hosted Gemma models
http://localhost:1400/gemini/chat/completions
http://localhost:1400/gemini/v1beta/models/{model}:generateContent
http://localhost:1400/gemma/chat/completions
http://localhost:1400/gemma/v1beta/models/{model}:generateContent

# Cerebras
http://localhost:1400/cerebras/v1/chat/completions

# X.AI
http://localhost:1400/xai/v1/chat/completions

# Azure AI
http://localhost:1400/azure/v1/chat/completions

# Scaleway
http://localhost:1400/scaleway/chat/completions

# Hyperbolic
http://localhost:1400/hyperbolic/chat/completions

# SambaNova
http://localhost:1400/sambanova/chat/completions
http://localhost:1400/sambanova/completions

# OpenRouter
http://localhost:1400/openrouter/chat/completions
http://localhost:1400/openrouter/models

# Chutes AI
http://localhost:1400/chutes/v1/completions

# Xiaomi MiMo Token Plan
http://localhost:1400/mimo/chat/completions

# NanoGPT
http://localhost:1400/nanogpt/v1/chat/completions
http://localhost:1400/nanogpt/v1/messages
http://localhost:1400/nanogpt/v1/responses
http://localhost:1400/nanogpt/v1/models?detailed=true
http://localhost:1400/nanogpt/v1/images/*
http://localhost:1400/nanogpt/v1/audio/*
http://localhost:1400/nanogpt/v1/files
http://localhost:1400/nanogpt/v1/batches

# NavyAI
http://localhost:1400/navyai/v1/chat/completions
http://localhost:1400/navyai/v1/messages
http://localhost:1400/navyai/v1/responses
http://localhost:1400/navyai/v1/models
http://localhost:1400/navyai/v1/images/generations
http://localhost:1400/navyai/v1/audio/*
http://localhost:1400/navyai/v1/embeddings
http://localhost:1400/navyai/v1/moderations
http://localhost:1400/navyai/v1/usage

# OpenCode Go native protocol routes
http://localhost:1400/opencode/v1/chat/completions
http://localhost:1400/opencode/v1/messages
http://localhost:1400/opencode/v1/models

# Codex Everywhere OpenAI-compatible routes
http://localhost:1400/codex-easy/v1/models
http://localhost:1400/codex-easy/v1/responses
http://localhost:1400/codex-easy/v1/chat/completions
http://localhost:1400/codex-easy/v1/images/*

# Kimi Code OpenAI-compatible routes
http://localhost:1400/kimi-code/v1/models
http://localhost:1400/kimi-code/v1/chat/completions

# LinkAPI native and OpenAI-compatible routes
http://localhost:1400/linkapi/v1/models
http://localhost:1400/linkapi/v1/messages
http://localhost:1400/linkapi/v1/responses
http://localhost:1400/linkapi/v1/chat/completions
http://localhost:1400/linkapi/v1/images/generations
http://localhost:1400/linkapi/v1/images/edits
http://localhost:1400/linkapi/v1beta/models/{model}:generateContent

# PaLM
http://localhost:1400/palm/models/chat-bison-001:generateText

# Nineteen AI
http://localhost:1400/nineteen/v1/completions

For detailed usage examples with headers and request bodies, refer to the API Endpoints section above.

Automatic provider priorities

POST /v1/chat/completions accepts dashboard-managed virtual models such as auto:glm-5.2. The seeded route tries NanoGPT's exact zai-org/glm-5.2:thinking model, OpenCode, then NavyAI and moves forward only after a definite authentication, payment, model-availability, rate-limit, or local circuit rejection. Direct provider:model requests remain unchanged.

Unified GLM-5.2 requests default to the strongest reasoning level supported by the selected provider. An explicit reasoning_effort still lowers the effort; max means the provider's real maximum rather than a literal unsupported value.

Long unified and roleplay requests also select a provider-supported prompt-cache mode automatically. NanoGPT standard/PAYG mode receives caching: true, while subscription-only mode deliberately omits that flag because NanoGPT routes it through PAYG provider selection. Known affinity-key transports receive a stable hashed key, and automatic-cache providers keep their native schema. Configure PROMPT_CACHE_ENABLED and PROMPT_CACHE_MIN_TOKENS; response headers report the selected mode. This does not cache or replay generated responses, and raw provider routes remain caller-controlled.

Administrators can reorder candidates and create more virtual models from the Operations dashboard. See Automatic model priorities for the exact retry boundary, response headers, persistence behavior, and API.

Provider-Specific Features

OpenAI: Full support for chat completions, embeddings, and function calling
Groq: Ultra-fast inference with token-based rate limiting
Google AI: Support for Gemini models and multimodal tasks
Gemini API / Gemma: OpenAI-compatible chat plus native generateContent, streaming, multimodal input, tools, thought signatures, token-count preflight, and Google Search grounding
Together AI: Access to various open-source models
Cerebras: Text generation and chat capabilities
X.AI: Access to X-1 and other models
Azure AI: Support for Azure-hosted models
Scaleway: OpenAI-compatible text and chat inference
Hyperbolic: OpenAI-compatible hosted model inference
SambaNova: Text generation with streaming support
OpenRouter: Gateway to multiple AI providers
OpenCode Go: Protocol-native OpenAI Chat Completions, Anthropic Messages, streaming, and live model discovery under /opencode/v1/*
Xiaomi MiMo Token Plan: MiMo-V2.5-Pro through the SGP OpenAI-compatible endpoint
NanoGPT: Raw OpenAI and Anthropic text APIs plus models, embeddings, images, video, audio, memory, search/extraction, moderation, batches, evals, TEE verification, partner auth, and x402 payments under /nanogpt/*
NavyAI: Raw OpenAI Chat and Responses, Anthropic Messages, images and video jobs, embeddings, speech, moderation, models/status, usage, and OAuth token flows under /navyai/*
Codex Everywhere: Raw OpenAI Responses, Chat Completions, key-group-specific model discovery, and conditional image routes under /codex-easy/v1/*
Kimi Code: OpenAI-compatible Chat Completions for k3 through the fixed https://api.kimi.com/coding/v1 coding endpoint
LinkAPI: Native Claude Messages, Gemini generateContent, OpenAI Responses, OpenAI-compatible chat, model discovery, and image generation/editing under /linkapi/*; JSON generation is also available through unified /v1/images/generations with a linkapi:<model> ID
PaLM API: Google's PaLM language models
Nineteen AI: High-performance inference for open-source models with streaming support
Chutes AI: DeepSeek and other hosted models with streaming completions

Cloudflare-native roleplay

POST /v1/roleplay keeps orchestration and state in the Cloudflare Worker and a session-scoped Durable Object. JanitorAI and other OpenAI-compatible clients can use the full /roleplay/v1/chat/completions alias. It keeps turns ordered, stores bounded continuity memory, and records per-model latency and reliability. OpenCode generations use Container egress because OpenCode rejects Worker-origin HTTP signatures.

The production policy sends GLM to NanoGPT subscription first, OpenCode Go glm-5.3 then glm-5.2, and NavyAI glm-5.2-venice last. LinkAPI remains a Kimi-only roleplay tier; OpenRouter is omitted from the roleplay chain. Automatic fallback handles explicit model, capacity, authentication, rate-limit, and service-unavailable rejections. Ambiguous transport and gateway failures stop to avoid duplicate generation. Each route carries its own context/output limits. The Worker filters out routes that cannot fit the current input, so a larger NavyAI or NanoGPT GLM context can be selected without assuming every gateway exposes the same capacity. Raw roleplay dialogue remains in Durable Object storage and is resent until the conversation crosses 128,000 estimated tokens. Only then does forced compaction replace older dialogue with structured continuity memory while keeping the newest 32 messages verbatim.

curl "$PROXY_BASE_URL/v1/roleplay" \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: story-42-turn-1" \
  -d '{
    "session_id":"story-42-main",
    "input":"Continue from the locked library door.",
    "character":{"name":"Mira","persona":"A guarded court mage."},
    "memory":{"mode":"auto"},
    "stream":true
  }'

Compaction thresholds derive from eligible provider contexts by default. Storage-only compaction preserves the complete current request when it still fits; context-forced compaction uses the continuity digest. If compaction cannot make a request fit, generation does not start. Omitted max_tokens uses the largest output that fits the selected route, with no proxy-wide 20k ceiling. Authenticated JanitorAI-origin requests are treated as unlimited even when the client sends a numeric placeholder. If an upstream SSE leg reaches finish_reason: length, the Worker suppresses that intermediate terminator and continues the same assistant response until the model returns a natural stop or the configured continuation safety bound is exhausted. A provider stop may use one separately bounded repair when the caller declared a mandatory IMAGE PROMPT: block and its canonical required fields are incomplete. Current and legacy field labels are accepted. Repairs that add no required field are discarded, and only one final image block is returned or stored.

See the roleplay endpoint guide for the complete request contract, lore activation, adaptive metrics, token controls, retention, and Cloudflare configuration.

NanoGPT and NavyAI raw gateways

Both integrations preserve upstream request and response protocols: JSON bytes, multipart boundaries, binary media, SSE event types, query parameters, status codes, and safe response metadata. They are single-attempt transports so paid generation requests are never duplicated by an automatic proxy retry.

Use OpenAI-style clients with these base URLs:

$PROXY_BASE_URL/nanogpt/v1
$PROXY_BASE_URL/navyai/v1

Use Anthropic-style clients with:

$PROXY_BASE_URL/nanogpt
$PROXY_BASE_URL/navyai

Normal clients send the MultiLLM Proxy key in Authorization: Bearer ... or X-Api-Key; the server replaces it with a validated NANOGPT_API_KEY[_N] or NAVYAI_API_KEY. When multiple NanoGPT keys are configured, the proxy checks the read-only model catalog, caches the working key, and rotates after an auth or rate-limit rejection. Unified chat, Responses, image, and roleplay requests can try the next key only after a definite 401, 402, 403, or 429; raw NanoGPT routes keep their single-attempt transport contract. To forward a caller-owned upstream bearer/API key, partner JWT, Navy OAuth token, or NanoGPT L402 credential, authenticate the proxy with X-MultiLLM-Api-Key and keep the provider credential in its native header. Set NANOGPT_PREFERRED_KEY_INDEX=1 to try the _1 credential before key 0; the remaining credentials stay available as fallbacks.

NanoGPT unified text routing defaults to NANOGPT_BILLING_MODE=subscription and uses https://nano-gpt.com/api/subscription. Key checks and live model discovery use that same subscription catalog, and /v1/responses is bridged to subscription Chat Completions because NanoGPT's subscription Responses path is not available. Set NANOGPT_BILLING_MODE=standard only for an account that intentionally permits PAYG. Raw /nanogpt/* and NanoGPT media routes retain the standard provider contract and remain subject to the account's own billing guard.

NanoGPT batch routes are automatically sent to its dedicated batch host. Browser-based NanoGPT and NavyAI authorization pages remain direct because the proxy deliberately does not retain upstream cookies or follow redirects.

See the complete capability, authentication, polling, x402, OAuth, and route matrices in NanoGPT gateway and NavyAI gateway.

Codex Everywhere OpenAI fast path

Cloudflare serves /codex-easy/* directly from the Worker without waking the Flask Container. The upstream is fixed to https://codex-easy.ai. Configure the preferred CODEX_EASY_API_KEY Worker secret; the existing CODEX_API_KEY name remains a fallback alias.

Choose the deployed base URL according to what the client appends:

Client behavior	Proxy base URL
Client appends `/v1` itself, including Codex Responses clients	`$PROXY_BASE_URL/codex-easy`
Client expects a base URL that already ends in `/v1`, including many Hermes and OpenAI-compatible setups	`$PROXY_BASE_URL/codex-easy/v1`

Direct routes are:

Operation	Proxy route	Caller authentication
Key-group model catalog	`$PROXY_BASE_URL/codex-easy/v1/models`	`Authorization: Bearer $ADMIN_API_KEY`
OpenAI Responses	`$PROXY_BASE_URL/codex-easy/v1/responses`	`Authorization: Bearer $ADMIN_API_KEY`
Chat Completions	`$PROXY_BASE_URL/codex-easy/v1/chat/completions`	`Authorization: Bearer $ADMIN_API_KEY`
Images	`$PROXY_BASE_URL/codex-easy/v1/images/*`	`Authorization: Bearer $ADMIN_API_KEY`

The Worker verifies the caller against ADMIN_API_KEY, removes that credential, and authenticates upstream with CODEX_EASY_API_KEY or its CODEX_API_KEY alias. The direct path is admin-only and bypasses Flask dashboard-user authentication, application-level request-size checks, RPM/TPM/daily limits, Flask request/rate-limit accounting, and request metrics. Use the Container-backed /v1/responses or /v1/chat/completions route with a codex-easy:<model> model ID when those controls are required.

Model catalogs are specific to the purchased API-key group. Query /codex-easy/v1/models before selecting a model. The grok-4.5 requests below demonstrate current Responses and Chat request shapes only; use the exact model ID returned for your key group:

curl "$PROXY_BASE_URL/codex-easy/v1/responses" \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"grok-4.5","reasoning":{"effort":"high"},"prompt_cache_key":"conversation-123","input":"Explain this repository","stream":true}'

curl "$PROXY_BASE_URL/codex-easy/v1/chat/completions" \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "X-Grok-Conv-Id: conversation-123" \
  -H "Content-Type: application/json" \
  -d '{"model":"grok-4.5","reasoning_effort":"high","messages":[{"role":"user","content":"Explain this repository"}],"stream":true}'

Request and response bytes are passed through unchanged, including native SSE and multipart/image bodies. The Codex Everywhere and LinkAPI raw OpenAI fast paths, /codex-easy/v1/* and /linkapi/v1/*, preserve a Responses prompt_cache_key in the request body and forward the Chat X-Grok-Conv-Id header. For Grok requests, xAI's prompt-caching guidance recommends a stable prompt_cache_key for Responses or x-grok-conv-id for Chat to improve cache routing. These fields do not guarantee a cache hit; caching remains an upstream behavior and stable request prefixes still matter.

/v1/images/* works only for image-generation key groups. Generation POSTs are single-attempt: the proxy never retries them and does not provide idempotency. Retry only when the selected upstream endpoint explicitly documents an idempotency guarantee.

Kimi Code OpenAI-compatible routes

Configure KIMI_CODE_API_KEY as a Cloudflare secret. The Worker authenticates /kimi-code/* callers before any Container wakeup and serves the configured k3 model catalog at the edge. Chat Completions then stream through the Container because Kimi's edge rejects Worker-origin egress; the Container makes one request to the fixed https://api.kimi.com/coding/v1 upstream.

Operation	Proxy route	Caller authentication
Model catalog	`$PROXY_BASE_URL/kimi-code/v1/models`	`Authorization: Bearer $ADMIN_API_KEY`
Chat Completions	`$PROXY_BASE_URL/kimi-code/v1/chat/completions`	`Authorization: Bearer $ADMIN_API_KEY`

Kimi Code's generation API is Chat Completions only in this integration; /kimi-code/v1/responses is not supported. Use model k3 on the raw route, or kimi-code:k3 through the unified /v1/chat/completions route when request-size checks, rate limits, and unified accounting are required. Both generation routes are single-attempt and preserve the provider stream.

For K3's strongest reasoning setting, send "reasoning_effort":"max". A stable prompt_cache_key can improve upstream cache affinity for repeated conversation prefixes, but it does not guarantee a cache hit:

curl "$PROXY_BASE_URL/kimi-code/v1/chat/completions" \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"k3","reasoning_effort":"max","prompt_cache_key":"conversation-123","messages":[{"role":"user","content":"Explain this repository"}],"stream":true}'

The direct path validates ADMIN_API_KEY, replaces it with KIMI_CODE_API_KEY, and preserves the OpenAI-compatible request and response stream. Generation requests are single-attempt to avoid duplicated work and billing.

Adaptive and explicit context optimization

Unified GLM-5.2 chat requests automatically run the deterministic optimizer once their estimated input exceeds GLM_AUTO_OPTIMIZE_TRIGGER_TOKENS (8,000 by default). It can remove only older high-confidence image-prompt blocks while retaining the surrounding assistant story, newest full image prompt, global system/developer instructions, recent turns, media, tools, and reasoning structures. It never summarizes or drops ordinary text and never makes another provider call. Set GLM_AUTO_OPTIMIZE=false to disable it.

Prompt detection uses a bounded, process-local SHA-256 analysis cache. Cache entries contain only content hashes and prompt-span offsets—never conversation text, credentials, or generated replies. Configure it with CONTEXT_ANALYSIS_CACHE_ENABLED, CONTEXT_ANALYSIS_CACHE_TTL_SECONDS (300), and CONTEXT_ANALYSIS_CACHE_MAX_ENTRIES (2,048). Exact content changes automatically miss the cache.

POST /optimize/v1/chat/completions remains the explicit Container-backed wrapper for every unified Chat Completions model. /v1/responses and provider-specific routes remain unchanged.

The default deterministic mode makes no extra model call. After trigger_input_tokens is exceeded, it can replace high-confidence older detailed image-generation prompts with a stable marker while retaining the newest detailed image prompt, recent turns, system/developer instructions, multimodal exchanges, tool chains, and reasoning/thinking structures. Set image_prompt_history to all to disable image-prompt compaction, and use preserve_message_indices for messages that must remain byte-for-byte present.

curl "$PROXY_BASE_URL/optimize/v1/chat/completions" \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model":"kimi-code:k3",
    "messages":[{"role":"user","content":"Continue the conversation."}],
    "reasoning_effort":"max",
    "prompt_cache_key":"conversation-123",
    "optimization":{
      "mode":"deterministic",
      "trigger_input_tokens":96000,
      "target_input_tokens":96000,
      "keep_recent_turns":8
    }
  }'

summarize mode requires an explicit summary_model in provider:model form. When older safe plain-text history needs compression, it makes exactly one additional billed, rate-limited summary request before the final request. Summary transport is restricted to codex-easy, kimi-code, or linkapi; summary calls use a bounded 45-second read timeout and two-slot per-process pool, never retry, and fall back to deterministic safe pruning on failure or local saturation. Example options are "mode":"summarize", "summary_model":"kimi-code:k3", and "summary_max_tokens":800.

By default, the summary model must use the same provider as the final model. The selected summary provider receives the eligible historical user/assistant plaintext verbatim before returning a bounded digest, so that history can include sensitive text. Prefer the same provider, use preserve_message_indices for messages that must never leave the final request, and remove secrets before sending. To deliberately send eligible history to another provider, set "allow_cross_provider_summary":true; omitting this explicit disclosure opt-in returns 400. The validated digest is reinserted as an untrusted historical assistant message so old assistant text is never promoted to user authority.

The optimizer accepts at most OPTIMIZER_MAX_REQUEST_BYTES (16 MiB by default) before parsing. The transformed final request must still satisfy the selected provider's MAX_REQUEST_BYTES, prompt, output, RPM, TPM, and daily limits. The final provider's model, key, output cap, and an RPM/daily slot are validated before any paid summary call.

Optimization metadata is returned in headers without changing the upstream JSON or SSE body: X-MultiLLM-Optimization, X-MultiLLM-Optimization-Mode, X-MultiLLM-Estimated-Input-Before, X-MultiLLM-Estimated-Input-After, X-MultiLLM-Image-Prompts-Compacted, X-MultiLLM-Messages-Summarized, X-MultiLLM-Optimization-Target-Met, X-MultiLLM-Summary, X-MultiLLM-Optimization-Cache-Hits, and X-MultiLLM-Optimization-Cache-Misses. Token values are provider-neutral byte-based estimates, not tokenizer-exact usage or billing counts.

LinkAPI native fast path

On Cloudflare, requests under /linkapi/* run directly in the Worker and do not wake the Flask Container. Use your deployed Worker origin as PROXY_BASE_URL:

Client protocol	Proxy URL	Caller authentication
Claude Messages	`$PROXY_BASE_URL/linkapi/v1/messages`	`x-api-key: $ADMIN_API_KEY` plus `anthropic-version`
OpenAI Responses	`$PROXY_BASE_URL/linkapi/v1/responses`	`Authorization: Bearer $ADMIN_API_KEY`
OpenAI compatible	`$PROXY_BASE_URL/linkapi/v1/chat/completions`	`Authorization: Bearer $ADMIN_API_KEY`
OpenAI model catalog	`$PROXY_BASE_URL/linkapi/v1/models`	`Authorization: Bearer $ADMIN_API_KEY`
OpenAI image generation	`$PROXY_BASE_URL/linkapi/v1/images/generations`	`Authorization: Bearer $ADMIN_API_KEY`
OpenAI image editing	`$PROXY_BASE_URL/linkapi/v1/images/edits`	`Authorization: Bearer $ADMIN_API_KEY`
Gemini native	`$PROXY_BASE_URL/linkapi/v1beta/models/{model}:generateContent`	Prefer `x-goog-api-key: $ADMIN_API_KEY`; `?key=$ADMIN_API_KEY` is compatibility-only

The Worker validates the caller against ADMIN_API_KEY, removes that credential, and authenticates upstream with LINKAPI_KEY. LINKAPI_BASE_URL is restricted to the allowlisted official LinkAPI hosts; arbitrary HTTPS origins are rejected.

This fast path is ADMIN_API_KEY-only and intentionally bypasses Flask dashboard-user authentication, application-level request-size checks, RPM/TPM/daily limits, Flask request/rate-limit accounting, and request metrics. When those controls are required, use the Container-backed /v1/chat/completions endpoint with a linkapi:<model> model ID, or /v1/images/generations with an image-capable linkapi:<model> ID.

Gemini clients should prefer the x-goog-api-key header. Query-string ?key= authentication is supported for compatibility, but it places the caller key in the URL, where clients and intermediaries may retain it, even though automatic Worker invocation logs are disabled.

Native request and response bodies, including SSE event types and bytes, are streamed without compatibility translation. On the raw OpenAI routes, the Worker leaves prompt_cache_key in Responses bodies and forwards the Chat X-Grok-Conv-Id header; for Grok, these are the request shapes recommended by xAI for cache routing, not a proxy or provider-level cache guarantee. The proxy never retries generation POSTs and does not provide idempotency, because repeating a request can duplicate work and billing. A caller should retry only when the selected upstream protocol and endpoint explicitly document an idempotency guarantee, using its own retry policy.

LinkAPI's live pricing page lists gpt-image-2-c for both /linkapi/v1/images/generations and /linkapi/v1/images/edits. Gemini Flash Image models use the native Gemini route, for example /linkapi/v1beta/models/gemini-2.5-flash-image:generateContent. See the LinkAPI image guide for complete examples, current model guidance, and the distinction between the raw and unified routes.

Configuration Options

The proxy server supports extensive configuration through environment variables and the config.py file:

Custom timeouts per provider
Retry mechanisms with configurable backoff
Token rate limiting
Model-specific parameter handling
Development and production environment settings

Documentation

Additional setup and deployment notes are organized under docs/.

Testing

Python tests live in tests/ and can be run with the project test runner:

npm test

The Cloudflare Worker and dashboard JavaScript suites use Node's built-in runner through the complete project script:

npm run test:worker

Security

Provider keys are replaced at the proxy boundary and redacted from supported logs and error payloads
Request validation and sanitization
Rate limiting and quota management
Secure session handling
CSRF protection

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
docs		docs
middleware		middleware
providers		providers
routes		routes
scripts		scripts
services		services
static		static
streaming		streaming
templates		templates
tests		tests
worker		worker
.bandit		.bandit
.codex		.codex
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
cloudflare-worker.mjs		cloudflare-worker.mjs
config.py		config.py
env_loader.py		env_loader.py
error_handlers.py		error_handlers.py
findings.md		findings.md
get_gemini_key.sh		get_gemini_key.sh
index.py		index.py
package-lock.json		package-lock.json
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
postcss.config.js		postcss.config.js
progress.md		progress.md
proxy.py		proxy.py
requirements.lock		requirements.lock
requirements.txt		requirements.txt
route_helpers.py		route_helpers.py
security_config.py		security_config.py
setup.sh		setup.sh
tailwind.config.js		tailwind.config.js
task_plan.md		task_plan.md
vercel-build.sh		vercel-build.sh
vercel.json		vercel.json
vercel.py		vercel.py
wrangler.jsonc		wrangler.jsonc
wsgi.py		wsgi.py

Navigation Menu

Repository files navigation

MultiLLM Proxy

Features

Setup

Required Configuration:

Provider-Specific Configuration:

Resilience and operational telemetry

Usage

Authentication

API Endpoints

Provider Endpoints

Automatic provider priorities

Provider-Specific Features

Cloudflare-native roleplay

NanoGPT and NavyAI raw gateways

Codex Everywhere OpenAI fast path

Kimi Code OpenAI-compatible routes

Adaptive and explicit context optimization

LinkAPI native fast path

Configuration Options

Documentation

Testing

Security

Contributing

License

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

MultiLLM Proxy

Features

Setup

Required Configuration:

Provider-Specific Configuration:

Resilience and operational telemetry

Usage

Authentication

API Endpoints

Provider Endpoints

Automatic provider priorities

Provider-Specific Features

Cloudflare-native roleplay

NanoGPT and NavyAI raw gateways

Codex Everywhere OpenAI fast path

Kimi Code OpenAI-compatible routes

Adaptive and explicit context optimization

LinkAPI native fast path

Configuration Options

Documentation

Testing

Security

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages