LiteLLM Deployment

A Dockerized LiteLLM proxy setup that provides a unified API for multiple Large Language Model providers including Anthropic Claude, OpenAI GPT, and others.

Overview

It configures and runs LiteLLM proxy server, which acts as a unified API gateway for multiple LLM providers.

Supported Models

Anthropic Claude

• claude-haiku-4-5-20251001 (Claude Haiku 4.5)
• claude-sonnet-4-5-20250929 (Claude Sonnet 4.5)
• claude-opus-4-1-20250805 (Claude Opus 4.1)
• claude-opus-4-20250514 (Claude Opus 4)
• claude-sonnet-4-20250514 (Claude Sonnet 4)
• claude-3-7-sonnet-20250219 (Claude Sonnet 3.7)
• claude-3-5-haiku-20241022 (Claude Haiku 3.5)
• claude-3-haiku-20240307 (Claude Haiku 3)

OpenAI

• gpt-5 (GPT-5)
• gpt-5-mini (GPT-5 Mini)
• gpt-5-nano (GPT-5 Nano)
• gpt-5-pro (GPT-5 Pro)
• gpt-4.1 (GPT-4.1)
• gpt-4.1-mini (GPT-4.1 Mini)
• gpt-4.1-nano (GPT-4.1 Nano)
• gpt-4o (GPT-4o)
• gpt-4o-mini (GPT-4o Mini)
• gpt-4-turbo (GPT-4 Turbo)
• gpt-4 (GPT-4)
• gpt-3.5-turbo (GPT-3.5 Turbo)
• o1 (O1)
• o1-mini (O1 Mini)
• o1-pro (O1 Pro)
• o3 (O3)
• o3-mini (O3 Mini)
• o4-mini (O4 Mini)

Prerequisites

• Docker and Docker Compose
• PostgreSQL database (recommended: Neon DB for cloud deployment)
• API keys for your chosen providers:
  • ANTHROPIC_API_KEY for Claude models
  • OPENAI_API_KEY for OpenAI models
• LITELLM_MASTER_KEY for proxy authentication
• DATABASE_URL for PostgreSQL connection

Quick Start

1. Environment Setup

Create a .env file in the project root with your API keys and configuration:

# API Keys
ANTHROPIC_API_KEY=your_anthropic_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

# LiteLLM Configuration
LITELLM_MASTER_KEY=your_master_key_here

# Database
DATABASE_URL=your_postgresql_connection_string_here

2. Start with Docker Compose

docker-compose up -d

The proxy will be available at http://localhost:4000

3. Alternative: Direct Python Installation

If you prefer to run without Docker:

# Install LiteLLM
pip install "litellm[proxy]"

# Start the proxy
litellm --config config.yaml --detailed_debug

Or use the provided batch file on Windows:

start-litellm.bat

Configuration

The main configuration is in config.yaml. This file defines:

• Model List: Available models and their provider configurations
• General Settings: Database storage, logging, and authentication
• API Keys: Environment variable references for secure key management

Adding New Models

To add new models, edit config.yaml and add entries to the model_list:

model_list:
  - model_name: your-model-name
    litellm_params:
      model: provider/model-name
      api_key: os.environ/YOUR_API_KEY

API Usage

Health Check

curl http://localhost:4000/health

Chat Completions

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer $LITELLM_MASTER_KEY' \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'

Generate API Key

curl 'http://localhost:4000/key/generate' \
  --header 'Authorization: Bearer $LITELLM_MASTER_KEY' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "key": "my-api-key",
    "models": ["claude-3-5-sonnet-20241022", "gpt-4o"],
    "metadata": {
      "user": "user@example.com"
    }
  }'

Create User

curl -X POST 'http://localhost:4000/user/new' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer $LITELLM_MASTER_KEY' \
  -d '{
    "user_alias": "john_doe",
    "user_email": "john@example.com"
  }'

Project Structure

litellm-configuration/
├── config.yaml              # Main configuration file
├── docker-compose.yml       # Docker Compose configuration
├── Dockerfile              # Docker image configuration
├── start-litellm.bat       # Windows batch file to start proxy
├── commnads.sh             # Example API commands
├── guidelines/             # Setup guides for different providers
│   ├── guideline_set_up_claude_code.md
│   ├── guideline_set_up_codex.md
│   ├── guideline_set_up_crush.md
│   └── guideline_set_up_litellm_proxy_app.md

Development

Debug Mode

The configuration includes --detailed_debug flag for development. For production, remove this flag to improve performance:

# In Dockerfile, change:
CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"]
# To:
CMD ["--port", "4000", "--config", "config.yaml"]

Adding New Providers

1. Add your provider's API key to the environment variables
2. Update config.yaml with the new model configuration
3. Restart the proxy service

Troubleshooting

Common Issues

1. Connection Refused: Ensure the proxy is running on port 4000
2. Authentication Errors: Verify your LITELLM_MASTER_KEY is correct
3. Model Not Found: Check that the model name in your request matches the configuration
4. Database Connection: Verify your DATABASE_URL is correct and accessible

Security Notes

• Keep your API keys secure and never commit them to version control
• Use environment variables for all sensitive configuration
• Regularly rotate your LITELLM_MASTER_KEY
• Monitor usage logs for unusual activity

References

https://docs.litellm.ai