azd-litellm

An azd template to deploy LiteLLM running in Azure Container Apps using an Azure PostgreSQL database.

To use this template, follow these steps using the Azure Developer CLI:

1. Log in to Azure Developer CLI. This is only required once per-install.

   azd auth login

2. Initialize this template using azd init:

   azd init --template build5nines/azd-litellm

3. Use azd up to provision your Azure infrastructure and deploy the web application to Azure.

   azd up

4. azd up will prompt you to enter these additional secret and password parameters used to configure LiteLLM:

   • databaseAdminPassword: The Admin password use to connect to the PostgreSQL database.
   • litellm_master_key: The LiteLLM Master Key. This is the LiteLLM proxy admin key.
   • litellm_salt_key: The LiteLLM Salt Key. This cannot be changed once set, and is used to encrypt model keys in the database.

   Be sure to save these secrets and passwords to keep them safe.

5. Once the template has finished provisioning all resources, and Azure Container Apps has completed deploying the LiteLLM container (this can take a minute or so after azd up completes to finish), you can access both the Swagger UI and Admin UI for LiteLLM.

   This can be done by navigating to the litellm service Endpoint returned from the azd deployment step using your web browser. You can also find this endpoint by navigating to the Container App within the Azure Portal then locating the Application Url.

   

   Navigating to the Endpoint URL will access Swagger UI:

   

   Navigating to /ui on the Endpoint URL will access the LiteLLM Admin UI where Models and other things can be configured:

   

Architecture Diagram

In addition to deploying Azure Resources to host LiteLLM, this template includes a DOCKERFILE that builds a LiteLLM docker container that builds the LiteLLM proxy server with Admin UI using Python from the litellm pip package.

Azure Resources

These are the Azure resources that are deployed with this template:

• Container Apps Environment - The environment for hosting the Container App
• Container App - The hosting for the LiteLLM Docker Container
• Azure Database for PostgreSQL flexible server - The PostgreSQL server to host the LiteLLM database
• Log Analytics and Application Insights - Logging for the Container Apps Environment
• Container Registry - Used to deploy the custom Docker container for LiteLLM

How to use Specific Version of LiteLLM

By default, this project uses the latest version of LiteLLM. There may be reasons you want to run a specific version of LiteLLM. This project deploys LiteLLM using its Python package, and you can update the version referenced to target a specific version of LiteLLM if necessary.

To do so, you can edit the /src/litellm/requirements.txt file. This file is the Python pip requirements file that specifies the PIP packages and their versions to use. The litellm[proxy] package is the package that is LiteLLM. You can find the available release versions on the litellm PIP package page.

Here's an example of the requirements.txt file reference of litellm package specifying the v1.65.1 release:

litellm[proxy]==1.65.1

By default, this project does not specify a version; which tells PIP to pull down the latest release. I hope this helps if you find yourself needing to run a specific version of LiteLLM.

Author

This azd template was written by Chris Pietschmann, founder of Build5Nines, Microsoft MVP, HashiCorp Ambassador, and Microsoft Certified Trainer (MCT).