🦀 Inference Gateway Rust SDK

A powerful and easy-to-use Rust SDK for the Inference Gateway

Connect to multiple LLM providers through a unified interface • Stream responses • Function calling • MCP tools support

Installation • Quick Start • Examples • Documentation

🦀 Inference Gateway Rust SDK

Installation

Run cargo add inference-gateway-sdk.

Usage

Creating a Client

Here is a full example of how to create a client and interact with the Inference Gateway API:

use inference_gateway_sdk::{
    CreateChatCompletionResponse, GatewayError, InferenceGatewayAPI,
    InferenceGatewayClient, ListModelsResponse, Message,
    MessageContent, MessageRole, Provider,
};
use log::info;

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();

    // Create a client
    let client = InferenceGatewayClient::new("http://localhost:8080/v1");

    // List all models and all providers
    let response: ListModelsResponse = client.list_models().await?;
    for model in response.data {
        info!("Model: {:?}", model.id);
    }

    // List models for a specific provider
    let response: ListModelsResponse = client.list_models_by_provider(Provider::Groq).await?;
    info!("Models for provider: {:?}", response.provider);
    for model in response.data {
        info!("Model: {:?}", model.id);
    }

    // Generate content - choose from available providers and models
    let response: CreateChatCompletionResponse = client
        .generate_content(
            Provider::Deepseek,
            "deepseek-v4-flash",
            vec![
                Message {
                    role: MessageRole::System,
                    content: MessageContent::String("You are a helpful assistant.".to_string()),
                    reasoning: None,
                    reasoning_content: None,
                    tool_call_id: None,
                    tool_calls: Vec::new(),
                },
                Message {
                    role: MessageRole::User,
                    content: MessageContent::String("Tell me a funny joke".to_string()),
                    reasoning: None,
                    reasoning_content: None,
                    tool_call_id: None,
                    tool_calls: Vec::new(),
                },
            ],
        )
        .await?;

    log::info!(
        "Generated content: {:?}",
        response.choices[0].message.content
    );

    Ok(())
}

The client is configured builder-style:

let client = InferenceGatewayClient::new("http://localhost:8080/v1")
    .with_token("my-token")
    .with_max_tokens(Some(512))
    .with_temperature(Some(0.2))
    .with_timeout(std::time::Duration::from_secs(30));

with_temperature applies to streaming and non-streaming chat completions; None leaves the API default. with_timeout bounds the whole request, response body included, so avoid a short timeout on a client used for streaming.

Listing Models

To list all available models from all configured providers, use the list_models method:

use inference_gateway_sdk::{
    GatewayError,
    InferenceGatewayAPI,
    InferenceGatewayClient,
    ListModelsResponse,
    Message,
};
use log::info;

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    // ...Create a client

    // List models from all providers
    let response: ListModelsResponse = client.list_models().await?;
    for model in response.data {
        info!("Model: {:?}", model.id);
    }

    // ...
}

Listing Models from a specific provider

To list all available models from a specific provider, use the list_models_by_provider method:

use inference_gateway_sdk::{
    GatewayError,
    InferenceGatewayAPI,
    InferenceGatewayClient,
    ListModelsResponse,
    Provider,
};
use log::info;

// ...Open main function

// List models for a specific provider
let response: ListModelsResponse = client.list_models_by_provider(Provider::Groq).await?;
info!("Models for provider: {:?}", response.provider);
for model in response.data {
    info!("Model: {:?}", model.id);
}

// Example with Google provider
let response: ListModelsResponse = client.list_models_by_provider(Provider::Google).await?;
info!("Google models: {:?}", response.provider);
for model in response.data {
    info!("Google Model: {:?}", model.id);
}

// ...Rest of the main function

MCP JSON-RPC Endpoint

The gateway also exposes itself as an MCP server on POST /mcp, aggregating every configured MCP server behind one JSON-RPC endpoint. Use mcp_json_rpc with one of the McpjsonrpcRequest constructors - the required MCP-Protocol-Version, Mcp-Method and Mcp-Name headers are derived from the request body:

use inference_gateway_sdk::{
    GatewayError,
    InferenceGatewayAPI,
    InferenceGatewayClient,
    McpjsonrpcRequest,
};
use log::info;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    // ...Create a client

    // The aggregated, namespaced tools of every healthy MCP server
    let response = client
        .mcp_json_rpc(McpjsonrpcRequest::tools_list(None))
        .await?;
    info!("tools/list result: {:?}", response.result);

    // Call a namespaced tool: mcp_<server alias>_<tool name>
    let response = client
        .mcp_json_rpc(McpjsonrpcRequest::tools_call(
            "mcp_deepwiki_ask_question",
            json!({
                "repoName": "inference-gateway/inference-gateway",
                "question": "How is MCP wired up?",
            }),
        ))
        .await?;

    match response.error {
        Some(error) => info!("JSON-RPC error {}: {}", error.code, error.message),
        None => info!("tools/call result: {:?}", response.result),
    }

    Ok(())
}

JSON-RPC error envelopes returned with HTTP 200, 400 (-32020, -32022) or 404 (-32601) come back as response.error rather than a GatewayError. A guardrail refusal is the exception: the gateway answers it with 403 Forbidden and a -32001 envelope, which surfaces as GatewayError::Forbidden carrying the policy message, so response.error never sees it. A GatewayError::Forbidden from mcp_json_rpc therefore means either the endpoint is not exposed (MCP_ENABLED=true and MCP_EXPOSE=true are required server-side) or the request was blocked by guardrails.

When gateway auth is enabled, mcp_protected_resource_metadata fetches the OAuth 2.0 Protected Resource Metadata (RFC 9728) for the endpoint, which names the authorization servers that mint tokens for it:

let metadata = client.mcp_protected_resource_metadata().await?;
info!("Authorization servers: {:?}", metadata.authorization_servers);

Generating Content

To generate content using a model, use the generate_content method:

use inference_gateway_sdk::{
    CreateChatCompletionResponse, GatewayError, InferenceGatewayAPI, InferenceGatewayClient,
    Message, MessageContent, MessageRole, Provider,
};

fn message(role: MessageRole, text: &str) -> Message {
    Message {
        role,
        content: MessageContent::String(text.to_string()),
        reasoning: None,
        reasoning_content: None,
        tool_call_id: None,
        tool_calls: Vec::new(),
    }
}

// Generate content - choose from available providers and models
let response: CreateChatCompletionResponse = client
    .generate_content(
        Provider::Deepseek,
        "deepseek-v4-flash",
        vec![
            message(MessageRole::System, "You are a helpful assistant."),
            message(MessageRole::User, "Tell me a funny joke"),
        ],
    )
    .await?;

log::info!(
    "Generated content: {:?}",
    response.choices[0].message.content
);

// Example with Google provider (Gemini models)
let response: CreateChatCompletionResponse = client
    .generate_content(
        Provider::Google,
        "gemini-3-pro",
        vec![
            message(MessageRole::System, "You are a helpful AI assistant."),
            message(MessageRole::User, "Explain quantum computing in simple terms"),
        ],
    )
    .await?;

log::info!(
    "Google generated content: {:?}",
    response.choices[0].message.content
);

Streaming Content

use futures_util::{pin_mut, StreamExt};
use inference_gateway_sdk::{
    CreateChatCompletionStreamResponse, FinishReason, GatewayError, InferenceGatewayAPI,
    InferenceGatewayClient, Message, MessageContent, MessageRole, Provider,
};
use log::info;

fn message(role: MessageRole, text: &str) -> Message {
    Message {
        role,
        content: MessageContent::String(text.to_string()),
        reasoning: None,
        reasoning_content: None,
        tool_call_id: None,
        tool_calls: Vec::new(),
    }
}

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();

    let model = "deepseek-v4-flash";

    let client = InferenceGatewayClient::new("http://localhost:8080/v1");
    let stream = client.generate_content_stream(
        Provider::Deepseek,
        model,
        vec![
            message(MessageRole::System, "You are a helpful assistant."),
            message(MessageRole::User, "Write a poem"),
        ],
    );
    pin_mut!(stream);
    // Iterate over the stream of Server Sent Events
    while let Some(ssevent) = stream.next().await {
        let ssevent = ssevent?;
        if ssevent.data == "[DONE]" {
            break;
        }

        // Deserialize the event response
        let generate_response_stream: CreateChatCompletionStreamResponse =
            serde_json::from_str(&ssevent.data)?;

        let Some(choice) = generate_response_stream.choices.first() else {
            continue;
        };

        if let Some(usage) = generate_response_stream.usage.as_ref() {
            // Get the usage metrics from the response
            info!("Usage Metrics: {:?}", usage);
            // Probably send them over to a metrics service
            break;
        }

        // Print the token out as it's being sent from the server
        if let Some(content) = choice.delta.content.as_ref() {
            print!("{}", content);
        }

        if let Some(reason) = choice.finish_reason.as_ref() {
            if matches!(reason, FinishReason::Stop) {
                info!("Finished generating content");
                break;
            }
        }
    }

    Ok(())
}

Messages API (Anthropic-compatible)

The gateway also exposes an Anthropic-compatible POST /messages endpoint. Providers without Messages support return GatewayError::BadRequest; use generate_content for those providers.

use futures_util::{pin_mut, StreamExt};
use inference_gateway_sdk::{
    CreateMessagesRequest, GatewayError, InferenceGatewayAPI, InferenceGatewayClient,
    MessagesMessage, MessagesMessageContent, MessagesMessageRole, MessagesResponseContentBlock,
    MessagesStreamEvent, MessagesStreamEventType, Provider,
};

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    let client = InferenceGatewayClient::new("http://localhost:8080/v1");

    let request = CreateMessagesRequest {
        max_tokens: 300,
        messages: vec![MessagesMessage {
            role: MessagesMessageRole::User,
            content: MessagesMessageContent::String("Tell me a fun fact about Rust.".to_string()),
        }],
        metadata: None,
        model: "claude-sonnet-5".to_string(),
        output_config: None,
        stop_sequences: Vec::new(),
        stream: false,
        system: None,
        temperature: None,
        thinking: None,
        tool_choice: None,
        tools: Vec::new(),
        top_k: None,
        top_p: None,
    };

    // Non-streaming
    let response = client
        .create_message(Some(Provider::Anthropic), request.clone())
        .await?;
    for block in &response.content {
        if let MessagesResponseContentBlock::TextBlock(text) = block {
            println!("{}", text.text);
        }
    }

    // Streaming - each event's `data` is a JSON-serialized `MessagesStreamEvent`
    let stream = client.create_message_stream(Some(Provider::Anthropic), request);
    pin_mut!(stream);
    while let Some(event) = stream.next().await {
        let event: MessagesStreamEvent = serde_json::from_str(&event?.data)?;
        match event.type_ {
            MessagesStreamEventType::ContentBlockDelta => {
                if let Some(text) = event.delta.as_ref().and_then(|d| d.text.as_deref()) {
                    print!("{text}");
                }
            }
            MessagesStreamEventType::MessageStop => break,
            _ => {}
        }
    }

    Ok(())
}

Tool-Use

You can pass to the generate_content function also tools, which will be available for the LLM to use:

use inference_gateway_sdk::{
    ChatCompletionTool, ChatCompletionToolType, FunctionObject,
    FunctionParameters, GatewayError, InferenceGatewayAPI,
    InferenceGatewayClient, Message, MessageContent, MessageRole, Provider,
};
use log::{info, warn};
use serde::{Deserialize, Serialize};
use serde_json::json;

fn message(role: MessageRole, text: &str) -> Message {
    Message {
        role,
        content: MessageContent::String(text.to_string()),
        reasoning: None,
        reasoning_content: None,
        tool_call_id: None,
        tool_calls: Vec::new(),
    }
}

#[tokio::main]
async fn main() -> Result<(), GatewayError> {
    // Configure logging
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();

    // API endpoint - store as a variable so we can reuse it
    let api_endpoint = "http://localhost:8080/v1";

    // Initialize the API client
    let client = InferenceGatewayClient::new(api_endpoint);

    // Define the model and provider
    let provider = Provider::Deepseek;
    let model = "deepseek-v4-flash";

    // Define the weather tool
    let parameters = json!({
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city name"
            }
        },
        "required": ["location"]
    });
    let tools = vec![ChatCompletionTool {
        type_: ChatCompletionToolType::Function,
        function: FunctionObject {
            name: "get_current_weather".to_string(),
            description: Some("Get the weather for a location".to_string()),
            parameters: Some(FunctionParameters(
                parameters.as_object().unwrap().clone(),
            )),
            strict: false,
        },
    }];

    // Create initial conversation
    let initial_messages = vec![
        message(
            MessageRole::System,
            "You are a helpful assistant that can check the weather.",
        ),
        message(MessageRole::User, "What is the current weather in Berlin?"),
    ];

    // Make the initial API request
    info!("Sending initial request to model");
    let response = client
        .with_tools(Some(tools.clone()))
        .generate_content(provider, model, initial_messages)
        .await?;

    info!("Received response from model");

    let Some(choice) = response.choices.first() else {
        warn!("No choice returned");
        return Ok(());
    };

    // Check for tool calls in the response
    if !choice.message.tool_calls.is_empty() {
        // Echo the assistant turn (with the tool_calls) before the tool replies.
        let mut assistant_turn = choice.message.clone();
        // Tool-call assistant messages typically have empty content; keep what
        // the model sent so providers that include explanatory text round-trip.
        let mut follow_up_convo = vec![
            message(
                MessageRole::System,
                "You are a helpful assistant that can check the weather.",
            ),
            message(MessageRole::User, "What is the current weather in Berlin?"),
        ];
        // Take the assistant message verbatim so tool_calls are preserved.
        assistant_turn.role = MessageRole::Assistant;
        follow_up_convo.push(assistant_turn);

        // Process each tool call
        for tool_call in &choice.message.tool_calls {
            info!("Tool Call Requested: {}", tool_call.function.name);

            if tool_call.function.name == "get_current_weather" {
                // Parse arguments into a typed value
                let args: Weather = tool_call.function.parse_arguments()?;
                let weather_result = get_current_weather(args);

                // Add the tool response to the conversation
                follow_up_convo.push(Message {
                    role: MessageRole::Tool,
                    content: MessageContent::String(weather_result),
                    reasoning: None,
                    reasoning_content: None,
                    tool_call_id: Some(tool_call.id.clone()),
                    tool_calls: Vec::new(),
                });
            }
        }

        // Send the follow-up request with the tool results
        info!("Sending follow-up request with tool results");

        let follow_up_client = InferenceGatewayClient::new(api_endpoint);
        let follow_up_response = follow_up_client
            .with_tools(Some(tools))
            .generate_content(provider, model, follow_up_convo)
            .await?;

        if let Some(choice) = follow_up_response.choices.first() {
            info!("Final response: {:?}", choice.message.content);
        } else {
            warn!("No response in follow-up");
        }
    } else {
        info!("No tool calls in the response");
        info!("Model response: {:?}", choice.message.content);
    }

    Ok(())
}

#[derive(Debug, Deserialize, Serialize)]
struct Weather {
    location: String,
}

fn get_current_weather(args: Weather) -> String {
    info!(
        "Getting weather function was called for {}",
        args.location
    );

    // In a real application, we would call an actual weather API here.
    // For this example, we'll just return a mock response.
    format!(
        "The weather in {} is currently sunny with a temperature of 22°C",
        args.location
    )
}

Health Check

To check if the Inference Gateway is running, use the health_check method:

// ...rest of the imports
use log::info;

// ...main function
let is_healthy = client.health_check().await?;
info!("API is healthy: {}", is_healthy);

Examples

Runnable examples live in the examples directory - see its README for the full list and instructions.

Supported Providers

The Inference Gateway Rust SDK supports the following providers:

Ollama (Provider::Ollama) - Local language model server
Ollama Cloud (Provider::OllamaCloud) - Ollama Cloud-hosted models
Groq (Provider::Groq) - High-speed inference provider
llama.cpp (Provider::Llamacpp) - llama.cpp OpenAI-compatible llama-server
OpenAI (Provider::Openai) - GPT models and other OpenAI services
Cloudflare (Provider::Cloudflare) - Cloudflare Workers AI
Cohere (Provider::Cohere) - Cohere language models
Anthropic (Provider::Anthropic) - Claude models
DeepSeek (Provider::Deepseek) - DeepSeek models
ElevenLabs (Provider::Elevenlabs) - Audio generation via create_speech, create_sfx and create_music (no chat completions)
Google (Provider::Google) - Google Gemini models via Generative AI API
Mistral (Provider::Mistral) - Mistral AI models
MiniMax (Provider::Minimax) - MiniMax models
Moonshot (Provider::Moonshot) - Moonshot AI models
NVIDIA (Provider::Nvidia) - NVIDIA NIM models
Z.ai (Provider::Zai) - Z.ai GLM models

Each provider may support different models and capabilities. Use the list_models_by_provider() method to discover available models for each provider.

Example:

use inference_gateway_sdk::{Provider, InferenceGatewayClient, InferenceGatewayAPI};

let client = InferenceGatewayClient::new("http://localhost:8080/v1");

// List Google models
let google_models = client.list_models_by_provider(Provider::Google).await?;
for model in google_models.data {
    println!("Google model: {}", model.id);
}

Documentation

Full API documentation is published on docs.rs.

Contributing

Please refer to the CONTRIBUTING.md file for information about how to get involved. We welcome issues, questions, and pull requests.

License

This SDK is distributed under the Apache 2.0 License, see LICENSE for more information.

Name		Name	Last commit message	Last commit date
.flox		.flox
.github		.github
examples		examples
src		src
tools/gen-types		tools/gen-types
.editorconfig		.editorconfig
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.markdownlint.json		.markdownlint.json
.markdownlintignore		.markdownlintignore
.releaserc.yaml		.releaserc.yaml
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
Taskfile.yml		Taskfile.yml
openapi.yaml		openapi.yaml
rust-toolchain.toml		rust-toolchain.toml

Navigation Menu

Repository navigation

About

Repository files navigation

🦀 Inference Gateway Rust SDK

A powerful and easy-to-use Rust SDK for the Inference Gateway

Installation

Usage

Creating a Client

Listing Models

Listing Models from a specific provider

MCP JSON-RPC Endpoint

Generating Content

Streaming Content

Messages API (Anthropic-compatible)

Tool-Use

Health Check

Examples

Supported Providers

Documentation

Contributing

License

About

Releases

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🦀 Inference Gateway Rust SDK

A powerful and easy-to-use Rust SDK for the Inference Gateway

Installation

Usage

Creating a Client

Listing Models

Listing Models from a specific provider

MCP JSON-RPC Endpoint

Generating Content

Streaming Content

Messages API (Anthropic-compatible)

Tool-Use

Health Check

Examples

Supported Providers

Documentation

Contributing

License

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Used by

Contributors

Languages