#

llm-inference

Here are 19 public repositories matching this topic...

katanemo / archgw

The smart edge and AI gateway for agents. Arch is a high-performance proxy server that handles the low-level work in building agents: like applying guardrails, routing prompts to the right agent, and unifying access to LLMs, etc. Natively designed to handle and process prompts, Arch helps you build agents faster.

proxy routing gateway prompt proxy-server openai envoy envoyproxy llms generative-ai llmops llm-inference llm-proxy ai-gateway llm-gateway llm-routing ai-gateway-support

Updated Oct 2, 2025
Rust

Michael-A-Kuykendall / shimmy

⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.

rust machine-learning transformers api-server developer-tools llama command-line-tool lora inference-server rust-crate huggingface huggingface-transformers huggingface-models llamacpp llm-inference local-ai gguf ollama-api openai-compatible

Updated Sep 24, 2025
Rust

microsoft / aici

AICI: Prompts as (Wasm) Programs

rust ai wasm inference transformer language-model model-serving wasmtime llm llmops llm-serving llm-inference llm-framework

Updated Jan 22, 2025
Rust

reinterpretcat / qwen3-rs

An educational Rust project for exporting and running inference on Qwen3 LLM family

educational-project llm llm-inference qwen3

Updated Aug 3, 2025
Rust

Mattbusel / Every-Other-Token

A real-time LLM stream interceptor for token-level interaction research

visualization api rust cli gui tokenizer tokens rust-lang openai token interpretability reasearch openai-api llm llm-training llm-inference

Updated Jul 31, 2025
Rust

danielgrittner / llama2-rs

LLaMA2 + Rust

rust ai deep-learning llm llm-inference llama2 llama2c

Updated Aug 8, 2023
Rust

cryscan / mia

A Lightweight, Cross-Platform Inference Framework. (WIP)

rust webgpu rwkv llm-inference

Updated Sep 23, 2025
Rust

tbogdala / sentient_core

A terminal style user interface to chat with AI characters using llama LLMs for locally processed AI.

rust ai llama chat-application terminal-ui llm llamacpp ggml llm-inference

Updated Mar 21, 2024
Rust

chriamue / chat-flame-backend

ChatFlameBackend is an innovative backend solution for chat applications, leveraging the power of the Candle AI framework with a focus on the Mistral model

rust-lang candle mistral phi backend-api huggingface-inference-endpoint llm-inference llama2

Updated Jan 21, 2024
Rust

CentML / llm-inference-bench

Lightweight and extensible LLM Inference serving benchmark tool written in Rust.

benchmarking llm-serving llm-inference

Updated Apr 4, 2024
Rust

overfocus / cya-llm

LLM Choose your own Adventure API using Rust & Htmx (HuggingFace models or Anthropic)

game rust choose-your-own-adventure chooseyourownadventure llms anthropic anthropic-claude llm-inference

Updated Oct 17, 2024
Rust

JohnClaw / chatllm.rs

rust api wrapper for llm-inference chatllm.cpp

rust chatbot inference bindings api-wrapper llama quantization gemma mistral cpu-inference llm llms chatllm ggml llm-inference qwen

Updated Nov 27, 2024
Rust

emkshv / telegram-llm-assistant

A minimalistic LLM-powered Telegram assistant written in Rust that uses a self-contained Sqlite database and is very easy to install.

cli telegram command-line telegram-bot self-hosted assistant command-line-tool assistant-chat-bots self-contained llm-inference

Updated Dec 9, 2024
Rust

Leon1777 / llama-discord-bot-rs

discord as a llm front-end

ai discord-bot llm-inference

Updated Mar 9, 2025
Rust

text-yoga / ask

WIP browser-based LLM question/answering for the web

llm llm-inference

Updated Dec 21, 2023
Rust

d1pankarmedhi / Phi3-rust

Serve Phi3 with Candle and Actix 🦀

transformers candle actix-web llm-inference phi3

Updated Mar 8, 2025
Rust

with-caer / curtana

Simplified zero-cost wrapper over llama.cpp powered by the lama-cpp-2 Crate.

rust crate llama embedding-models llm-inference

Updated Sep 19, 2025
Rust

magic003 / llama2-rs

Inference Llama 2 in Rust

rust transformer llm llm-inference llama2

Updated Jul 19, 2025
Rust

rajashekar / lc

Lightweight LLM Client

cli llm llm-inference llm-tools llm-evaluation

Updated Sep 1, 2025
Rust

Improve this page

Add a description, image, and links to the llm-inference topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-inference topic, visit your repo's landing page and select "manage topics."