An endpoint server for efficiently serving quantized open-source LLMs for code.

A Large Language Model based tool for generating human like responses to natural language inputs for network not connected over internet.

Pre-loaded LLMs served as an OpenAI-Compatible API via Docker images.

演示 vllm 对中文大语言模型的神奇效果

Embedding based semantic search app for poetry [App and EDA notebooks]

Fine-tuning and serving LLMs on any cloud

This repository demonstrates LLM execution on CPUs using packages like llamafile, emphasizing low-latency, high-throughput, and cost-effective benefits for inference and serving.

EchoSight is a tool that helps visually impaired individuals by audibly describing images taken with a Raspberry Pi Camera or inputted via image path or URL across different operating systems.

Dockerized LLM inference server with constrained output (JSON mode), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API provider.

MLOps library for LLM deployment w/ the vLLM engine on RunPod's infra.

LLM, Fine Tuning, Llama 2, Gemma, Mixtral, vLLM, LangChain, RAG, ChromaDB, FAISS

Low latency JSON generation using LLMs ⚡️

vLLM Router

A discord bot which can call LLMs using either Hugging Face or vLLM on Windows platform. Combined with function calling.

Call many AIs from a single API.

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.

Cog wrapper for NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO

Cog wrapper for cognitivecomputations/Wizard-Vicuna-13B-Uncensored

Cog wrapper for vllm implementation of Qwen/Qwen1.5-32B

Cog wrapper for vllm implementation of Qwen/Qwen1.5-110B