__       __ __          
.---.-.--.--.--.-----.-----.-----.--------.-----.___.----.--.--.-----|  |_ ___|  |  .--------.
|  _  |  |  |  |  -__|__ --|  _  |        |  -__|___|   _|  |  |__ --|   _|___|  |  |        |
|___._|________|_____|_____|_____|__|__|__|_____|   |__| |_____|_____|____|   |__|__|__|__|__|

https://github.com/jondot/awesome-rust-llm

Awesome Rust LLM is an awesome style list that keeps track and curates the best Rust based LLM frameworks, libraries, tools, tutorials, articles and more. PRs are welcome!

Models & Inference

• llm - a Rust library for running inference from a number of supported LLMs, loads ggml based models
• rust-bert - all in one pipelines for for transformer-based models (BERT, DistilBERT, GPT2,...). Good for local embedding, port of transformers (python)
• llm-chain - chaining LLMs in Rust
• smartgpt (how it works)- use LLMs with the ability to complete complex tasks using plugins
• diffusers - Stable Diffusion using Rust, 45% faster than Pytorch
• postgresml - an amazing Postgres extension to do model fetching, inference all with SQL as part of your Postgres instance

Projects

• aichat - a pure Rust CLI implementing AI chat, with advanced features such as real-time streaming, text highlighting and more

• browser-agent - a headless browser driven by GPT-4. Sends off a simplified page representation and receives & executes instructions from GPT using a custom message format

• tenere - TUI interface for LLMs

How it works? here's a prompt snippet:

You must respond with ONLY one of the following commands AND NOTHING ELSE:
    - CLICK X - click on a given element. You can only click on links, buttons, and inputs!
    - TYPE X \"TEXT\" - type the specified text into the input with id X and press ENTER
    - ANSWER \"TEXT\" - Respond to the user with the specified text once you have completed the objective

• ajeto - LLM personal assistant
• shortgpt - Ask shortgpt for instant and concise answers
• autorust - macros that generate AI driven rust code in compile time
• clerk - LLM based file organizer
• gptcommit - prepare commit message with GPT

LLM Memory

• indexify - A retrieval and long term memory service for LLMs
• memex - Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.
• motorhead - a memory and information retrieval server for LLMs.
  • Uses Redis as vector store for long term memory,
  • Works with OpenAI API
  • js example, python example

Core Libraries

• tiktoken - tiktoken is a Python library with a Rust core implementing a fast BPE tokeniser for use with OpenAI's models

  • BPE is done in Rust
  • Made by OpenAI

• tiktoken-rs - a Rust focused library based on the tiktoken core with additional enhancements for use in Rust code. (Pyton parts in tiktoken done in pure Rust)

// Rust
use tiktoken_rs::p50k_base;

let bpe = p50k_base().unwrap();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());

• polars - a faster, pure Rust pandas alternative

• rllama - a pure Rust implemenation of LLaMa inference. Great for embedding into other apps or wrapping for a scripting language.

• whatlang - Rust library using a multiclass logistic regression model to detect languages

• OpenAI API - a strongly typed Rust client for the OpenAI API

Tools

• spider - crawler / spider written in Rust for when you need a whole-website dump. Unlike Scrapy, focuses on dumping data but the post-processing is done later.

Services

• dust - a full service for workflow running with composable blocks. Core is in Rust, various frontends in Typescript.

Vector Stores

• pgvecto.rs - Vector database plugin for Postgres, written in Rust, specifically designed for LLM. 20x faster than pgvector
• qdrant - Qdrant - Vector Database for the next generation of AI applications