A modern OCR engine, written in Rust

Truly universal encoding detector in pure Rust - port of Python version

PDF library in Rust

Work-in-progress educational programming game

Bindings to copy of SentencePiece esaxx library (fast suffix array and frequent substrings).

Minimalist ML framework for Rust

Solves subset sum problem and returns a set of decomposed integers.

An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.

Extra iterator adaptors, iterator methods, free functions, and macros.

Native Rust port of Google's HighwayHash, which makes use of SIMD instructions for a fast and strong hash function

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

aHash is a non-cryptographic hashing algorithm that uses the AES hardware instruction

A fast Rust JSON library based on SIMD.

Data and tools for generating and inspecting OLMo pre-training data.

char <-> Unicode character name (maintained fork of huonw/unicode_names)

Fast suffix arrays for Rust (with Unicode support).

Simple object to count Rust iterables

All-in-one text de-duplication

An awesome list of data deduplication use cases, papers, tools, and methods.

RFCs for changes to Rust

very efficient rank and select

An ecosystem of Rust libraries for working with large language models

Pure Rust multimedia format demuxing, tag reading, and audio decoding library

Ogg container decoder and encoder written in pure Rust

Consistent HashRing