Rust Bindings

[jack-michaud]

I saw that similar bindings were created for whisper.cpp in whisper-rs (https://github.com/tazz4843/whisper-rs), and I think it would be great to have similar bindings for llama.cpp as well.

As a Rust developer, I would use these to create an inference and embeeddings HTTP server and eventual create a Langchain binding for this (langchain-ai/langchain#1473).

I'd be willing to help with the bindings in any way I can. I'll be using whisper-rs as the template.

[Ronsor]

You may want to see #77 and #171 about an external C/C++ API. Rust bindings could be based on those.

[jack-michaud]

Thanks for the heads up. I've already started setting up the rust bindgen code pointing at utils.h to get a proof of concept, but it should point at a stable API.

[karelnagel]

I found this repo rustformers/llm#4 and also a discord https://discord.com/invite/GBTh6HeG/login about llama rust

[setzer22]

I came here to do a shameless plug, but someone already beat me to it 😄

The repo isn't bindings, it's a port of the C++ codebase, but we're keeping up with upstream updates. We're still using ggml underneath so it is just as fast. We also have a few nice things like a discord bot library, a command-line REPL to chat with Alpaca and a working implementation for #64! The whole codebase has also been "library-fied" and it's easier to build your own applications on top of it.

[jack-michaud]

Hey @setzer22, that's awesome. That scratches this issue's itch for me. I'll close this up. Thanks for your shameless plug!

[linonetwo]

https://github.com/mdrokz/rust-llama.cpp
https://github.com/edgenai/llama_cpp-rs