Swift bindings for llama.cpp thanks to which you'll be able to run compatible LLM models directly on your device.
- Lightweight and easy to use
- Works on macOS and Linux
- Supports streaming via structured concurrency
- Swift 6 ready!
- Unit tests
- Model downloads from URL and HuggingFace
Use swift package manager:
.package(url: "https://github.com/srgtuszy/llama-cpp-swift", branch: "main")
Here's a quick example on how to use it. For more, please refer to an example app in example/ folder.
// Initialize model
let model = try Model(modelPath: "<model path>")
let llama = try LLama(model: model)
// Results are delivered through an `AsyncStream`
let prompt = "what is the meaning of life?"
for try await token in await llama.infer(prompt: prompt, maxTokens: 1024) {
print(token, terminator: "")
}