Text generation

JVM library for text generation, written in Kotlin and based on the C++ library llama.cpp

Features

• Generate text from a string

Installation

• Download latest release

• Add library dependency

  dependencies {
       implementation(file("/path/to/jar"))
  }

• Unzip binaries

Usage

TL;DR

See the example module for implementation details

• Call generate to process the string and get a generated output

Step-by-step

• Load binaries

  • CPU

    TextGeneration.Llama.loadCPU(
     ggmlBase = "/path/to/ggml-base", 
     ggmlCpu = "/path/to/ggml-cpu",
     ggmlRpc = "/path/to/ggml-rpc",
     ggml = "/path/to/ggml",
     llama = "/path/to/llama",
     textGeneration = "/path/to/text-generation",
    )

  • CUDA

    TextGeneration.Llama.loadCPU(
     ggmlBase = "/path/to/ggml-base", 
     ggmlCpu = "/path/to/ggml-cpu",
     ggmlCuda = "/path/to/ggml-cuda",
     ggmlRpc = "/path/to/ggml-rpc",
     ggml = "/path/to/ggml",
     llama = "/path/to/llama",
     textGeneration = "/path/to/text-generation",
    )

• Create an instance

  TextGeneration.Llama.create(
      modelPath = "/path/to/model"
  )

• Call history to get the history of text generation

• Call generate to process the string and get a generated output

• Call reset to reset the internal state and history

• Call close to release resources

Requirements

• JVM version 9 or higher

License

This project is licensed under the Apache License 2.0

Acknowledgments

• llama.cpp