Performance tweaks

[ericcurtin]

Some performance tips from @ggerganov

https://x.com/ggerganov/status/1909657397964292209

we should turn these on by default for the cases described, CUDA, METAL, etc. @kpouget this might be up your alley :)

[rhatdan]

• Enable context reuse of the server by adding --cache-reuse 256. It will improve your prompt processing performance noticeably
• Enable -fa for CUDA or Metal
• Use draft model for speculative decoding

[rhatdan]

Opened a PR for this, but I don't understand wha
Use draft model for speculative decoding means.

[ggerganov]

Opened a PR for this, but I don't understand wha Use draft model for speculative decoding means.

Draft models are usually small models from the same family as the primary model. They are used to create speculative drafts of tokens that can be evaluated faster by the primary model and potentially lead to end-to-end performance improvements.

This has information how speculative decoding is enabled for the server: ggml-org/llama.cpp#10455

Let me know if something is not clear, happy to help.

[rhatdan]

@ggerganov PTAL at the PR.

Don't think that RamaLama can do anything about this other then potentially pulling two models?

./bin/llama-server \
    -m  ../models/qwen2.5-32b-coder-instruct/ggml-model-q8_0.gguf \
    -md ../models/qwen2.5-0.5b-coder-instruct/ggml-model-q4_0.gguf \
    -ngl 99 -ngld 99 -fa --port 8033 -c 32768 \
    --draft-max 16 --draft-min 5

[ericcurtin]

I wonder should we just forgo using a draft model by default for the sake of simplicity (although it could be an additional option if someone wanted to use it)

[ggerganov]

Yes, automatically selecting the draft model is difficult. Should leave it as an advanced manual option for the user in some way.

[rhatdan]

We would need an option to ramalama serve --draft-model PATH, but pulling this via URL or via OCI would probably be the only option. Does ollama registry have a way to download these?

[ericcurtin]

We would need an option to ramalama serve --draft-model PATH, but pulling this via URL or via OCI would probably be the only option. Does ollama registry have a way to download these?

AFAIK (I could stand to be corrected), it's one .gguf per artifact in Ollama. But maybe they have or will extend that format two multiple .gguf's per Ollama OCI artifact 🤷