Good ideas from llama.cpp

[setzer22]

I've been tracking the llama.cpp repo. I'll use this issue to list any good ideas / things we should be aware of to keep up with in Rust land:

• GPTQ quantization 👀 GPTQ Quantization (3-bit and 4-bit) ggml-org/llama.cpp#9
• Not sure how that is even possible (isn't the task I/O bound?), but people are claiming great speedups when loading the modelling in parallel. This should be pretty easy to implement using rayon. Faster loading of the model ggml-org/llama.cpp#85 (comment)
• Seems there's an issue with the normalization function used. It should be RMSNorm. Would be good to keep an eye on this, and simply swap the the ggml function once it's implemented on the C++ side 👀 Use RMSNorm ggml-org/llama.cpp#173 (comment)
• It looks like dropping to F16 for the memory_k and memory_v reduces memory usage. It is not known whether this hurts quality, but we should follow the C++ side and add a flag to drop to F16 for the memory. This would also make the cached prompts added as part of Implementation of prompt caching #14 take half the size on disk, which is a nice bonus: Use F16 for memory_k and memory_v (as suggested in #146) ggml-org/llama.cpp#154 (review)
• Looks like the fix from bytesFromNibbles error #1 just landed upstream. We should make sure to fix it here too FIX: "inline" -> "static inline" for bytesFromNibbles and packNibbles ggml-org/llama.cpp#161
• The tokenizer used in llama.cpp has some issues. It would be better to use sentencepiece, which is the one that was used during the original LLaMA training. There seems to be a rust crate for sentencepiece. We should check if a drop-in replacement is possible Differences with the llama tokenizer ggml-org/llama.cpp#167

[philpax]

Suggest pinning this issue :>

[Narsil]

For the tokenizer item I suggest using https://github.com/huggingface/tokenizers/

Should work out of the box once converted (when this PR lands: huggingface/transformers#21955 it should become a simple let tokenizer = Tokenizer::from_file("filename") ) Cheers!

[philpax]

RMS norm landed, but they've reported regressions. Need to keep an eye on that.

[dongs0104]

@Narsil Llamatokenizer need to byte fallback option.🥹

For the tokenizer item I suggest using https://github.com/huggingface/tokenizers/

Should work out of the box once converted (when this PR lands: huggingface/transformers#21955 it should become a simple let tokenizer = Tokenizer::from_file("filename") ) Cheers!

[Narsil]

Good news everyone !

huggingface/tokenizers#1183

(If this goes, I'll try to make a release soon after)

[philpax]

Awesome! Looking forward to it :D

[dnlmlr]

A small comment on the parallel loading: It is definitely possible to improve IO reads by parallelizing. This is much more effective on SSDs but still works on HDDs due to caching at different layers. However this should be configurable since the performance can start to degrade at certain points of parallelism, depending on the storage medium and also stuff like the kernel and buffer sizes

[Narsil]

@dnlmlr Do you have bench to back that up ? I didn't found that to be the case whenever I tried.

Memory-mapping was always consistently better than reading a file (Provided you need the whole file) and it doesn't require parallism (at user-level that is, no idea how the kernel is handling it)

[philpax]

@setzer22 Are you okay with me closing this issue and splitting it into individual issues?

[setzer22]

Yup, sounds good 👍

[philpax]

This issue has been superseded by #35, #62, #78, #79 and #80.