Should use `mmap` for model loading

[l29ah]

So it doesn't create an extra copy in RAM and lives in the kernel page cache happily, loading instantly on subsequent runs.

[apaz-cli]

@ggerganov I'm working on putting together the PR. Almost done.

I don't know anything about the order that ggml accesses the weights in. Would you say that it's sequential? If so, there's also madvise().

[apage43]

you probably don't want to use madvise+MADV_SEQUENTIAL, as in addition to increasing the amount of readahead it also causes pages to be evicted after they've been read - the entire model is going to be executed at least once per output token and read all the weights, MADV_SEQUENTIAL would potentially kick them all out and reread them repeatedly.

what may be more appropriate is to use MADV_WILLNEED on the whole model to kick off paging it all in without needing to wait for it to finish - but mmap can be tricky and you would probably want to make it an option rather than the default as it may not be a perf improvement on all setups and can wind up being slower than regular I/O due to causing lots of TLB shootdowns - you would want to benchmark it, as its not unlikely you may be trading improved time-to-first-token for worse overall throughput

[jart]

That will definitely happen with posix_fadvise(sequential), which has a very gentle impact on file caches on Linux. What we might end up wanting here is madvise(random). In order to do that though, we first would need to find a way to avoid the loading and deserialization process where c/c++ data structures are constructed in memory, and instead have the runtime data structures just be mapped directly from the file. That would ensure 100% reduction in startup time, which means we can start generating tokens asap, and pages get loaded off disk on an as-needed basis. Once we're able to implement that design pattern, madvise(random) vs. madvise(sequential) would be a tool that lets the kernel know how to utilize an under-utilized disk, to make predictions on avoiding page faults.

I'm still getting up to speed on this codebase, so I'd like to hear everyone's ideas on how best we'd ensure object data structures (or their floating point content at the very least) could be made directly mappable, thus side-stepping the loading process. One dirty hack for example I've been considering, would be overriding the memory allocators to get all objects at a fixed address, and persisting that to disk. That way, when all the C/C++ objects are loaded back into memory using MAP_FIXED, no relocations would need to be performed. That's obviously a less portable and non-ideal solution, but it'd help us get instant loading happening as quickly as possible, and furthermore permit us an opportunity to explore precisely how sparse the model's memory usage patterns actually are.

[ggerganov]

@jart
Thanks for stepping in. I will share briefly an idea that might be useful. Just mind I haven't looked into details of the discussion - will do in a few days once things cool off a bit here.

I think ggml_context is extremely well fit for mmap if I understand how it works. The ggml_context uses an externally provided buffer of memory with a pre-determined size:

https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L569-L575

All tensors and model parameters are "emplaced" into this buffer. There are no extra allocations ocuring.
Once you load the model, you can simply dump the memory buffer provided to ggml_context and next time, you can simply load this buffer instead of constructing it. Everything should work.

Edit: So above I incorrectly referenced the "eval" ggml_context which has it's own buffer. The "model" ggml_context is here:

https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L228-L240

Same stuff. If the pointer is NULL it's allocated inside ggml for convenience.

[nazthelizard122]

All tensors and model parameters are "emplaced" into this buffer. There are no extra allocations ocuring.

In other words, there is a set memory cap?
Like many other people I have basically no experience in AIs or memory buffers
I'm just a guy pushing buttons until either something explodes or an ai becomes self aware lol

[bitRAKE]

For loading from a physical/network drive resharding the larger models to a single file might help, imho. Whereas loading multiple files in parallel would be slower.

https://github.com/jankais3r/LLaMA_MPS/blob/main/reshard.py

[apaz-cli]

I'm getting ready to take another swing at it. My idea of what to do so far:

1. Create functions in utils.h called llama_load_buffer(), llama_save_buffer(), and llama_destroy_buffer(). These will mmap() (or just malloc() and read), save, and munmap() (or just free()) the buffers respectively. So, files saved on one machine can't necessarily be loaded by another. These files would be stored in some folder, and have the names of the original files. Either in models, or /tmp, or a new folder. This will also hopefully be useful for implementing saving the model state.

2. Add a new command line argument that tells llama_model_load() to look in this cache folder first. If it finds the file, llama_load_buffer() the file to get your ggml_init_params. Then do whatever else needs to be done (initialize the vocab, get hparams, etc) and exit the function. If the argument is present, call llama_save_buffer() first. Also, call llama_destroy_buffer() at the appropriate location.

I can do 1, I'll submit a PR for that shortly, but it isn't super clear to me how memory is laid out so that I can do 2. In particular, I'm wondering about the "whatever else needs to be done" part. I'm certain that I'm missing something, and it probably wouldn't be obvious to me what I'm breaking even after hours of monkeying.

[jart]

My concern with doing that is, wouldn't it effectively double the disk usage? LLaMA is big enough that folks are already likely to be stretched thin on disk space after creating the second copy needed to quantize the model. I'm still working on studying the codebase to determine what exactly are the transformation that need to be made at runtime. For example, if it's just a bunch of float16's on disk, and we're using a bunch of float16's in memory, then I don't see why the buffer field of these tensors couldn't just be populated with pointers to the appropriate positions in the file. Unless of course it needed to be reshaped or shifted to meet things like AVX alignment requirements. In that case, we'd ideally want to modify the quantizer script so that it generates something suitable for our purposes, so that only a single conversion step needs to be performed.