llama_model_loader: support multiple split/shard GGUFs#6187

Merged

phymbert merged 28 commits into

hp/split/load-model

Mar 22, 2024

Commits on Mar 20, 2024

split: support in llama_model_loader
phymbert
committed

Commits on Mar 21, 2024

Commits on Mar 22, 2024

llama_model_loader: if n_tensors declared not equals to loaded tensors in split, throw an exception instead of asserting
phymbert
committed
llama_model_loader: ensure mappings vector has the expected size
phymbert
committed
llama_model_loader: use at instead of operator[] if this should never add to the map.
phymbert
committed
llama_model_loader: immediately add the backend buffer to the model buffers in order to free them if an error occurs in the next allocation. Reserve the expected size.
phymbert
committed
llama_model_loader: be sure the model mappings has enough capacity before allocating backend buffer
phymbert
committed
llama_model_loader: fix map -> unordered map
phymbert
committed
llama_split_prefix: use a clearer version, not pass split path len but dest max len.

phymbert
and
ngxson
committed
llama : minor
ggerganov
committed
llama : introduce some typedef helpers
ggerganov
committed
docs: add model shard in hot topic
phymbert
committed
llama_model_loader: put mapping in a unique_ptr from the moment it is allocated

phymbert
and
slaren
committed
fix llama_split_prefix
ngxson
committed