Feature Request: RPC offloading using a local model copy

[Abdulhanan535]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

RPC loading of model through network takes a lot of time which makes it useless.
i think what will be best is like download whole model on both side and then just loading specific number of layer the network tells you to load, making it load only those index of layers speeding up the loading and also saving time on every launch.

Motivation

RPC loading of model through network takes a lot of time.

Possible Implementation

download whole model on both side and then just loading specific number of layer the network tells you to load.

[GoudaCouda]

I would love to see some RPC enhancement

[Zorg33]

The network handling is also very slow. I rarely see speed above 1Gbps on a 10G direct link connection between 2 servers. It fluctuates like crazy (using fast nvme SSDs and no any other load on the servers).

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[piisawheel]

I too would like to see this feature. Or, at the very least a Cache feature, where the entire model can be cached and held until changed out. That way the load only needs to happen over the network on first use. I believe dllama uses an approach like this and their claim is that network updates for inference and not model loading increases throughput significantly, but I'm having issues setting up the dllama cluster properly :/. This really should be put higher in the list.

[suspicious-pineapple]

this would be great

[rgerganov]

This has been one of the most requested features for the RPC backend, so I will start working on it.

@slaren already gave some ideas here like hashing the payload of set_tensor and using local copies on the server side.

I propose the following implementation for this:

1. Allow RPC servers to advertise certain capabilities to their clients. This can be enabled with CLI arguments like this:

./rpc-server --enable-cache

2. Clients send a handshake message after connecting to the server and retrieve the server version and capabilities. If the server supports caching tensor data, we first send a hash and if we receive cache-miss we send the tensor data.

[ggerganov]

The server can maybe optionally use mmap when loading the cached data.

[lingster]

I have been taking a look at this and wondering if we could create a new RPC_CMD_LOAD_TENSOR, which passes in the model path/name(or a hash of the model) and tensor/weights to load. We may have to make the assumption that the model file exists in the same location on both the client and server.
If we only load the client's requested tensors this should scale across GPUs of differing sizes.

One other load optimisation is to parallelise the loading on each GPU backend. It seems in the function llama_model_loader::load_all_data() this loads the model sequentially from one GPU to the next.