RPC issues and comments

[steampunque]

Local LAN 1x 1070 1x 4070 1x 4070 configured with new RPC with patched server to use RPC.

I did a run to fully offload mixtral Q4_K_M into the 3 GPUs with RPC all looked good:

llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: CPU buffer size = 70.31 MiB
llm_load_tensors: RPC buffer size = 7043.34 MiB (1070)
llm_load_tensors: RPC buffer size = 9391.12 MiB (4070)
llm_load_tensors: RPC buffer size = 8711.09 MiB (4070)

All layers offloaded and the timings I am getting are:

pp 105.99 tokens per second
tg 25.68 tokens per second

This compares to around 5tps generation with CPU+GPU to 1 4070 so over 5x speedup is nice. And it seems to be working OK. Some issues I have found so far :

The rpc servers are spamming rpc_get_tensor and rpc_set_tensor messages to console, this needs to be shut off unless doing debugging.

I initially tried a partial offload to two machines (8G +12G) but I got an out of memory crash on one of the servers, so I
am guessing that RPC mode currently does not support mixed CPU and GPU offload, i.e. GPU offload only so if your models doesn't fit in the memory there is no possibility to pick up the rest of the layers with CPU? More of a question. It should be possible to pick up the remaining layers that won't RPC into GPUs on the host running the server (my host has 128G RAM).

When the rpc servers crash, they cannot be restarted without doing a hard restart of the RPC subsystem (restart rpcbind, etc.) Something is not being cleaned up correctly when the rpc servers SEGV crash.

Also Great work on this feature it is extremely useful! It will be very good to support mixed CPU and GPU with this mode though so the crazy models such as dbrx commandr+ falcon180 and the 70G llama3 monster could be run if desired.

[ggerganov]

I am guessing that RPC mode currently does not support mixed CPU and GPU offload, i.e. GPU offload only so if your models doesn't fit in the memory there is no possibility to pick up the rest of the layers with CPU?

Might be possible to start 2 RPC servers on the same machine - one CPU and one GPU?

[rgerganov]

Thanks for the feedback!

The rpc servers are spamming rpc_get_tensor and rpc_set_tensor messages to console, this needs to be shut off unless doing debugging.

Sure, will turn this off by default.

I am guessing that RPC mode currently does not support mixed CPU and GPU offload

The problem is that we don't report available memory on CPU and Metal and it defaults to 1MB:
https://github.com/ggerganov/llama.cpp/blob/master/examples/rpc/rpc-server.cpp#L41
As a stopgap solution I will add command line arguments for free_mem and total_mem until we implement this for all platforms.

It should be possible to pick up the remaining layers that won't RPC into GPUs on the host running the server (my host has 128G RAM).

Agree, will be working on this.

When the rpc servers crash, they cannot be restarted without doing a hard restart of the RPC subsystem (restart rpcbind, etc.) Something is not being cleaned up correctly when the rpc servers SEGV crash.

I guess this is because rpc-server fails to start on the same port and it needs some time for the old socket to expire. Does changing the port helps?

[steampunque]

I am guessing that RPC mode currently does not support mixed CPU and GPU offload, i.e. GPU offload only so if your models doesn't fit in the memory there is no possibility to pick up the rest of the layers with CPU?

Might be possible to start 2 RPC servers on the same machine - one CPU and one GPU?

good idea I did not think of that. For a big model like Mixtral 8x22b loading would be very slow though pushing 50-60Gbyes thru local RPC socket instead of moving the layers straight from of file into memory ... and also slower on inference though Im guessing that would not be a huge overhead hit since CPU is not fast anyway.

[steampunque]

I am guessing that RPC mode currently does not support mixed CPU and GPU offload

The problem is that we don't report available memory on CPU and Metal and it defaults to 1MB: https://github.com/ggerganov/llama.cpp/blob/master/examples/rpc/rpc-server.cpp#L41 As a stopgap solution I will add command line arguments for free_mem and total_mem until we implement this for all platforms.

That would be perfectly fine stopgap solution thanks!

When the rpc servers crash, they cannot be restarted without doing a hard restart of the RPC subsystem (restart rpcbind, etc.) Something is not being cleaned up correctly when the rpc servers SEGV crash.

I guess this is because rpc-server fails to start on the same port and it needs some time for the old socket to expire. Does changing the port helps?

Most likely it would work but then host (serverrpc in my case) needs to be also restarted on the new port address for the machine that crashed. To handle this cleanly most likely need to install a signal handler trapping ideally any signal that can abort the rpc server (SIGHUP SIGTERM) and the errors (SIGSEGV) and clean up the port before exit. I think SEGVs are extremely tricky to handle this way though https://stackoverflow.com/questions/2663456/how-to-write-a-signal-handler-to-catch-sigsegv so might be better to find the source of the SEGV when it ran out of memory and just go to a soft reset state when whatever happens that would have caused the SEGV (running out of GPU memory did it here) happens. When I stopped the program with CTL-C on the command line it was able to restart on the same port no problem but I didn't try stopping it using kill.

[rgerganov]

I think that setting SO_REUSEADDR on the socket might help here. Will give it a try and submit a PR if it works.