Runtime GPU backend selection

[philpax]

At present, you can only compile support for one backend at a time, and you're locked into the choice of backend. As CLBlast and CuBLAS are both supported on Linux and Windows, this means that developers must distribute twice the number of binaries, and users may not be sure which binary to download.

I believe it may be possible to build support for all backends for a given platform, and switch which one gets activated at runtime. This option would only be on start-up (i.e. the application would need to restart to switch backends, due to ggml's leaky abstractions), but I think it should be possible otherwise.

This would allow developers to distribute one binary per platform, and (potentially) automatically select the best backend based on the available hardware.

[clarkmcc]

Hilarious, I'm literally halfway through writing up a question on your GPU documentation PR about this. I would be interested in this for my little app project. Ideally I wouldn't have to release a binary for each OS/backend combination.

[philpax]

Yeah, don't worry, I'm thinking about downstream deployment concerns too 😅

[LLukas22]

Shouldn't switching the backend be possible after ggml-org/llama.cpp#2239 is implemented?

[philpax]

Not sure if that enables runtime switching. Should definitely ease the implementation of this (once we eat the initial pain of porting over), but I think it's still possible to do with the current interface (...assuming none of the backends initialize magic static state prior to any functions being called)