clip : offload to GPU

[ggerganov]

With the recent support for running convolutions on the GPU (#4060) we should be able to offload CLIP to run fully on the GPU.

https://github.com/ggerganov/llama.cpp/blob/3d68f364f15778dc326f5024f2e5af1ad6dfddef/examples/llava/clip.cpp#L231-L236

• Implement ggml_acc CUDA / Metal kernels
• Avoid ggml_repeat where possible using broadcast
• Should use the new ggml-backend API (see https://github.com/ggerganov/ggml/blob/master/examples/gpt-2/main-backend.cpp)

[cmp-nct]

It seems minor but I believe supporting CLIP is a major step ahead, it's such a fundamental model

[ggerganov]

Ideally, CLIP should be supported as a separate model arch in llama.cpp, but it will take some extra work to achieve this: abetlen/llama-cpp-python#813 (comment)

We should do it at some point in the future.

[monatis]

Ideally, CLIP should be supported as a separate model arch in llama.cpp,

Maybe we can start with porting full text and vision encoder parts from my clip.cpp to llama.cpp/examples/llava/clip.[h/cpp] and with the community's testing and feedback support we can polish the implementation gradually. Then we can include it directly in llama.cpp as an additional arch once we are confident about its public API and functionality. Or I can continue to develop it externally in that repo and merge later. @ggerganov WDYT?

[cmp-nct]

I'd love to see full clip support in llama.cpp soon.
The current clip is cut down to only support what llava needed, the one of monatis contains a lot more functionality.
Imho we should aim to get the full support of features in, most important is probably llava but as standalone image analysis clip is very valuable.

[FSSRepo]

@ggerganov I have implemented broadcast for the ggml_add and ggml_mul operations (only for CPU and CUDA backend). I am just waiting for my pull request to be merged into stable diffusion and will then have some time to incorporate the changes I made in ggml.

[ggerganov]

Great! Would be great to PR them in llama.cpp (i.e. here) so that we can test CLIP performance.
I think I will be able to help with the Metal implementation

[FSSRepo]

Great! Would be great to PR them in llama.cpp (i.e. here) so that we can test CLIP performance. I think I will be able to help with the Metal implementation

See #4205, I think that, for now, we shouldn't merge that pull request until applying the changes I made to ggml in the main project. This way, we'll also have a more comprehensive implementation, eliminating the repeats and all that.

[y10ab1]

Do we have any updates on this feature? I am eager to use it!

[cmp-nct]

@ggerganov @FSSRepo
Would be awesome to get this pushed into ggml and llama.cpp
Did you see my discussion on CogVLM ? #4350
It's a vision model that beats GPT4-vision and should run well on 8-9GB VRAM quantized, it's the first time I have seen anything beating Open-AI. We definitely will need full CLIP offload but the main obstacle is that it has an additional architecture (more than just the 2 layers of projection layer of llava) that connects Vicuna-7 with Big-ViT.
I know it's a bit off topic here, just pushing that because I think it's so significant and totally overlooked

[FSSRepo]

@cmp-nct It seems that the architecture of vision model CLIP and Llama differs from the implementation here. The truth is that there will be a lot of work to do if we want to have it here.