llava-cli: improve llava-cli and the API for using LLaVA

[phymbert]

From:

• server : improvements and maintenance #4216 (comment)

1. cleaning up the clip/llava libs and improving the API
2. in the old implementation, there were many internal object exposed to the server and the memory management was dubious
3. there was no obvious path for supporting parallel multimodal slots

[phymbert]

@ggerganov please tell me how I can help on this

[phymbert]

ping @damian0815 as you originally started llava.h

[JoanFM]

Hello,

Is there any progress in here? I wonder if I could be of any help.

I think it would be nice to make multimodality much more of a first class citizen in llama.cpp. I would be interested on supporting jina-clip-v1 model after the refactoring.

[ngxson]

I'm recently playing around with the currently llava implementation.

Currently, a clip model has its own clip_model_load which does not use mmap. While clip_image_batch_encode exists that could be used to process parallel slots, it's not used by llava.cpp. One of the idea that I have in my mind is to somehow reuse llama_load_model_from_file to load the model and llama_decode to decode batch of patches/images.

But that's only very draft idea, probably too complicated to implement atm. @ggerganov what do you think about this?