How to use multimodal models with Outlines?

[cegutica]

I've read the issues related to multimodality but it is not clear how to use multimodal models with Outlines. If it's true that this type of models are supported by Outlines, could you add an example notebook or at least explain which methods and parameters we should use?

[lapp0]

Could you please take a look at https://github.com/outlines-dev/outlines/blob/main/docs/reference/models/transformers_vision.md

Let me know if you have any other questions

[arkohut]

This is super helpful.

But using transformers directly is not the fastest way. vllm or other framework like lmdeploy expose openai compatible api for multimodal models. Is that possible to integrate outlines with openai compatible api for multimodal models?

[lapp0]

We already use vllm for outlines.serve which provides an openai compatible api. An image input integration makes sense to me. Could you create an issue?

[arkohut]

Does it mean any openai compatible api works?

[lapp0]

A server you deploy with structured generation enabled via outlines and vLLM should work with the openai client library.

outlines.serve works as specified in https://outlines-dev.github.io/outlines/reference/serve/vllm/

Or you can use vLLM directly for an openAI compatible interface with structured generation https://github.com/vllm-project/vllm/blob/master/docs/source/serving/openai_compatible_server.md#extra-parameters

[arkohut]

model = outlines.models.transformers_vision(
    'OpenGVLab/InternVL2-4B',
    device='cuda'
)
---
You are using a model of type internvl_chat to instantiate a model of type llava_next. 
This is not supported for all configurations of models and can yield errors.

Can you give an example for https://huggingface.co/OpenGVLab/InternVL2-4B

[lapp0]

If there's a processor class which provides inputs for model.generate, which accepts the same arguments as in transformers_vision then it's possible. AutoProcessor

Currently the default is models.transformers_vision(hub_uri, model_class=LlavaNextForConditionalGeneration, processor=LlavaNextProcessor), so you'd have to substitute the appropriate model and processor classes.

[cegutica]

Hi @lapp0, just to confirm what I understood, does this mean that any other model that doesn't have a processor class cannot be used with Outlines? If that's the case, is it possible to include these models in the future?

[lapp0]

@cegutica It's difficult to include models which have bespoke code directly into Outlines. We currently support models which have processor classes, for example

However, you can easily use outlines.processors in their code by setting the logits_processor argument. Please check this portion of Outlines documentation, and let me know if you have any questions.