[Bug]: The deployment function requires a divisible relationship, but the model structure does not meet this requirement. What should I do?

[BrenchCC]

🐛 Describe the bug

def get_siglip_patch_grid_length(*, image_size: int, patch_size: int) -> int:
assert image_size % patch_size == 0
return image_size // patch_size

"vision_config": { --   | "hidden_act": "gelu_pytorch_tanh",   | "hidden_size": 1152,   | "image_size": 384,   | "intermediate_size": 4304,   | "layer_norm_eps": 1e-06,   | "model_type": "siglip_vision_model",   | "num_attention_heads": 16,   | "num_hidden_layers": 27,   | "patch_size": 14

[DarkLight1337]

Which HF repo are you using?

[BrenchCC]

TIGER-Lab/Mantis-8B-siglip-llama3
Perhaps I should write a new Python script to support this model.

[DarkLight1337]

It should only take a small modification since the architecture is the same as LlavaForConditionalGeneration. We just need to support loading the SigLIP model inside vLLM.

[BrenchCC]

It should only take a small modification since the architecture is the same as LlavaForConditionalGeneration. We just need to support loading the SigLIP model inside vLLM.

I have add the SigLIP to LlavaForConditionalGeneration, but the SigLIP need to get grid_length, which requires a divisible relationship. In Mantis, the LlavaForConditionalGeneration have been rewrite.

[DarkLight1337]

I'm working on a PR to add SigLIP support to LLaVA architecture.

[DarkLight1337]

It should only take a small modification since the architecture is the same as LlavaForConditionalGeneration. We just need to support loading the SigLIP model inside vLLM.

I have add the SigLIP to LlavaForConditionalGeneration, but the SigLIP need to get grid_length, which requires a divisible relationship. In Mantis, the LlavaForConditionalGeneration have been rewrite.

I think the model is still the same (since the HuggingFace repo didn't supply its own modeling_ file), just that a different vision encoder is being loaded.

[BrenchCC]

Thank for your reply.
https://github.com/TIGER-AI-Lab/Mantis/blob/main/mantis/models/mllava/modeling_llava.py This is the modeling_ file. I will continue to find how to solve this problem.

[DarkLight1337]

Thank for your reply. https://github.com/TIGER-AI-Lab/Mantis/blob/main/mantis/models/mllava/modeling_llava.py This is the modeling_ file. I will continue to find how to solve this problem.

Oh, I didn't check the repo on GitHub. Thanks for pointing that out! I guess you can work on implementing it then.

[DarkLight1337]

Can you try out #7153 and see if vLLM can run the model without modification? There is a chance this could work since that particular model uses LlavaForConditionalGeneration, not MLlavaForConditionalGeneration.