Fix AutoConfig and AutoModel support for Llava-Next-Video

[TKONIY]

What does this PR do?

Bug fix: Llava-Next-Video model cannot be loaded by AutoConfig and AutoModel.

• Support AutoConfig: Unify the model_type with the config.json in hugging face.
• Support AutoModel: Add llava* models to MODEL_MAPPING.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp

[zucchini-nlp]

Oh, I see now. I changed hub models to underscore naming but didn't know transformers had it this way. IMO the best way here is to modify files on the hub to use model_type=llava-next-video. Feel free to open a PR on the hub :)

For AutoModel we don't support VLMs because they have no base model (i.e. without lm_head) and the ConditionalGeneration models should be loaded with AutoModelForVision2Seq.

[TKONIY]

Oh, I see now. I changed hub models to underscore naming but didn't know transformers had it this way. IMO the best way here is to modify files on the hub to use model_type=llava-next-video. Feel free to open a PR on the hub :)

For AutoModel we don't support VLMs because they have no base model (i.e. without lm_head) and the ConditionalGeneration models should be loaded with AutoModelForVision2Seq.

Thanks! I have removed the AutoModel comit.

But I think that model_type="llava_next_video" would be better because the "llava_next" model type name is also underscore. And the model_type in the implementation files were also "llava_next_video", which need to be consistent.

transformers/src/transformers/models/llava_next_video/configuration_llava_next_video.py

Line 95 in 5fd7ca7

model_type = "llava_next_video"

transformers/src/transformers/models/llava_next_video/diff_llava_next_video.py

Line 104 in 5fd7ca7

model_type = "llava_next_video"

[zucchini-nlp]

@TKONIY fair enough. Can you run make fix-copies to make th CI happy?

[zucchini-nlp]

LGTM, thanks! Retriggered the tests which were failing with timeout

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for fixing!

[TKONIY]

Dear @amyeroberts, do you know when this commit will be merged into a released version of Transformers? It is important since vLLM's video support can be merged into the main branch only when Transformers can correctly load Llava-Next-Video by AutoConfig, which is fixed by this commit. And the pending video support also blocks other PRs. Therefore, it would be great if you could help us by adding this fixing to the next release. Thank you!

[amyeroberts]

@TKONIY All commits currently merged into main will be part of the next minor release (but not necessarily a patch release). Our release schedule is monthly, so v4.45 will probably be this week or the next

[TKONIY]

@amyeroberts Thanks for the information. Really appreciate your work!