Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: vllm overrides transformer's Autoconfig for mllama #9076

Closed
1 task done
lyuqin-scale opened this issue Oct 4, 2024 · 7 comments · Fixed by #9083
Closed
1 task done

[Bug]: vllm overrides transformer's Autoconfig for mllama #9076

lyuqin-scale opened this issue Oct 4, 2024 · 7 comments · Fixed by #9083
Labels
bug Something isn't working

Comments

@lyuqin-scale
Copy link

Your current environment

vllm 0.6.2

Model Input Dumps

No response

🐛 Describe the bug

This line overrides transformer's autoconfig for mllama, which should be removed

_CONFIG_REGISTRY_OVERRIDE_HF: Dict[str, Type[PretrainedConfig]] = {

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@lyuqin-scale lyuqin-scale added the bug Something isn't working label Oct 4, 2024
@heheda12345
Copy link
Collaborator

We override it only for changing is_encoder_decoder=True, as described here. Other parts are not modified.

@lyuqin-scale
Copy link
Author

lyuqin-scale commented Oct 4, 2024

@heheda12345 but it does change the behavior of AutoConfig if vllm is imported together
image

@lyuqin-scale
Copy link
Author

comparing with
image

@lyuqin-scale
Copy link
Author

due to the above, transformer's AutoModelForPreTraining.from_pretrained failed
image

@lyuqin-scale
Copy link
Author

and if we don't import vLLM, it loads successfully. However, we need to import both transformers and vLLM, and use transformer to load model at runtime for some tasks
image

@heheda12345
Copy link
Collaborator

For a quick fix, you can do something like here and here

We are thinking of how to make it cleaner.

@lyuqin-scale
Copy link
Author

lyuqin-scale commented Oct 4, 2024

We've had temporary workaround by explicitly using transformers.MllamaForConditionalGeneration to load the mllama model. But we believe a fix from vLLM side could benefit the other vLLM users who aren't aware of this override

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants