Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support for quantized models? #316

Closed
bryanhpchiang opened this issue Jun 30, 2023 · 1 comment
Closed

support for quantized models? #316

bryanhpchiang opened this issue Jun 30, 2023 · 1 comment
Labels
feature request

Comments

@bryanhpchiang
Copy link

for those with limited VRAM, any plans to support quantized versions of models?
@zhuohan123 zhuohan123 mentioned this issue Jul 2, 2023
Closed
76 tasks
@LiVincent-Zhang LiVincent-Zhang mentioned this issue Aug 10, 2023
Closed
@hmellor
Copy link
Collaborator

hmellor commented Mar 6, 2024

Closing because quantisation is supported via AWQ, GPTQ and SqueezeLLM.

@hmellor hmellor closed this as completed Mar 6, 2024
jikunshang pushed a commit to jikunshang/vllm that referenced this issue Sep 30, 2024
@iboiko-habana
Change default values for decode bucket flags (vllm-project#316)
45ee586 
Change default values for decode bucket flags
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hmellor @WoosukKwon @bryanhpchiang