Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Support tensor parallelism for GGUF quantization #7520

Merged
merged 9 commits into from
Aug 19, 2024
Merged

[Core] Support tensor parallelism for GGUF quantization #7520

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
528b358
remove gguf tp raise
Isotr0py Aug 9, 2024
462783e
save draft
Isotr0py Aug 14, 2024
c8ffa77
fix tp=2 inference
Isotr0py Aug 14, 2024
ebf5378
code format
Isotr0py Aug 15, 2024
ac9dbe3
add gguf tp test
Isotr0py Aug 16, 2024
c2f8101
Merge branch 'vllm-project:main' into gguf-tp
Isotr0py Aug 16, 2024
3e3dc91
fix gguf test prompt format
Isotr0py Aug 18, 2024
bc61e56
code format
Isotr0py Aug 18, 2024
7292ba1
Merge branch 'vllm-project:main' into gguf-tp
Isotr0py Aug 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
remove gguf tp raise
  • Loading branch information
@Isotr0py
Isotr0py committed Aug 9, 2024
commit 528b35885c17f29b9d6911510ccddb1264f668d1
3 changes: 0 additions & 3 deletions vllm/model_executor/layers/quantization/gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,6 @@ def get_config_filenames(cls) -> List[str]:

@classmethod
def from_config(cls, config: Dict[str, Any]) -> "GGUFConfig":
if get_tensor_model_parallel_world_size() > 1:
raise ValueError(
"GGUF quantization hasn't supported tensor parallelism yet.")
return cls()

def get_quant_method(self, layer: torch.nn.Module,
Expand Down