Skip to content

[Misc] Slight improvement of the BNB #19418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 10, 2025
Merged

[Misc] Slight improvement of the BNB #19418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
973c6f0
Done
jeejeelee Jun 10, 2025
804565a
Update vllm/model_executor/model_loader/bitsandbytes_loader.py
Isotr0py Jun 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions vllm/model_executor/layers/quantization/bitsandbytes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,7 @@ def get_min_capability(cls) -> int:

@staticmethod
def get_config_filenames() -> list[str]:
return [
"adapter_config.json",
]
return []
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect config , delete it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This change removes adapter_config.json from the list of configuration filenames for BitsAndBytes. Could you please provide some context for this modification? For example, was this file no longer necessary for BNB configurations, or is this part of a broader update to how adapters are handled with BitsAndBytes quantization?


@classmethod
def from_config(cls, config: dict[str, Any]) -> "BitsAndBytesConfig":
Expand Down
3 changes: 2 additions & 1 deletion vllm/model_executor/model_loader/bitsandbytes_loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,8 @@ def _unquantized_generator(self, hf_weights_files, use_safetensors,
def _get_bnb_target_modules(self, model: nn.Module) -> None:

for name, module in model.named_modules():
if isinstance(module, (LinearBase, )):
if (isinstance(module, LinearBase) and
hasattr(module.quant_method, "quant_config")):
if modules_info := self.modules_mapping.get_sub_modules(name):
# Map vllm's names to transformers's names.
rep_name, sub_modules = modules_info
Expand Down