Fix hqq skipped modules and dynamic quant

[mobicham]

What does this PR do?

This PR tries to fix a couple of issues with hqq that popped up lately:

• skip_modules: skip modules was not working properly. For example, if you specify the vision tower to skip during the quantization step, it was ignored. Now it works.
• Some skipped modules were creating issues when the quantized model is loaded.
• dynamic_config=True has been broken lately after some changes in transformers. This is now fixed.

You can now skip hqq quantization for the vision tower in VLMs as follows:

import torch
device        = 'cuda:0'
compute_dtype = torch.bfloat16
cache_dir     = None
model_id      = 'google/gemma-3-12b-it'

########################################################################
#Load model
from transformers import HqqConfig, Gemma3ForConditionalGeneration, AutoProcessor

processor = AutoProcessor.from_pretrained(model_id, cache_dir=cache_dir)

#quant_config = HqqConfig(nbits=4, group_size=64, axis=1, skip_modules=['lm_head', 'vision_tower'])

q4_config = {'nbits':4, 'group_size':64}
q3_config = {'nbits':3, 'group_size':64}
quant_config  = HqqConfig(dynamic_config={
  'self_attn.q_proj':q4_config,
  'self_attn.k_proj':q4_config,
  'self_attn.v_proj':q4_config,
  'self_attn.o_proj':q4_config,

  'mlp.gate_proj':q3_config,
  'mlp.up_proj'  :q3_config,
  'mlp.down_proj':q3_config,
}, skip_modules=['lm_head', 'vision_tower'])

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=compute_dtype,
    attn_implementation="sdpa",
    cache_dir=cache_dir,
    quantization_config=quant_config,
    device_map="cuda",
)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=compute_dtype)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=128, do_sample=False)[0][input_len:]
    decoded    = processor.decode(generation, skip_special_tokens=True)

print(decoded)

⚠️ There's still currently an issue related to saving/loading some quantized hqq models because transformers doesn't make sure the loaded state dict of a certain module contains all the necessary attributes.
Currently, I am using my custom solution which works well - it makes sure that a safetensors chunk contains all the attributes for a given module before saving. We should have a open a separate issue for this though.

Who can review?

@ArthurZucker @SunMarc

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[SunMarc]

Thanks ! Can you add a test to make sure that skip_modules works ?

[mobicham]

Thanks ! Can you add a test to make sure that skip_modules works ?

Thank you @SunMarc! Sure, added test_model_serialization_dynamic_quant_with_skip which tests both dynamic quant and the skip.