[DeepSeek-V3/R1] Anyone had success quantizing DeepSeek-V3 using llm-compressor?

[ashgold]

Describe the bug
A clear and concise description of what the bug is.

Expected behavior
A clear and concise description of what you expected to happen.

Environment
Include all relevant environment information:

1. OS [e.g. Ubuntu 20.04]: Ubuntu 22.04
2. Python version [e.g. 3.7]: 3.12.9
3. LLM Compressor version or commit hash [e.g. 0.1.0, f7245c8]: v0.5.1
4. ML framework version(s) [e.g. torch 2.3.1]: torch==2.5.1
5. Other Python package versions [e.g. vLLM, compressed-tensors, numpy, ONNX]: compressed-tensors v0.9.1
6. Other relevant environment information [e.g. hardware, CUDA version]:

Has anyone succeeded in quantizing DeepSeek-V3 using llm-compressor?
I tried to quantize DeepSeek-V3 using 8xH100 nodes, referring to all the issues on github, but kept getting OOMs.

I used mit-han-lab/pile-val-backup dataset (512 samples and 2048 sequence length).

Below are my attempts. All attempts resulted in an OOM.

Preparing intermediates cache: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [00:02<00:00, 186.39it/s]
(1/61): Calibrating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [00:01<00:00, 286.37it/s]
2025-05-27T15:58:31.207270-0700 | on_sequential_batch_end | INFO - Quantizing model.layers.3.mlp.experts.0.gate_proj using 1 samples
2025-05-27T15:58:31.213474-0700 | compress | METRIC - time 0.01s
2025-05-27T15:58:31.213871-0700 | compress | METRIC - GPU 0 | usage: 46.09% | total memory: 85 GB
2025-05-27T15:58:31.213931-0700 | compress | METRIC - GPU 1 | usage: 99.97% | total memory: 85 GB
2025-05-27T15:58:31.213964-0700 | compress | METRIC - GPU 2 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.213989-0700 | compress | METRIC - GPU 3 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.214014-0700 | compress | METRIC - GPU 4 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.214036-0700 | compress | METRIC - GPU 5 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.214056-0700 | compress | METRIC - GPU 6 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.214074-0700 | compress | METRIC - GPU 7 | usage: 29.38% | total memory: 85 GB
2025-05-27T15:58:31.214124-0700 | compress | METRIC - Compressed module size: 29.704192 MB
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/pipelines/sequential/helpers.py", line 53, in forward
    outputs = forward_fn(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 7, in forward
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/transformers/models/deepseek_v3/modeling_deepseek_v3.py", line 504, in forward
    hidden_states = self.mlp(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/transformers/models/deepseek_v3/modeling_deepseek_v3.py", line 215, in forward
    hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/transformers/models/deepseek_v3/modeling_deepseek_v3.py", line 201, in moe
    expert_output = expert(expert_input)
                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/transformers/models/deepseek_v3/modeling_deepseek_v3.py", line 114, in forward
    down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
                                                                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1844, in _call_impl
    return inner()
           ^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1803, in inner
    hook_result = hook(self, args, result)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/utils/hooks.py", line 93, in wrapped_hook
    return hook(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 322, in calibrate_module
    self._hessians[module], self._num_samples[module] = accumulate_hessian(
                                                        ^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/gptq_quantize.py", line 66, in accumulate_hessian
    H += inp.matmul(inp.t())
         ^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 196.00 MiB. GPU 1 has a total capacity of 79.11 GiB of which 26.88 MiB is free. Process 549622 has 79.07 GiB memory in use. Of the allocated memory 78.28 GiB is allocated by PyTorch, and 136.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 234, in on_initialize
    run_sequential(
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/pipelines/sequential/pipeline.py", line 67, in run_pipeline
    subgraph.forward(model, **inputs)
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/pipelines/sequential/helpers.py", line 55, in forward
    raise RuntimeError(
RuntimeError: Raised an exception during execution of the following code:

1
2
3
4 def forward(self, unsqueeze, arange, getitem_6, model_rotary_emb, clone, masked_fill, model_layers_2):
5 clone[(slice(None, None, None), slice(None, None, None), slice(None, None, None), slice(None, getitem_6, None))] = masked_fill; setitem = clone; getitem_6 = masked_fill = setitem = None
6 getitem_12 = model_layers_2[0]; model_layers_2 = None
7 model_layers_3 = getattr(self.model.layers, "3")(getitem_12, attention_mask = clone, position_ids = unsqueeze, past_key_value = None, output_attentions = False, use_cache = False, cache_position = arange, position_embeddings = model_rotary_emb); getitem_12 = clone = unsqueeze = arange = model_rotary_emb = None
8 return {'model_layers_3': model_layers_3}
9

This is likely due to a violation of shape assumptions made when tracing
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/home/jovyan/workspace/awq/quantize_gptq.py", line 103, in <module>
     
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 179, in oneshot
    one_shot()
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 131, in __call__
    self.apply_recipe_modifiers(
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 173, in apply_recipe_modifiers
    session.initialize(**session_kwargs)
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/core/session.py", line 118, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/core/lifecycle.py", line 105, in initialize
    data = mod.initialize(state=self.state, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/stage.py", line 83, in initialize
    modifier.initialize(state, **kwargs)
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/modifier.py", line 90, in initialize
    self.initialized_ = self.on_initialize(state=state, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 268, in on_initialize
    raise exception
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 256, in on_initialize
    run_layer_sequential(
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/pipelines/layer_sequential/pipeline.py", line 73, in run_pipeline
    callback_modifier.on_sequential_batch_end()
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 345, in on_sequential_batch_end
    loss, quantized_weight, scale, zero_point, g_idx = quantize_weight(
                                                       ^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/gptq_quantize.py", line 108, in quantize_weight
    W = W.to(dtype=GPTQ_PRECISION)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 56.00 MiB. GPU 1 has a total capacity of 79.11 GiB of which 26.88 MiB is free. Process 549622 has 79.07 GiB memory in use. Of the allocated memory 78.31 GiB is allocated by PyTorch, and 108.11 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

calculate_offload_device_map

if reserve_for_hessians=True, We couldn't use GPU.

device_map = calculate_offload_device_map(
     MODEL_ID, num_gpus=8, reserve_for_hessians=False, torch_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=device_map, torch_dtype=torch.bfloat16)

recipe = [
    GPTQModifier(
        ignore=["lm_head", "re:.*self_attn.*", "re:.*shared_experts.*", "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"],
        targets=["Linear"],
        scheme="W4A16",
        group_size=128,
        offload_hessians=False #offloading hessian makes calibration process terribly slow.
    ),
]

40GiB * 8 H100s

with init_empty_weights():
    dummy_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
    device_map = infer_auto_device_map(
        dummy_model,
        offload_buffers=True,
        max_memory={
            0: "40GiB",
            1: "40GiB",
            2: "40GiB",
            3: "40GiB",
            4: "40GiB",
            5: "40GiB",
            6: "40GiB",
            7: "40GiB",
            "cpu": "1600GiB"
        },
        no_split_module_classes=dummy_model._no_split_modules,
    )
    del dummy_model
 
 
recipe = [
    GPTQModifier(
        ignore=["lm_head", "re:.*self_attn.*", "re:.*shared_experts.*", "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"],
        targets=["Linear"],
        scheme="W4A16",
        group_size=128,
        offload_hessians=False
    ),
]

10GiB * 8 H100s

with init_empty_weights():
    dummy_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
    device_map = infer_auto_device_map(
        dummy_model,
        offload_buffers=True,
        max_memory={
            0: "10GiB",
            1: "10GiB",
            2: "10GiB",
            3: "10GiB",
            4: "10GiB",
            5: "10GiB",
            6: "10GiB",
            7: "10GiB",
            "cpu": "1600GiB"
        },
        # no_split_module_classes=dummy_model._no_split_modules,
    )
    del dummy_model
 
# Select model and load it.
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=device_map, torch_dtype=torch.bfloat16)
print(f"hf_device_map: {model.hf_device_map}")
 
 
# layers not to convert reference: https://huggingface.co/ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g-experts/blob/main/config.json
# Configure the quantization algorithm to run.
recipe = [
    GPTQModifier(
        ignore=["lm_head", "re:.*self_attn.*", "re:.*shared_experts.*", "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"],
        targets=["Linear"],
        scheme="W4A16",
        group_size=128,
        offload_hessians=False
    ),
]

dampening_frac 0.1

with init_empty_weights():
    dummy_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
    device_map = infer_auto_device_map(
        dummy_model,
        # offload_buffers=True,
        # fallback_allocation=True,
        max_memory={
            0: "30GiB",
            1: "30GiB",
            2: "30GiB",
            3: "30GiB",
            4: "30GiB",
            5: "30GiB",
            6: "30GiB",
            7: "30GiB",
            "cpu": "1600GiB"
        },
        no_split_module_classes=dummy_model._no_split_modules,
    )
    del dummy_model
 
recipe = [
    GPTQModifier(
        ignore=["lm_head", "re:.*self_attn.*", "re:.*shared_experts.*", "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"],
        targets=["Linear"],
        scheme="W4A16",
        group_size=128,
        dampening_frac=0.1, #it can reduce memory for hessian.
        offload_hessians=False #offloading hessian makes calibration process terribly slow.
    ),
]

[ehartford]

I have only had success with AutoAWQ
I will try again with LLM Compressor

[ashgold]

I have only had success with AutoAWQ
I will try again with LLM Compressor

I also performed AWQ Quantization using AutoAWQ. However, in AutoAWQ, an error occurs when the ignore pattern is added to DeepSeek-V3.

ignore=["lm_head", "re:.self_attn.", "re:.shared_experts.", "re:.mlp\.(gate|up|gate_up|down)_proj."]

[ehartford]

I spent several hours yesterday with llm-compressor and was unable to get it working for AWQ on DeepSeek-V3.

I got it working with AutoAWQ on an excellent 8x mi300x node generously lent to me by Hot Aisle, using the following code:

import os
import torch
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# Monkey patch to handle DeepSeek rotary embeddings
def patch_deepseek_rotary():
    """Patch the quantizer to handle DeepSeek models without rotary_emb"""
    import awq.quantize.quantizer
    
    original_quantize = awq.quantize.quantizer.AwqQuantizer.quantize
    
    def patched_quantize(self):
        # Temporarily set transformers version to bypass rotary_emb check
        import transformers
        original_version = transformers.__version__
        
        # Check if this is a DeepSeek model
        model_type = getattr(self.awq_model, 'model_type', '')
        if 'deepseek' in model_type.lower():
            # Temporarily downgrade version to skip rotary_emb computation
            transformers.__version__ = "4.47.0"
        
        try:
            # Call original quantize method
            original_quantize(self)
        finally:
            # Restore original version
            transformers.__version__ = original_version
    
    awq.quantize.quantizer.AwqQuantizer.quantize = patched_quantize
    print("Applied DeepSeek rotary embedding patch")

patch_deepseek_rotary()

model_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-bf16'
quant_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-awq'

quant_config = { 
    "zero_point": True, 
    "q_group_size": 64, 
    "w_bit": 4, 
    "version": "GEMM" 
}

print(f"Loading model from {model_path}...")
model = AutoAWQForCausalLM.from_pretrained(
    model_path,
    safetensors=True,  # Use safetensors if available
    device_map="cpu",  # Load on CPU first to save memory
)

print("Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

print("Starting quantization...")
print(f"Quantization config: {quant_config}")

try:
    # Quantize the model
    model.quantize(
        tokenizer, 
        quant_config=quant_config,
        calib_data="pileval",  # Use a standard calibration dataset
        max_calib_samples=128,  # Reduce if you encounter OOM
        max_calib_seq_len=512,  # Reduce if you encounter OOM
    )
    
    print("Quantization completed successfully!")
    
    # Save quantized model
    print(f"Saving quantized model to {quant_path}...")
    model.save_quantized(quant_path)
    tokenizer.save_pretrained(quant_path)
    
    print(f'Model is quantized and saved at "{quant_path}"')
    
except Exception as e:
    print(f"Error during quantization: {e}")

I suggest renaming this issue to "Enable AWQ quantization for Deepseek-V3 architecture"

[ehartford]

Why llm-compressor Fails with DeepSeek-V3/R1

Root Cause: MoE Architecture Graph Compilation

The failure occurs during llm-compressor's module analysis and recipe compilation phase when trying to build a computation graph for the model.

Specific Technical Issues:

1. MoE Scale Overwhelms Module Discovery

   • 256 experts × 59 layers = 15,104 expert modules
   • Each expert has gate_proj, up_proj, down_proj = 45,312 Linear modules
   • Module tree traversal with regex matching becomes computationally explosive
   • The code may be stuck enumerating or pattern-matching all these modules

2. Dynamic Expert Routing vs Static Graph

   # What llm-compressor tries to compile:
   graph = trace_model(model, sample_inputs)  # Expects deterministic execution
   
   # What actually happens in MoE:
   selected_experts = top_k(router_scores, k=8)  # Different experts each time!

   • Can't create a static computation graph when expert selection is data-dependent
   • Tracer waits forever for a stable execution path

3. Non-standard Architecture Patterns

   • kv_a_proj_with_mqa doesn't match standard attention patterns
   • Two-stage projections (q_a→q_b, kv_a→kv_b) break transformer assumptions
   • Special layer 61 with embedded shared head confuses layer detection

4. Regex Pattern Matching at Scale

   # Your ignore patterns:
   ignore=["lm_head", "re:.*self_attn.*", "re:.*shared_experts.*", ...]
   
   # Must check against ~45K modules
   for module_name in all_45k_modules:
       for pattern in ignore_patterns:
           if re.match(pattern, module_name):  # O(n×m) with large constants

5. Module Hook Registration Explosion

   • AWQ needs to register hooks for calibration
   • Attempting to register hooks on 15K+ expert modules may cause memory/performance issues
   • The registration process itself might be hanging

Why AutoAWQ Works:

• Simpler module discovery (likely just finds Linear layers directly)
• No complex graph compilation phase
• Handles MoE layers as regular modules without special analysis
• Doesn't try to optimize based on architecture understanding

Recommended Fix:

Add a fast-path for MoE models that skips detailed analysis:

if detected_architecture == "moe" or num_modules > 10000:
    use_simple_discovery = True  # Skip graph compilation

The core issue is that llm-compressor's sophisticated analysis features don't scale to models with 15,000+ expert modules, causing it to hang during the preparation phase before any actual quantization work begins.

[dsikka]

@brian-dellabetta can you take a look?

[dsikka]

I spent several hours yesterday with llm-compressor and was unable to get it working for AWQ on DeepSeek-V3.

I got it working with AutoAWQ on an excellent 8x mi300x node generously lent to me by Hot Aisle, using the following code:

import os
import torch
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# Monkey patch to handle DeepSeek rotary embeddings
def patch_deepseek_rotary():
    """Patch the quantizer to handle DeepSeek models without rotary_emb"""
    import awq.quantize.quantizer
    
    original_quantize = awq.quantize.quantizer.AwqQuantizer.quantize
    
    def patched_quantize(self):
        # Temporarily set transformers version to bypass rotary_emb check
        import transformers
        original_version = transformers.__version__
        
        # Check if this is a DeepSeek model
        model_type = getattr(self.awq_model, 'model_type', '')
        if 'deepseek' in model_type.lower():
            # Temporarily downgrade version to skip rotary_emb computation
            transformers.__version__ = "4.47.0"
        
        try:
            # Call original quantize method
            original_quantize(self)
        finally:
            # Restore original version
            transformers.__version__ = original_version
    
    awq.quantize.quantizer.AwqQuantizer.quantize = patched_quantize
    print("Applied DeepSeek rotary embedding patch")

patch_deepseek_rotary()

model_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-bf16'
quant_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-awq'

quant_config = { 
    "zero_point": True, 
    "q_group_size": 64, 
    "w_bit": 4, 
    "version": "GEMM" 
}

print(f"Loading model from {model_path}...")
model = AutoAWQForCausalLM.from_pretrained(
    model_path,
    safetensors=True,  # Use safetensors if available
    device_map="cpu",  # Load on CPU first to save memory
)

print("Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

print("Starting quantization...")
print(f"Quantization config: {quant_config}")

try:
    # Quantize the model
    model.quantize(
        tokenizer, 
        quant_config=quant_config,
        calib_data="pileval",  # Use a standard calibration dataset
        max_calib_samples=128,  # Reduce if you encounter OOM
        max_calib_seq_len=512,  # Reduce if you encounter OOM
    )
    
    print("Quantization completed successfully!")
    
    # Save quantized model
    print(f"Saving quantized model to {quant_path}...")
    model.save_quantized(quant_path)
    tokenizer.save_pretrained(quant_path)
    
    print(f'Model is quantized and saved at "{quant_path}"')
    
except Exception as e:
    print(f"Error during quantization: {e}")

I suggest renaming this issue to "Enable AWQ quantization for Deepseek-V3 architecture"

@ehartford FYI, as indicated in the ReadMe, AWQ support in 0.5.1 is preliminary with only experimental support provided thus far. Enhanced support will be available in upcoming releases

[ashgold]

I spent several hours yesterday with llm-compressor and was unable to get it working for AWQ on DeepSeek-V3.

I got it working with AutoAWQ on an excellent 8x mi300x node generously lent to me by Hot Aisle, using the following code:

import os
import torch
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# Monkey patch to handle DeepSeek rotary embeddings
def patch_deepseek_rotary():
    """Patch the quantizer to handle DeepSeek models without rotary_emb"""
    import awq.quantize.quantizer
    
    original_quantize = awq.quantize.quantizer.AwqQuantizer.quantize
    
    def patched_quantize(self):
        # Temporarily set transformers version to bypass rotary_emb check
        import transformers
        original_version = transformers.__version__
        
        # Check if this is a DeepSeek model
        model_type = getattr(self.awq_model, 'model_type', '')
        if 'deepseek' in model_type.lower():
            # Temporarily downgrade version to skip rotary_emb computation
            transformers.__version__ = "4.47.0"
        
        try:
            # Call original quantize method
            original_quantize(self)
        finally:
            # Restore original version
            transformers.__version__ = original_version
    
    awq.quantize.quantizer.AwqQuantizer.quantize = patched_quantize
    print("Applied DeepSeek rotary embedding patch")

patch_deepseek_rotary()

model_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-bf16'
quant_path = '/home/hotaisle/workspace/models/DeepSeek-R1-0528-awq'

quant_config = { 
    "zero_point": True, 
    "q_group_size": 64, 
    "w_bit": 4, 
    "version": "GEMM" 
}

print(f"Loading model from {model_path}...")
model = AutoAWQForCausalLM.from_pretrained(
    model_path,
    safetensors=True,  # Use safetensors if available
    device_map="cpu",  # Load on CPU first to save memory
)

print("Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

print("Starting quantization...")
print(f"Quantization config: {quant_config}")

try:
    # Quantize the model
    model.quantize(
        tokenizer, 
        quant_config=quant_config,
        calib_data="pileval",  # Use a standard calibration dataset
        max_calib_samples=128,  # Reduce if you encounter OOM
        max_calib_seq_len=512,  # Reduce if you encounter OOM
    )
    
    print("Quantization completed successfully!")
    
    # Save quantized model
    print(f"Saving quantized model to {quant_path}...")
    model.save_quantized(quant_path)
    tokenizer.save_pretrained(quant_path)
    
    print(f'Model is quantized and saved at "{quant_path}"')
    
except Exception as e:
    print(f"Error during quantization: {e}")

I suggest renaming this issue to "Enable AWQ quantization for Deepseek-V3 architecture"

Have you tried applying the ignore pattern I mentioned with AutoAWQ? When I tried it, it didn't work out well due to an error.

[brian-dellabetta]

Hi everyone, you can track our resolution to these issues around AWQ and MoEs in #1444 . This, combined with #1419 and #1263 should go a long way in reducing AWQ runtime and memory requirements. Several updates to our initial port of AutoAWQ have landed, but these still need to land before AWQ will work well with large MoE models.

We plan to do a release with broader announcement on AWQ when these land