[CORE] [QUANT] Support for GPTQModel's dynamic quantization per module override/control

[Qubitium]

GPTQModel v0.9.10-dev0 main branch has merged dynamic, per layer/module support of different gptq bits, sym, desc_act using a regex style definition. This is a work in process and we are awaiting feedback before release. We are targeting both vllm and sglang compat with the quant so would like to work with vllm to see if what is the best way forward.

Previously a gptq model has a single config that applies to all layers and all modules within nested layers. This change allows pin-point targeting of different gptq quantization config for specific layers and/or specific modules within specific layers for better optimization.

Sample model: https://huggingface.co/ModelCloud/TinyLlama-1.1B-Chat-v1.0-dynamic-GPTQ-2024-8-3

full quant config for sample:

{
  "bits": 4,
  "dynamic": {
    ".*\\.(?:1[0-5])\\..*": {
      "bits": 8
    },
    ".*\\.(?:1[6-9]|20|21)\\..*": {
      "bits": 8,
      "group_size": 64
    }
  },
  "group_size": 128,
  "desc_act": true,
  "static_groups": false,
  "sym": true,
  "lm_head": false,
  "damp_percent": 0.005,
  "damp_auto_increment": 0.0015,
  "true_sequential": true,
  "model_name_or_path": "./test_dynamic_model",
  "model_file_base_name": "model",
  "quant_method": "gptq",
  "checkpoint_format": "gptq",
  "meta": {
    "quantizer": "gptqmodel:0.9.10-dev0"
  }
}

Dynamic config explained:

# sample tinyllama 1.1B model has 22 layers
# default is 4bit, group_size 128
# layer index start at 0

# last 1/2 of the layers 10-21 has 8bit vs 4bit for 0-9
# last 1/4 of the layers 16-21 has 8bit and group_size 64
dynamic = {
  # `.*\.` matches the layers_node prefix
  r".*\.(?:1[0-5])\..*": {"bits": 8,}, # match layer 10-15
  r".*\.(?:1[6-9]|20|21)\..*": {"bits": 8, "group_size": 64,}, # match layer 16-21
}

Same code to quantize using dynamic control: https://github.com/ModelCloud/GPTQModel/blob/main/tests/test_dynamic.py

Design choices:

1. Need a def table to notify quantizer (GPTQModel) and infer engine (vllm) which layers has dynamic (override) quant config.
2. Possible to generate a static all inclusive per layer/module def/table in json but content would not be human friendly as each nested layer with each nested module would need an entry. If a model has 44 layers and each layer has 6-8 modules, we are looking a t a 44x8 lines of json minimum.
3. GPTQModel decided on a design where a simple regex: str key mapped to dict[str, int or bool] for both quantization and model inference/loading. Multiple regex/dynamic pairs can be defined and for matching, the rules are looped and first one that match, is applied.
4. Upload loading, and looping over each layer/module, we check for dynamic (override) match and if matches, override the static quant config files for that layer/module.

Compat Notes:

dynamic config require that the model inference does not remerge the layers with different dyanmic/quant param values. MergedColumnParallel in Llama model in vllm for example merges mlp.gate and mlp.up. Dynamic override works but in this case, because they are fused/merged, these two layers must have exact same quant config values. Can't have one with 4bit and the other with 8bits.

TODO:

1. unit test
2. finalize design of loading so GPTQModel and vllm can agree, how to best pass/share dynamic layer/module quant override config via quantize_config json

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[mgoin]

Hi @Qubitium thanks for sharing your interesting work!

We have the notion of variable quantization already in vLLM through our compressed-tensors integration. With this we can blend integer and float quantization of weights and/or activations within a single model config in a similar explicit target or regex manner. I recommend digging into the non-uniform support we already have for compressed-tensors and fp8 methods.
It would be interesting to see if your library could export into compressed-tensors format so it would work out-of-the-box in vLLM and Transformers!

Regarding merged layers, I think the performance and complexity cost of needing to support possibly unmerging layers like QKV or GateUp is too high. I want to recommend keeping the quantization level of merged layers the same so we (and several other inference engines) don't run into this issue.

If you are still open to editing your format, I also think dynamic isn't a clear term here since there is already the notion of static or dynamic quantization, which means something else. Also, the quantization isn't changing in any dynamic way. I would recommend using a name like non-uniform quantization, since we are not performing uniform quantization anymore but have settled on a non-uniform scheme.

[Qubitium]

@mgoin Wow, I totally missed this PR. After cursory check of the https://github.com/vllm-project/vllm/pull/6515/files PR, our pr is entirely redundant. The core concept is similar including re matching. The only little advantage of this pr, and very little at this point, is minimal code-change to bootstrap gptq flexible layer/module quant.

I will need to digest the vllm pr/unit tests to test with gptqmodel export. If gptq model can integrate with compressed_config protocol, then there is zero reason for this pr.

Regarding merged layers, I think the performance and complexity cost of needing to support possibly unmerging layers like >QKV or GateUp is too high. I want to recommend keeping the quantization level of merged layers the same so we (and >several other inference engines) don't run into this issue.

Yes, this our finding as well. Merged layers should retain the same scheme.

If you are still open to editing your format, I also think dynamic isn't a clear term here since there is already the notion of >static or dynamic quantization, which means something else. Also, the quantization isn't changing in any dynamic way. I >would recommend using a name like non-uniform quantization, since we are not performing uniform quantization ?>anymore but have settled on a non-uniform scheme.

I want the config to to be compatible to vllm/sglang, and since sglang for the most part re-uses/import vllm model weight/model layers. Do not want another protocol parser so if vllm compressed_config protocol works like I think it does, then this is good base moving forward for gptqmodel as well.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Qubitium.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mgoin]

Could you please fix the precommit format and lint checks?

[Qubitium]

@mgoin Lint is being stupid here. We must check for False since None and False are two distinct states here. How do we force lint to pass here? Check comments in below code block.

I added # noqa: E712 to surpress but not working. How can we turn off this correct logic lint check? We must check for == False

# False = skip module, None = no override, else = Positive match
if self.get_dynamic_override(layer_name=prefix) == False: 

Error: vllm/model_executor/layers/quantization/gptq_marlin.py:211:16: E712 Avoid equality comparisons to `False`; use `if not ...:` for false checks

Also, our local format.sh is passing with flying colors, but ci has disconnect with locally run format.sh (ruff) causing issues. Outside the scope of this PR, I think somone at CI needs to look at why CI test and locally run format.sh are having mismatched results.

[Qubitium]

@mgoin Fixed. Turned out we had to add # noqa: to both lines if a logical line was split into two lines.

[Qubitium]

@mgoin Not ready. Found bug during our internal testing. For non-Marlin kernel code path,dynamic is not correctly applied.

[Qubitium]

@mgoin Ready for re-view. Sorry this is taking too long. We want to get it right.

Changes since your last review:

• Dynamic override get/logic moved to gptq_utils.py
• Fixed gptq cuda kernel not applying dynamic correctly: previously only Marlin kernel was fully tested
• Cleaned up related ci tests
• Lint/pre-commit CI passing

[mgoin]

Thanks for fixing the issues, I think this should be good to go if the CI is green with the GPTQ changes.

[mgoin]

Thank you! The failing tests are unrelated so we will merge shortly