HFQuantizer implementation for compressed-tensors library

[bfineran]

This PR adds an HFQuantizer for the compressed-tensors library.

Supported quantization features include:

• FP8, INT4, INT8 (for Q/DQ arbitrary precision is allowed for INT)
• Activation quantization (static)
• Dynamic per-token activation quantization
• Supports quantization of arbitrary layer types

Compressed tensors supports running in a Q/DQ format with transformer models and running compressed within vLLM (running compressed with transformers on the roadmap).

This initial PR includes a HFQuantizer and QuantizationConfig inplementations as well as a simple test. Documentation is being added.

To run with this branch, compressed-tensors needs to be currently installed from a development branch (can be released to pypi prior to landing this PR pending support from the transformers team) - pip install https://github.com/neuralmagic/compressed-tensors.git@rename_config.

Happy to provide any further information needed.

Sample model load:

from transformers import AutoModelForCausalLM
compressed_tensors_model = AutoModelForCausalLM.from_pretrained("nm-testing/tinyllama-oneshot-w4a16-group128-v3")

Suggested Reviewers:
@SunMarc @younesbelkada

[SunMarc]

Hi @bfineran, thanks for contributing and sorry for the delay ! From the PR, I see that we are decompressing the model at after loading the quantized model in order to run it. What would it take to run compressed model on transformers ? I don't mind merging this first but I want to make sure that we enable users to quantized model using compressed-library in the end. Happy to discuss more on how to collaborate together over slack if you want !

[bfineran]

Hi @SunMarc right now running compressed is WIP - we've prioritized a very flexible Q/DQ environment to enable a wide range of quantization settings and will likely roll out running compressed scenario by scenario. Will reach out over slack to discuss more about the project and will also update with additional documentation soon.

[SunMarc]

Thanks for your great work @bfineran ! The PR is mostly good on my side.
This new quantizer relies heavily on your own functions, so you will need to be a bit careful to not introduce breaking changes in the future (e.g. apply_quantization_config ) with transformers. However, I guess it can makes sense since compressed-tensors deals with many compressed format + no need to commit to transformers.
There is just a blocker on the loading part of the quantizer that needs to be fixed !

[SunMarc]

Can you add the following to CompressedTensorsConfig ? This way we can print the quantization config

    def to_diff_dict(self) -> Dict[str, Any]:
        """
        Removes all attributes from config which correspond to the default config attributes for better readability and
        serializes to a Python dictionary.
        Returns:
            `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance,
        """
        config_dict = self.to_dict()

        # get the default config dict
        default_config_dict = CompressedTensorsConfig().to_dict()

        serializable_config_dict = {}

        # only serialize values that differ from the default config
        for key, value in config_dict.items():
            if value != default_config_dict[key]:
                serializable_config_dict[key] = value

        return serializable_config_dict

[SunMarc]

Thanks for your patience and for iterating @bfineran @Satrat ! This looks good ! I left a few minor comments. I would be nice on your side to add test to check if everything works as expected also. We delegate a lot of stuff on your side as we are using the apply_quantization_config function !

[SunMarc]

Gentle ping @ArthurZucker

[SunMarc]

Can you rebase on main @Satrat ? The failing might are probably due to that

[jvlinsta]

Eagerly awaiting this! Great work @neuralmagic team ;)

[Satrat]

Can you rebase on main @Satrat ? The failing might are probably due to that

@SunMarc done! Looks like the tests are passing now after rebasing

[hyaticua]

Very eagerly awaiting this merge. Thanks to everyone in involved!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

LGTM, I think it would be cool to improve the doc a bit and mention:

• this is only for inference (for now? 👀 )
• you cannot save back the loaded model (right?)
• potential use-case (by that I mean loading a pre-quantized checkpoint is faster than loading non quantized, than quantizing).
• for the format supported, I am curious about which FP8 and int4 and int8 (https://github.com/huggingface/optimum-quanto has int4 and int8, https://github.com/pytorch/ao/blob/main/torchao/prototype/low_bit_optim/subclass_4bit.py also has int4, would be nice to understand which "backend" it uses (of course, it's compressed-tensors, but a small explanation of that would help! ) and link to these types: basically help users!
  🤗

[ArthurZucker]

What are the available formats?

[dsikka]

@ArthurZucker this includes the different compression formats, depending on how the model is quantized/saved on disk, including:

1. dense
2. int-quantized
3. float-quantized
4. pack-quantized
5. marlin-24

[ArthurZucker]

If you have example use-cases of the config's different parameters and why they should be use would be awesome here as well!

[dsikka]

@ArthurZucker Thank you for your feedback! We've updated the compressed_tensors.md addressing the aforementioned points.

[SunMarc]

Thanks for iterating on the docs ! LGTM ! Since I see that you release a new version of compressed_tensors, I'll merge this PR ! After this PR is merged @Satrat, could you modify the quantization_config of neural magic models to use compressed_tensors as we talked before ?