Initial CompressedTensors config + Activation Quantization support for static W8A8 per tensor

[dsikka]

Adding layer_name

• Depending on how we end up parsing ignore and targets (layer_name vs layer_type) we may not need layer_name to be added to the linear_method. Will experiment using a compressed-tensors function in a follow-up PR

Summary

• Initial implementation for Compressed Config support + Activation Quantization for static per tensor w8a8
• Includes fused kernels added by @varun-sundar-rabindranath

Testing/Sample Script:

from vllm import LLM, SamplingParams
import torch

# Sample prompts.
prompts = [
    "Hello, my name is",
    "The capital of France is",
    "The US president is",
    "The future of AI is"
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.80, top_p=0.95)

# Create an LLM.
llm = LLM(model="nm-testing/tinyllama-one-shot-static-quant-test", enforce_eager=True, dtype=torch.float32, quantization="sparseml")

outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Next Steps:

• Verification of the different inputs expected for targets and ignore --> use functions to parse the layer names which can be shared by sparseml and vllm; would live in compressed tensors (https://github.com/neuralmagic/compressed-tensors/blob/67005d76107d4659787f1efd53fe7e6b1d192818/src/compressed_tensors/quantization/lifecycle/apply.py#L86)
• Updates to further optimize fake qunat

[robertgshaw2-neuralmagic]

Minor nits. Generally looks good. Biggest things are:

• Clean up the StaticW8A8Scheme
• Remove unnecessary cuda stuff

@varun-sundar-rabindranath can you do a quick audit of the kernels and let us know which can be removed?

[robertgshaw2-neuralmagic]

why sparseml and not compressed-tensors?

[dsikka]

To comply with the vllm input handling. The quantization method listed in the sparsmel model config is sparseml so we can change this if we change the value listed in the config.

[robertgshaw2-neuralmagic]

Can you coordinate with Sara on this?

I think it should be compressed-tensors in the HF config

[Satrat]

I'm fine with the update, but it is currently called "sparseml" in the compressed-tensors repo so it will need to be updated there too