README.md

QUIK

This repository contains the code for QUIK, a method for quantizing the majority of the weights and activations to 4bit post-training.

QUIK is described in the following paper: https://arxiv.org/abs/2310.09259

Install

Dependencies

• cmake
• C++ compiler (GCC/clang/...)
• nvcc

Instructions

git clone https://github.com/IST-DASLab/QUIK.git
cd QUIK
pip install -e .  # or pip install .

Example

LLama example

cd experiments
pip install -r requirements.txt
python llama.py --fp_features_num 256 --model meta-llama/Llama-2-7b-hf --hf_token <your_hf_token> --dataset c4 \ 
--w_bits 4 --w_clip --a_bits 4 --save_qmodel_path save_gptq_model_path --int8_down_proj --sim_eval --benchmark 

Benchmark will be run on all available GPUs.

Linear layer benchmarks

Linear layer benchmarks can be run with python layer_benchmark.py. One can vary input size with command line parameters.

Model adapt to QUIK

First, one has to quantize the model weights using GPTQ algorithm. In llama.py it is done with llama_sequential function. From that we get quantized weights (that are still stored in torch.float16). Then ones needs create QUIK Linear layers using qlinear.MixedQLinear.from_float that must replace original Linear layers. See llama_replace_with_kernels in llama.py. Now the quantized model is ready for use.

Fake Quantization examples

To run the fake quantization example, check fake_quant directory.

Citation

The full paper is available on arxiv. The full citation is

@article{QUIK,
  title={QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models},
  author={Ashkboos, Saleh and Markov, Ilia and Frantar, Elias and Zhong, Tingxuan and Wang, Xincheng and Ren, Jie and Hoefler, Torsten and Alistarh, Dan},
  journal={arXiv preprint arXiv:2310.09259},
  year={2023}
}