Optimize RAM usage of AutoScheme

[wenhuach21]

Models	Scheme	VRAM Cost (torch compile)	Time Cost torch compile	VRAM Cost wo torch compile	Time Cost wo torch compile
Qwen3-8B	W2A16/W4A16/W8A16	34G	30s * len of options	61G	40s * len of options
Qwen3-8B	MXFP4/MXFP8	36G	60s * len of options	54G	120s * len of options
Qwen3-8B	GGUF*	54G	30s * len of options	50G	23S * len of options
Qwen3-32B	W2A16/W4A16/W8A16	OOM with 240G	---	OOM with 240G	---
Qwen3-32B	MXFP4/MXFP8	160G	200s * len of options	200G	240s * len of options
Qwen3-32B	GGUF*	210G	80s * len of options	200G	60s * len of options

[wenhuach21]

VRAM usage has been optimized #944

Models	Scheme	VRAM Cost	Time Cost
Qwen3-8B	W2A16 / W4A16 / W8A16	14G	60s * len of options
Qwen3-8B	MXFP4 / MXFP8	18G	60s * len of options
Qwen3-8B	GGUF*	14G	80s * len of options
Qwen3-32B	W2A16 / W4A16 / W8A16	29G	180s* len of options
Qwen3-32B	MXFP4 / MXFP8	29G	180s* len of options
Qwen3-32B	GGUF*	18G	300s * len of options
Llama-3.3-70B	W2A16 / W4A16 / W8A16	32G	420s * len of options

[kalle07]

with new version:

(venv) c:\Users\xxx\Documents\python\autoround\llama.cpp\gguf-py>pip show auto-round
Name: auto_round
Version: 0.8.0
Summary: Repository of AutoRound: Advanced Weight-Only Quantization Algorithm for LLMs
Home-page: https://github.com/intel/auto-round
Author: Intel AIPT Team
Author-email: wenhua.cheng@intel.com, weiwei1.zhang@intel.com, heng.guo@intel.com
License: Apache 2.0
Location: c:\Users\kallemst\Documents\python\autoround\venv\Lib\site-packages
Requires: accelerate, datasets, lm-eval, numba, numpy, packaging, pillow, py-cpuinfo, sentencepiece, tbb, threadpoolctl, torch, tqdm, transformers

i can not use auto-round q2ks it without "--disable_opt_rtn" ... if i disable it overload my 16GB VRAM
7B is 30GB in size - is that the only option with a bit loss on my system?

import subprocess
from pathlib import Path
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
# from auto_round_mllm import Autoround
from pathlib import Path
import torch

# Basisverzeichnis
script_dir = Path(__file__).parent

# Modellpfad
model_dir = Path(r"e:\NextCoder-7B\safetensors")
output_dir = script_dir / "tmp_auto_next"

# Pfad zur EXE
exe_path = Path(r"c:\Users\xxx\Documents\python\autoround\venv\Scripts\auto_round_mllm.exe")

# Argumente – je nachdem, was das Tool erwartet
cmd = [
    str(exe_path),
    "--model", str(model_dir),
    "--output", str(output_dir),
    "--bits", "4",
    "--group_size", "128",
    #"--format", "auto_round",
    "--disable_opt_rtn",
    "--batch_size", "4",
    "--low_gpu_mem_usage",
    "--iters", "0",
    "--nsample", "512",
    "--seqlen", "2048",
    "--model_dtype", "fp16",
    "--scale_dtype", "fp16",
    "--format", "gguf:q2_k_s",
    #"--quant_nontext_module",
    #"--enable_torch_compile",
]

# Hier keine capture_output — sonst wird nichts angezeigt
result = subprocess.run(cmd)

print(f"\nProzess beendet mit Code: {result.returncode}")

[wenhuach21]

is v0.7 working fine while v0.8 fails? I suspect both may have failed.

The PR I mentioned earlier (#886) hasn’t been merged into v0.8 yet, since the code was frozen for the 0.8 release at that time. It will be included in v0.9, which is expected to be released in 1–2 weeks.

Besides, you may try to install the main branch on your system.

[kalle07]

no, but I can only use auto-round gguf:q2_k_s it with "--disable_opt_rtn" ... if i disable, it overload my 16GB VRAM.
model 7B is 30GB in size - is that the only option (with a bit loss) on my system?
or is that optimized in v0.9?

[wenhuach21]

Got it. Since we don’t have a Windows + CUDA setup, I can’t verify it directly. I believe this issue was fixed in PR #886, but I’m not entirely sure. The fix will be included in v0.9.

[kalle07]

i see ...

this line
INFO base.py L832: change scale_dtype to torch.float32
this comes from llama.cpp?
iv no idea of all ... but while Q2 or Q3 ist float32 necessary?
i can not change that with your parameters.

[wenhuach21]

Although LlamaCPP supports the BF16 compute dtype, we found some issues on our tested CPUs. Therefore, similar to LlamaCPP, we use the more common FP32 compute dtype. Under this compute type, there is no advantage in using other scale dtypes such as BF16 or FP16.