70B W4A16 OOM on BMG

[mengniwang95]

RTN mode for llama2-70b

[wenhuach21]

It would be better to attach more env info or hardware info as we verified fp8 woq which worked well on BMG for 70B models

[mengniwang95]

auto-round==0.6.0+ipex 2.7.0+xpu

cmd:
auto-round --model /model/Llama-2-70b-chat-hf --bits 4 --group_size 128 --format auto_awq --output_dir ./tmp_autoround

�[38;20m2025-08-05 15:31:54 INFO llm.py L383: start to quantize /model/Llama-2-70b-chat-hf�[0m

Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]
Loading checkpoint shards: 7%|▋ | 1/15 [00:00<00:09, 1.54it/s]
Loading checkpoint shards: 13%|█▎ | 2/15 [00:01<00:08, 1.49it/s]
Loading checkpoint shards: 20%|██ | 3/15 [00:02<00:08, 1.48it/s]
Loading checkpoint shards: 27%|██▋ | 4/15 [00:02<00:07, 1.50it/s]
Loading checkpoint shards: 33%|███▎ | 5/15 [00:03<00:06, 1.50it/s]
Loading checkpoint shards: 40%|████ | 6/15 [00:04<00:05, 1.50it/s]
Loading checkpoint shards: 47%|████▋ | 7/15 [00:04<00:05, 1.50it/s]
Loading checkpoint shards: 53%|█████▎ | 8/15 [00:05<00:04, 1.51it/s]
Loading checkpoint shards: 60%|██████ | 9/15 [00:05<00:03, 1.51it/s]
Loading checkpoint shards: 67%|██████▋ | 10/15 [00:06<00:03, 1.52it/s]
Loading checkpoint shards: 73%|███████▎ | 11/15 [00:07<00:02, 1.51it/s]
Loading checkpoint shards: 80%|████████ | 12/15 [00:07<00:01, 1.51it/s]
Loading checkpoint shards: 87%|████████▋ | 13/15 [00:08<00:01, 1.51it/s]
Loading checkpoint shards: 93%|█████████▎| 14/15 [00:09<00:00, 1.51it/s]
Loading checkpoint shards: 100%|██████████| 15/15 [00:09<00:00, 1.61it/s]
�[38;20m2025-08-05 15:32:13 INFO autoround.py L271: using torch.float16 for quantization tuning�[0m
�[38;20m2025-08-05 15:32:13 INFO autoround.py L276: 'enable_torch_compile' is set to False by default. Enabling it can reduce tuning cost by 20%, but it might throw an exception.�[0m
�[38;20m2025-08-05 15:32:13 INFO autoround.py L1294: start to cache block inputs�[0m
Traceback (most recent call last):
File "/usr/local/bin/auto-round", line 8, in
sys.exit(run())
File "/usr/local/lib/python3.10/dist-packages/auto_round/main.py", line 41, in run
tune(args)
File "/usr/local/lib/python3.10/dist-packages/auto_round/script/llm.py", line 502, in tune
model, folders = autoround.quantize_and_save(export_dir, format=args.format)
File "/usr/local/lib/python3.10/dist-packages/auto_round/autoround.py", line 669, in quantize_and_save
model, _ = self.quantize()
File "/usr/local/lib/python3.10/dist-packages/auto_round/autoround.py", line 1295, in quantize
all_inputs = self.try_cache_inter_data_gpucpu(all_first_block_names, self.nsamples, layer_names=layer_names)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/auto_round/autoround.py", line 1729, in try_cache_inter_data_gpucpu
self.model = self.model.to(self.device)
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3851, in to
return super().to(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1355, in to
return self._apply(convert)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 915, in _apply
module._apply(fn)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 915, in _apply
module._apply(fn)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 915, in _apply
module._apply(fn)
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 942, in _apply
param_applied = fn(param)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1341, in convert
return t.to(
RuntimeError: Native API failed. Native API returns: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

[wenhuach21]

you are not using rtn mode, rtn mode need '--iters 0'

[mengniwang95]

yes, tuning is required for mlperf

[wenhuach21]

then follow the reduce the gpu memory usage section in step by step.md to tune hyperparameters or use multiple gpus. There is nothing we could do in AutoRound side now

[xin3he]

Depends on intel/torch-xpu-ops#2232