[Feature] Log/info/Save/Restore quantization steps

[mratsim]

Inspired by trying to push through #1409 or gptq experiments.

It's incredibly frustrating to fix quantization workflows to have new failures force you to redo hours of quantization.

Some stories:

1. AWQ quantization is done on CPU then GPU, I tried to workaround [AWQ] Insane memory requirement: over 900GB for 32B model #1409 by increasing swap space but:
   1. When killed by Linux OOM, the whole terminal is killed so there is no trace of what sample #ID led to OOM. I.e. if left unattended, I have to do guesswork because there is no log available anymore.
   2. After the CPU OOM is fixed, I get a GPU OOM and so have to redo 40min of CPU computation (on overclocked Ryzen 9 9950X). It would be much much much more user-friendly to do a GPU test run at the very beginning, ensuring ahead of time that all hardware requirements are met instead of being surprised after people committed time.
2. GPTQ quantization can have errors due to numerical instability and finding no solution with hessian.
   1. Those should definitely be logged to disk.
   2. It would be great time savings to be allowed to restart GPTQ from layers that converged and allow more samples or more seq lengths or a different dampening fraction for the follow-up layers that failed.

[brian-dellabetta]

Hi @mratsim , thank you for the report, this is helpful for us to understand pain points of using llm-compressor.

Regarding your OOM errors, #1426 and neuralmagic/compressed-tensors#301 should go a long way to resolving these. The latter has some nice memory usage charts to show we shouldn't hit OOM when saving.

Regarding GPTQ, I think the concern with offloading hessians is the large storage requirement, but maybe a few could be logged, cc @kylesayrs

Regarding checkpointing, that could be a good feature to add to the sequential pipelines, but we'd have to think about how to expose it. Hopefully these latest enhancements will make the remove the need for a lot of this

[brian-dellabetta]

Hi @mratsim , we have released a 0.6.0 version of llm-compressor -- https://pypi.org/project/llmcompressor/

This should resolve the OOM errors you were hitting with AWQ, and model saving at the end will not require more memory than the current model in memory. Some key PRs that have landed since your post:

• Use model compression pathways #1419
• AWQModifier fast resolve mappings, better logging, MoE support #1444
• AWQ QuantizationMixin + SequentialPipeline #1426
• [AWQ] Support accumulation for reduced memory usage #1435
• [Performance] Sequential onloading #1263

i will mark this as resolved for now, please re-open if you hit further issues. We have some GPTQ improvements in-flight as well:

• Use torch.compile to speed up GPTQ algo #1561