[BUG] cholesky_cusolver" not implemented for 'BFloat16'

[wenhuach21]

   return func(*args, **kwargs)
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/transformers/modeling_utils.py", line 5130, in from_pretrained
    hf_quantizer.preprocess_model(
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/transformers/quantizers/base.py", line 238, in preprocess_model
    return self._process_model_before_weight_loading(model, **kwargs)
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/transformers/quantizers/quantizer_auto_round.py", line 64, in _process_model_before_weight_loading
    model, used_backends = convert_hf_model(model, target_device)
  File "/home/wenhuach/auto-round-main/auto_round/inference/convert_model.py", line 590, in convert_hf_model
    used_backends = _replace_by_quant_layers(model, layer_configs, backend, target_device, orig_backend)
  File "/home/wenhuach/auto-round-main/auto_round/inference/convert_model.py", line 372, in _replace_by_quant_layers
    new_layer = _create_quant_layer(layer, layer_backend, config, in_features, out_features)
  File "/home/wenhuach/auto-round-main/auto_round/inference/convert_model.py", line 418, in _create_quant_layer
    QuantLinear = dynamic_import_inference_linear(layer_backend, config["bits"], config["group_size"], config["sym"])
  File "/home/wenhuach/auto-round-main/auto_round/inference/backend.py", line 486, in dynamic_import_inference_linear
    return get_gptqmodel_infer_linear(backend, bits, group_size, sym)
  File "/home/wenhuach/auto-round-main/auto_round/inference/backend.py", line 524, in get_gptqmodel_infer_linear
    import gptqmodel  # pylint: disable=E0401
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/gptqmodel/__init__.py", line 19, in <module>
    from .models import GPTQModel, get_best_device
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/gptqmodel/models/__init__.py", line 18, in <module>
    from .auto import MODEL_MAP, GPTQModel
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/gptqmodel/models/auto.py", line 62, in <module>
    from ..quantization import QUANT_CONFIG_FILENAME  # noqa: E402
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/gptqmodel/quantization/__init__.py", line 19, in <module>
    from .gptq import GPTQ
  File "/home/wenhuach/anaconda3/envs/autoround/lib/python3.10/site-packages/gptqmodel/quantization/gptq.py", line 47, in <module>
    torch.linalg.cholesky(tmp_eye)
RuntimeError: "cholesky_cusolver" not implemented for 'BFloat16'

We couldn’t identify the root cause of the issue, so the straightforward workaround is to set tmp_eye to float32. This change should have no impact on the GPTQmodel.

[Qubitium]

@wenhuach21 That's so strange. I don't think we modified any core gptq math code. Which torch version are you running and are you testing this on torch + xpu or toch + cuda? Could be this method is not implemented in the xpu release of torch which we can bypass with xpu check.

We will test this on xpu + rocm + cuda to see what's going on also.

[wenhuach21]

we just found that transformers modify the torch default type.

[Qubitium]

@wenhuach21 ahhhhh....Which transformers version are you using? So I can sync on my end. I need to check if we are using any default tensor setup without dtype since we need to fix this and not assume default tensor is always fp2 coming from transformers.

[wenhuach21]

4.56.1. https://github.com/huggingface/transformers/blob/b9282355bea846b54ed850a066901496b19da654/src/transformers/modeling_utils.py#L1327 This function will change the torch default dtype based on the model's config

[Qubitium]

@Cyrilvallez Can you check if this is correct behavior? We did not expect transformers to force modify torch defaults since other projects may use use both transformers and non-transformers code in the same codebase/calls.

[Cyrilvallez]

Hey @Qubitium! I don't think the behavior of that function changed, it always temporarily changed the dtype globally. The default dtype is then changed back to what it was before in the body function, and there is also a decorator restore_default_dtype in case the function crashes during execution. I don't see anything that could have changed that behavior 🤔