Sudden nan values from the loss during LoRA training

[MH-Python]

Thank you for the nice compact work.
We have started recently to face an ambiguous error casing the loss to become nan during the training. After enabling anomaly detection " torch.autograd.set_detect_anomaly(True)"
We got this:

UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:
...stacktrace...
.venv/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 569, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
...stacktrace...
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'MmBackward0' returned nan values in its 1th output.

Could it be caused by some numerical instability (nan or inf)?