Why not merge BA during training?

[rp7sv]

During training, the low-rank matrices create a bypass, which introduces additional latency. So why not merge the low-rank weights directly into W during training, just like we do at inference time? I'm not sure whether this would affect gradient flow？

[XuRan-Zhao]

During training, the low-rank matrices create a bypass, which introduces additional latency. So why not merge the low-rank weights directly into W during training, just like we do at inference time? I'm not sure whether this would affect gradient flow？

May I ask if you know the answer to this question now? Can you clarify it? I also have the same doubts.

[jim-min]

But then you cannot freeze the W, which is the reason we use LoRA?