torch compile causes gguf tuning large accuracy drop

[wenhuach21]

--format xxx,fake and evaluate the fake model. Not sure the real model is affected or not

The main branch is fine. I'm currently running some experiments, so we need to be cautious when using torch.compile.