I'm experimenting unexpected loss values when running old recipes on the same datasets

[nouranali]

I'm currently trying to regenerate old results from KD and at first loss started at 2 then decreased but now it starts at ~17

I was using torchtune 0.6.0 and now 0.6.1
when reverting to 0.6.0 same thing happens

[joecummings]

old results from KD

Which version is this?

[nouranali]

old results from KD

Which version is this?

V 0.6.1

[joecummings]

Which specific KD recipe are you using and can you share an example config?

[nouranali]

Which specific KD recipe are you using and can you share an example config?

I tried several recipes for different models, I used the default configs to make sure it's not a dataset related issue, I used the llama 8b to 1b and the same happened, the problem is with ld loss mot class loss so I'm guessing something changed in that?

[pbontrager]

Do you need a new feature? It's usually recommended to keep your recipes tied to the torchtune version they were created with for reproducibility. Then for newer projects you can make a fresh copy of our recipes. If you have your own custom recipe features, then of course you have to keep them up to date.

[nouranali]

Do you need a new feature? It's usually recommended to keep your recipes tied to the torchtune version they were created with for reproducibility. Then for newer projects you can make a fresh copy of our recipes. If you have your own custom recipe features, then of course you have to keep them up to date.

No I don't use a custom recipe, I've re tried the llama one provided in the repo already and same thing happened, your're right about the version but I reverted to v0.6.0 and the same thing happened, I have old vs. new logs but I can't share them as they're related to my work, but something definitely changed in KD loss calculations that I can't seem to find.
I want to know if this happened with anyone else

[krammnic]

I was able to reproduce this on internal data either. Let me try to create public repro

[nouranali]

I was able to reproduce this on internal data either. Let me try to create public repro

Was able or wasn't able?

[krammnic]

I was able to reproduce this on internal data either. Let me try to create public repro

Was able or wasn't able?

Was able

[nouranali]

I was able to reproduce this on internal data either. Let me try to create public repro

Was able or wasn't able?

Was able

Can you elaborate? Do you mean you got the same results as before or reproduced the bug?

[krammnic]

I was able to reproduce this on internal data either. Let me try to create public repro

Was able or wasn't able?

Was able

Can you elaborate? Do you mean you got the same results as before or reproduced the bug?

I observe the difference between loss curves in recent torchtune versions for the KD recipe, so yes it seems to be a bug.

[nouranali]

I was able to reproduce this on internal data either. Let me try to create public repro

Was able or wasn't able?

Was able

Can you elaborate? Do you mean you got the same results as before or reproduced the bug?

I observe the difference between loss curves in recent torchtune versions for the KD recipe, so yes it seems to be a bug.

Yes, me too. I tried almost all of the recipes with all the models and the results are significantly different