Update stateful_callbacks state before saving checkpoint #32115
Conversation
|
Gentle ping @muellerzr |
|
@pedrobrs can you rebase from main? This should fix the failing tests |
pedrobrs
force-pushed
the
exportable_state_callbacks_update_state_on_save
branch
from
d57a56f
to
f68b8b9
Compare
|
looks like this logic broke some of the tests in test_trainer.py: Looks like this is because this change doesn't actually wind up restoring the state? |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
I saw that error when pushing the first time, I'll investigate the issue to identify and fix the problem. RuntimeError: Failed to import transformers.models.audio_spectrogram_transformer.feature_extraction_audio_spectrogram_transformer because of the following error (look up to see its traceback): I may not be able to resolve this issue on my own. What should I do about it? |
|
This one is unrelated to you and was already fixed on main! |
pedrobrs
force-pushed
the
exportable_state_callbacks_update_state_on_save
branch
from
f68b8b9
to
63552fa
Compare
|
cc @ArthurZucker for final 🚀 |
…#32115) * update ExportableState callbacks state before saving trainer_state on save_checkpoint * run make fixup and fix format * manage multiple stateful callbacks of same class
…#32115) * update ExportableState callbacks state before saving trainer_state on save_checkpoint * run make fixup and fix format * manage multiple stateful callbacks of same class
…#32115) * update ExportableState callbacks state before saving trainer_state on save_checkpoint * run make fixup and fix format * manage multiple stateful callbacks of same class
Fixes
This PR addresses an issue where stateful callbacks, such as EarlyStoppingCallback, were not being updated before saving checkpoints. As a result, resuming training would not have access to the latest state of these callbacks.
Description
Updated the state of stateful callbacks before saving checkpoints to ensure that their state is preserved and correctly restored when resuming training.
Suggested Reviewers
@muellerz @amyeroberts @SunMarc