Save metadata and integration when save_weights_only is set #2396
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mvpatel2000
reviewed
Jul 27, 2023
|
@mvpatel2000 the issue with always including metadata is that its kind of an API change. If there is code out there relying on a weights only checkpoint only having one key. That being said, it may be worth making that change, not sure. But I think this is the right implementation given that. At a later date we will remove this flag and make it the default |
dakinggg
reviewed
Jul 27, 2023
eracah
changed the title
|
@mvpatel2000, @dakinggg, ok fixed to add metadata saving to the save_weights_only option |
Skylion007
reviewed
Jul 28, 2023
mvpatel2000
approved these changes
Jul 29, 2023
dakinggg
approved these changes
Jul 29, 2023
Co-authored-by: Mihir Patel <mihir.v.patel7@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When
save_weights_onlyis specified to the Trainer, only the weights are saved and no metadata or integrations are saved. However, when doing inference it is often necessary to have the metadata along with the weights. The only current way to get metadata and integrations is to save everything in the state_dict (including the very large optimizer state). This PR enables saving the weights and the metadata and integrations, so you can save an inference-friendly checkpoint without the extra heft of the optimizer or other (not necessary for inference) state fields.What issue(s) does this change relate to?
fix CO-2226