Skip to content

Fix FSDP related issues #1745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jul 21, 2023
Merged

Fix FSDP related issues #1745

merged 5 commits into from
Jul 21, 2023

Conversation

@pacman100
Copy link
Contributor

@pacman100 pacman100 commented Jul 19, 2023 •
edited
Loading

What does this PR do?

  1. Fixes load_result never being passed when using FSDP in Trainer
  2. Fixes NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet transformers#24874
  3. In torch nightlies they have removed the support for ignored_parameters, updating the same.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jul 19, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@pacman100 pacman100 mentioned this pull request Jul 19, 2023
Merged
@pacman100 pacman100 marked this pull request as ready for review July 19, 2023 17:16
@pacman100 pacman100 requested review from muellerzr and sgugger and removed request for muellerzr July 19, 2023 17:16
@pacman100 pacman100 changed the title Update fsdp_utils.py Fix FSDP related issues Jul 19, 2023
sgugger
sgugger approved these changes Jul 19, 2023
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes!

@pacman100 pacman100 merged commit e8ed10a into huggingface:main Jul 21, 2023
@winglian winglian mentioned this pull request Jul 24, 2023
Closed
4 tasks
winglian pushed a commit to OpenAccess-AI-Collective/accelerate that referenced this pull request Jul 25, 2023
@pacman100 @winglian
Fix FSDP related issues (huggingface#1745)
8ea64cc 
* Update fsdp_utils.py

* other FSDP fixes

* revert as this is resulting in more vram usage

* revert

* Update fsdp_utils.py
@pacman100 pacman100 mentioned this pull request Aug 8, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgugger sgugger sgugger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet

3 participants

@pacman100 @HuggingFaceDocBuilderDev @sgugger