Add _allgather_base & _reduce_scatter_base to dist backend
#3919
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
rebase should solve the issue. Can you fill out the description with what issue this pr intend to solve? |
hgt312
force-pushed
the
dist_backend_patch
branch
from
02ff40e to
5ba3aae
Compare
|
@hgt312 Hi Guangtai, it seems your purpose is to have simpler all-gather and reduce-scatter APIs. In that case, your application code can directly call |
hgt312
force-pushed
the
dist_backend_patch
branch
from
52647d3 to
b2458c1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for
dist._allgather_baseanddist._reduce_scatter_basefor pytorchxladist backend. These two ops are simpler so we needn't to use concat/split when implement them and they are used by DeepSpeed/FSDP