re-merge from NVIDIA main #68
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Switches the cache to using md5 hashes of a text description instead of crafted filenames to determine a "cache hit". Changes the default location of these files to be an "index-cache" directory inside the data root. Should leave the data directories a bit cleaner, especially with these filenames being a bit "uglier". For GPT the code will first look in this default location before building a new index and caching it the specified data cache path (or this default if none is given). For Blendable dataset it will only look for and save the indices if a data cache path is provided, otherwise it will just rebuild every time.
Add option to overlap p2p communication. See merge request ADLR/megatron-lm!621
Add option to specify a data cache path separate from data directory. See merge request ADLR/megatron-lm!608
Fix GPTDataset assert. See merge request ADLR/megatron-lm!624
Fixed rotary_pos_emb's position in layer's forward args. See merge request ADLR/megatron-lm!625
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Fix indexation for output tensor after gradscaler call See merge request ADLR/megatron-lm!627
Perform grad sync at correct place in interleaved pipeline parallelism See merge request ADLR/megatron-lm!628
Supporting loading checkpoints without add_position_embedding arg. See merge request ADLR/megatron-lm!623
Add workarounds for non-determinism in Megatron training See merge request ADLR/megatron-lm!607
Update gitlab to catch pytest errors See merge request ADLR/megatron-lm!635
Remove use of deprecated np.float in indexed_dataset.py See merge request ADLR/megatron-lm!634
Retro fix for tensor parallelism. See merge request ADLR/megatron-lm!632
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Among other things, fixes a backward compatibility issue of the checkpoint merging tools introduced by the previous merge.