Timeout on creating the index mappings #15
Labels
bug
Something isn't working
Comments
mayank31398
pushed a commit
to mayank31398/BigCode-Megatron-LM
that referenced
this issue
Jun 18, 2023
* add direct meg-ds to hf format script (NVIDIA#110) * add direct meg-ds to hf format script (part2) (NVIDIA#111) * add direct meg-ds to hf format script * split into 2 function * update the usage doc * make scripts executable * add shebang Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: Stas Bekman <stas@stason.org>
mayank31398
pushed a commit
to mayank31398/BigCode-Megatron-LM
that referenced
this issue
Jun 21, 2023
* add direct meg-ds to hf format script (NVIDIA#110) * add direct meg-ds to hf format script (part2) (NVIDIA#111) * add direct meg-ds to hf format script * split into 2 function * update the usage doc * make scripts executable * add shebang Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: Stas Bekman <stas@stason.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When launching very long training runs, building the index mappings can take more than 1 minute.
The consequence is that the other ranks will timeout. https://github.com/bigcode-project/Megatron-LM/blob/multi-query-attention/megatron/training.py#L962
However the timeout passed to
torch.distributed.initializeis 10 mins. Why isn't this value used intorch.distributed.broadcast?The workaround for now is to first create the index mappings on a single worker, as a preliminary run.
The text was updated successfully, but these errors were encountered: