Skip to content

fix failure on restart after round 1 train and no eval #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 19 commits into from
Closed

fix failure on restart after round 1 train and no eval #4

wants to merge 19 commits into from

Conversation

@stas00
Copy link

@stas00 stas00 commented Jul 23, 2021

Was getting:

 File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 132, in pretrain
    = build_train_valid_test_data_iterators(
  File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 842, in build_train_valid_test_data_iterators
    = build_train_valid_test_data_iterators(
  File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 842, in build_train_valid_test_data_iterators
    assert args.train_samples is None, \
AssertionError: only backward compatiblity support for iteration-based training

on restart after a very short first run.

stas00 and others added 19 commits July 2, 2021 10:46
@stas00
explain what this repo is about.
fb1a120
@stas00
add the explanation about the forking problems.
5502865
@sbmaruf @TevenLeScao
Hf tok pipeline (#2)
b1a30de 
* hf tokenizer pipeline

* unused arg removed

* Changing Hf to HF

* Changing Hf to HF

Co-authored-by: Teven <teven.lescao@gmail.com>
@TevenLeScao
Create README for preprocessing
ebb6a07
@TevenLeScao
Update README.md
a0ab8f4
@TevenLeScao
Update README.md
ca4ea15
@TevenLeScao
HF tokenizers (on the training side) (#5)
0b2f0df
@TevenLeScao
Tiny gpt training script for local testing
d61ca5c
@stas00
update to deepspeed usage; rm irrelevant info
c5e0b24
@stas00
install instructions; consistent indent
7d880b7
@stas00
consistency
8cf27f2
@stas00
update docs
12503bd
@stas00
add example script for one node
2048ca5
@stas00
adjust the script
5ab4fad
@TevenLeScao
Fixed single node test scripts to use DS
1556d7e
@TevenLeScao
Fixed single node test scripts to use DS
a7a534a
@stas00
train_samples check fix
d13f5c6
@stas00
Revert "train_samples check fix"
0e78387 
This reverts commit d13f5c6.
@stas00
fix bug when restarting with no eval in round 1
0bea06e
@stas00 stas00 closed this Jul 23, 2021
@stas00 stas00 deleted the train-no-eval-restart2 branch July 23, 2021 20:17
jeffra pushed a commit that referenced this pull request Dec 10, 2021
@conglongli
Merge pull request #4 from awan-10/moe-train
338e237 
Moe training integration
hyoo pushed a commit to hyoo/Megatron-DeepSpeed that referenced this pull request Sep 21, 2022
@ngoyal2707
Merge pull request deepspeedai#4 from punitkoura/submodule-cleanup
ae0b844 
Adding setup.py files to enable building fairseq_v2
@wangguojim wangguojim mentioned this pull request Aug 11, 2023
Open
yueming-yuan added a commit to Supercomputing-System-AI-Lab/Megatron-DeepSpeed that referenced this pull request Sep 14, 2025
@yueming-yuan
Merge pull request deepspeedai#4 from yueming-yuan/tmp
f1929f1 
add scripts on frontier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@stas00 @sbmaruf @TevenLeScao