Exception: cuda rng state model-parallel-rng is not added #369
Comments
|
See here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
i start the job the i met this error:
cuda: 12.0
torch: 1.14
deepspeed --num_gpus 2 pretrain_gpt_v2.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --distributed-backend nccl --num-layers 2 --hidden-size 64 --num-attention-heads 2 --seq-length 1024 --max-position-embeddings 1024 --micro-batch-size 1 --rampup-batch-size 2 2 1_000 --global-batch-size 16 --train-samples 100 --optimizer adam --adam-beta1 0.9 --adam-beta2 0.95 --adam-eps 1e-8 --lr 1e-4 --lr-warmup-samples 5 --clip-grad 1.0 --weight-decay 1e-1 --vocab-file /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-vocab.json --merge-file /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-merges.txt --fp16 --log-interval 10 --save-interval 100 --eval-interval 100 --eval-iters 10 --checkpoint-activations --save alibi_test --load alibi_test --data-path /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/BookCorpusDataset_text_document --tensorboard-dir output_dir_tensorboard --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --deepspeed --deepspeed_config ./ds_config.json --zero-stage 1 --deepspeed-activation-checkpointing
[2023-03-06 04:10:57,331] [WARNING] [runner.py:186:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-03-06 04:10:57,454] [INFO] [runner.py:548:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None pretrain_gpt_v2.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --distributed-backend nccl --num-layers 2 --hidden-size 64 --num-attention-heads 2 --seq-length 1024 --max-position-embeddings 1024 --micro-batch-size 1 --rampup-batch-size 2 2 1_000 --global-batch-size 16 --train-samples 100 --optimizer adam --adam-beta1 0.9 --adam-beta2 0.95 --adam-eps 1e-8 --lr 1e-4 --lr-warmup-samples 5 --clip-grad 1.0 --weight-decay 1e-1 --vocab-file /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-vocab.json --merge-file /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-merges.txt --fp16 --log-interval 10 --save-interval 100 --eval-interval 100 --eval-iters 10 --checkpoint-activations --save alibi_test --load alibi_test --data-path /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/BookCorpusDataset_text_document --tensorboard-dir output_dir_tensorboard --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --deepspeed --deepspeed_config ./ds_config.json --zero-stage 1 --deepspeed-activation-checkpointing
[2023-03-06 04:10:59,741] [INFO] [launch.py:135:main] 0 NCCL_VERSION=2.16.5
[2023-03-06 04:10:59,741] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2023-03-06 04:10:59,741] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-03-06 04:10:59,741] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2023-03-06 04:10:59,741] [INFO] [launch.py:162:main] dist_world_size=2
[2023-03-06 04:10:59,741] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0,1
using world size: 2, data-parallel-size: 2, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
using torch.float16 for parameters ...
------------------------ arguments ------------------------
abort_on_unmet_fused_kernel_constraints ......... False
accumulate_allreduce_grads_in_fp32 .............. False
adam_beta1 ...................................... 0.9
adam_beta2 ...................................... 0.95
adam_eps ........................................ 1e-08
adlr_autoresume ................................. False
adlr_autoresume_interval ........................ 1000
apply_query_key_layer_scaling ................... True
apply_residual_connection_post_layernorm ........ False
attention_dropout ............................... 0.1
attention_softmax_in_fp32 ....................... False
bert_binary_head ................................ True
bert_load ....................................... None
bf16 ............................................ False
bias_dropout_fusion ............................. True
bias_gelu_fusion ................................ True
biencoder_projection_dim ........................ 0
biencoder_shared_query_context_model ............ False
block_data_path ................................. None
checkpoint_activations .......................... True
checkpoint_in_cpu ............................... False
checkpoint_num_layers ........................... 1
clip_grad ....................................... 1.0
codecarbon_dir .................................. None
consumed_train_samples .......................... 0
consumed_train_tokens ........................... 0
consumed_valid_samples .......................... 0
contigious_checkpointing ........................ False
cpu_optimizer ................................... False
cpu_torch_adam .................................. False
curriculum_learning ............................. False
data_impl ....................................... infer
data_parallel_size .............................. 2
data_path ....................................... ['/mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/BookCorpusDataset_text_document']
dataloader_type ................................. single
DDP_impl ........................................ local
decoder_seq_length .............................. None
deepscale ....................................... False
deepscale_config ................................ None
deepspeed ....................................... True
deepspeed_activation_checkpointing .............. True
deepspeed_config ................................ ./ds_config.json
deepspeed_mpi ................................... False
distribute_checkpointed_activations ............. False
distributed_backend ............................. nccl
embed_layernorm ................................. False
embedding_path .................................. None
encoder_seq_length .............................. 1024
eod_mask_loss ................................... False
eval_interval ................................... 100
eval_iters ...................................... 10
eval_only ....................................... None
evidence_data_path .............................. None
exit_duration_in_mins ........................... None
exit_interval ................................... None
ffn_hidden_size ................................. 256
finetune ........................................ False
fp16 ............................................ True
fp16_lm_cross_entropy ........................... False
fp32_residual_connection ........................ False
gigaflos_no_embeds .............................. 0
global_batch_size ............................... 16
glu_activation .................................. None
hidden_dropout .................................. 0.1
hidden_size ..................................... 64
hysteresis ...................................... 2
ict_head_size ................................... None
ict_load ........................................ None
img_dim ......................................... 224
indexer_batch_size .............................. 128
indexer_log_interval ............................ 1000
inference ....................................... False
init_method_std ................................. 0.02
init_method_xavier_uniform ...................... False
initial_loss_scale .............................. 4294967296
kill_switch_path ................................ None
kv_channels ..................................... 32
layernorm_epsilon ............................... 1e-05
lazy_mpu_init ................................... None
load ............................................ alibi_test
local_rank ...................................... 0
log_batch_size_to_tensorboard ................... True
log_interval .................................... 10
log_learning_rate_to_tensorboard ................ True
log_level ....................................... None
log_level_replica ............................... None
log_loss_scale_to_tensorboard ................... True
log_num_zeros_in_grad ........................... False
log_params_norm ................................. False
log_path ........................................ None
log_timers_to_tensorboard ....................... True
log_validation_ppl_to_tensorboard ............... True
loss_on_targets_only ............................ False
loss_scale ...................................... None
loss_scale_window ............................... 1000
lr .............................................. 0.0001
lr_decay_iters .................................. None
lr_decay_samples ................................ None
lr_decay_style .................................. linear
lr_decay_tokens ................................. None
lr_warmup_fraction .............................. None
lr_warmup_iters ................................. 0
lr_warmup_samples ............................... 5
make_vocab_size_divisible_by .................... 128
mask_prob ....................................... 0.15
masked_softmax_fusion ........................... True
max_position_embeddings ......................... 1024
mean_noise_span_length .......................... None
memory_centric_tiled_linear ..................... False
merge_file ...................................... /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-merges.txt
micro_batch_size ................................ 1
min_loss_scale .................................. 1.0
min_lr .......................................... 0.0
mmap_warmup ..................................... False
no_load_optim ................................... None
no_load_rng ..................................... None
no_save_optim ................................... None
no_save_rng ..................................... None
noise_density ................................... None
num_attention_heads ............................. 2
num_channels .................................... 3
num_classes ..................................... 1000
num_layers ...................................... 2
num_layers_per_virtual_pipeline_stage ........... None
num_workers ..................................... 2
onnx_safe ....................................... None
openai_gelu ..................................... False
optimizer ....................................... adam
override_lr_scheduler ........................... False
pad_vocab_size_to ............................... None
params_dtype .................................... torch.float16
partition_activations ........................... False
patch_dim ....................................... 16
pipeline_model_parallel_size .................... 1
position_embedding_type ......................... PositionEmbeddingType.absolute
pp_partition_method ............................. None
profile_backward ................................ False
query_in_block_prob ............................. 0.1
rampup_batch_size ............................... ['2', '2', '1_000']
rank ............................................ 0
remote_device ................................... none
reset_attention_mask ............................ False
reset_position_ids .............................. False
retriever_report_topk_accuracies ................ []
retriever_score_scaling ......................... False
retriever_seq_length ............................ 256
reweight_loss_based_on_position_frequency ....... False
sample_rate ..................................... 1.0
save ............................................ alibi_test
save_interval ................................... 100
scatter_gather_tensors_in_pipeline .............. True
scattered_embeddings ............................ False
seed ............................................ 1234
seq_length ...................................... 1024
sgd_momentum .................................... 0.9
short_seq_prob .................................. 0.1
skip_train_iteration_range ...................... None
split ........................................... 969, 30, 1
split_transformers .............................. False
sync_tp_duplicated_parameters ................... False
synchronize_each_layer .......................... False
tensor_model_parallel_size ...................... 1
tensorboard_dir ................................. output_dir_tensorboard
tensorboard_log_interval ........................ 1
tensorboard_queue_size .......................... 5
test_weighted_split_names ....................... None
test_weighted_split_paths ....................... None
test_weighted_split_paths_path .................. None
test_weighted_split_splits ...................... None
test_weighted_split_weights ..................... None
tile_factor ..................................... 1
titles_data_path ................................ None
tokenizer_name_or_path .......................... None
tokenizer_type .................................. GPT2BPETokenizer
train_iters ..................................... None
train_samples ................................... 100
train_tokens .................................... None
train_weighted_split_paths ...................... None
train_weighted_split_paths_path ................. None
universal_checkpoint ............................ False
use_bnb_optimizer ............................... False
use_checkpoint_lr_scheduler ..................... False
use_contiguous_buffers_in_ddp ................... False
use_cpu_initialization .......................... None
use_one_sent_docs ............................... False
use_pin_memory .................................. False
valid_num_workers ............................... 2
valid_weighted_split_names ...................... None
valid_weighted_split_paths ...................... None
valid_weighted_split_paths_path ................. None
valid_weighted_split_splits ..................... None
valid_weighted_split_weights .................... None
virtual_pipeline_model_parallel_size ............ None
vocab_extra_ids ................................. 0
vocab_file ...................................... /mnt/dp_mega/Microsoft-Megatron-DeepSpeed/dataset/gpt2-vocab.json
weight_decay .................................... 0.1
world_size ...................................... 2
zero_allgather_bucket_size ...................... 0.0
zero_contigious_gradients ....................... False
zero_reduce_bucket_size ......................... 0.0
zero_reduce_scatter ............................. False
zero_stage ...................................... 1
-------------------- end of arguments ---------------------
will use batch size rampup starting from global batch size 2 to global batch size 16 with batch size increments 2 over 1000 samples.
The text was updated successfully, but these errors were encountered: