multi GPU error

[yuxuan2015]

torchrun --nproc_per_node gpu example.py --ckpt_dir pyllama_data/7B --tokenizer_path pyllama_data/tokenizer.model

Traceback (most recent call last):
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 119, in
Traceback (most recent call last):
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 119, in
fire.Fire(main)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
fire.Fire(main)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
Traceback (most recent call last):
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 119, in
component, remaining_args = _CallAndUpdateTrace(component, remaining_args = _CallAndUpdateTrace(

File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
fire.Fire(main)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component = fn(*varargs, **kwargs)
component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 78, in main

File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component = fn(*varargs, **kwargs)
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 78, in main
generator = load(
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 42, in load
generator = load(
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 42, in load
assert world_size == len(
AssertionError: Loading a checkpoint for MP=1 but world size is 4
assert world_size == len(component, remaining_args = _CallAndUpdateTrace(

File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
AssertionError: Loading a checkpoint for MP=1 but world size is 4
component = fn(*varargs, **kwargs)
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 78, in main
generator = load(
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 42, in load
assert world_size == len(
AssertionError: Loading a checkpoint for MP=1 but world size is 4
Traceback (most recent call last):
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 119, in
fire.Fire(main)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 78, in main
generator = load(
File "/data1/Semantic_team/speech/chatgpt/llama/example.py", line 42, in load
assert world_size == len(
AssertionError: Loading a checkpoint for MP=1 but world size is 4
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 8748) of binary: /data1/anaconda3/envs/llama_py310/bin/python3.1
Traceback (most recent call last):
File "/data1/anaconda3/envs/llama_py310/bin/torchrun", line 8, in
sys.exit(main())
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data1/anaconda3/envs/llama_py310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

[papasega]

Hello, I have the same error :

""" AssertionError: Loading a checkpoint for MP=1 but world size is 4
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 8748) of binary: /data1/anaconda3/envs/llama_py310/bin/python3."""

And I fixe it by :

torchrun --nproc_per_node 1 example.py --ckpt_dir $TARGET_FOLDER/7B ......

I think you have to fixe --nproc_per_node buy the number of GPU you have. In my case I have just 1

[kechan]

I ran into this issue on Apple M2 Max. My nproc_per_node is 1 and prominent error messages:

File "/Users//Developer/python39_env/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 895, in init_process_group
default_pg = _new_process_group_helper(
File "/Users//Developer/python39_env/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 998, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL " "built in")

File "/Users//Developer/python39_env/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/Users//Developer/python39_env/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Any idea what I should do?

[ejsd1989]

@yuxuan2015 @kechan checking in to see if this was resolved in your testing?