Fix DDP issues and Support DDP for all training scripts #448

Isotr0py · 2023-04-25T15:46:39Z

Fix [Bug] LORA train.py fails in DDP multi-gpu mode with --network_train_unet_only flag because of None #365
Fix [Request] Fix multi-gpu training for train_db.py and fine_tune.py #359
Suppport DDP for all training scripts

Isotr0py · 2023-04-25T17:19:00Z

Well, some issues occurred using train_db.py with sampling images while train_network.py works normally.
I'm not sure whether other scripts have this issue.
I need some time to locate the issue and fully test all scripts.

alleniver · 2023-04-26T01:06:55Z

nice work!

kohya-ss · 2023-04-26T12:38:40Z

Thank you for this! It looks good!

Unfortunately I am unable to test in a distributed training, but as soon as I have time, I will verify that it works on a single GPU as well.

kohya-ss · 2023-05-03T07:37:57Z

I've merged. Thank you again!

nofacedeepfake · 2023-05-06T09:18:12Z

Hello, thank you very much for developing this direction, how do I run two 3090ti?

Isotr0py · 2023-05-06T10:20:24Z

@nofacedeepfake You may need to change accelerate config as below:

- compute_environment: LOCAL_MACHINE
- distributed_type: MULTI_GPU
- mixed_precision: no
- use_cpu: False
- num_processes: 2
- machine_rank: 0
- num_machines: 1
- main_process_ip: None
- main_process_port: None
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}

So you can do as follows:

Method1: run accelerate config and answer:
Which type of machine are you using? -> multi-GPU
How many GPU(s) should be used for distributed training? -> <Number of GPUs> (here is 2)
Other than that, the default settings are fine.
Method2: add --multi_gpu at your training commandline like:
accelerate launch --multi_gpu --num_processes=2 train_network.py

nofacedeepfake · 2023-05-09T06:53:28Z

Thank you very much, and can you describe in detail in what file I replace it ? I don't quite understand where to change - compute_environment: LOCAL_MACHINE

distributed_type: MULTI_GPU
mixed_precision: no
use_cpu: False
num_processes: 2
machine_rank: 0
num_machines: 1
main_process_ip: None
main_process_port: None
main_training_function: main
deepspeed_config: {}
fsdp_config: {}

and add --multi_gpu on your training command line, e.g:
accelerate launch --multi_gpu --num_processes=2 train_network.py

@nofacedeepfake Возможно, вам потребуется изменить конфигурацию ускорения, как показано ниже:
- компьютерная среда: LOCAL_MACHINE
- распространяемый тип: MULTI_GPU
- смешанная точность: НЕТ
- use_cpu: Ложь
- нум_процессы: 2
- machine_rank: 0
- num_machines: 1
- основной процесс_ip: Нет
- основной процесс_порт: Нет
- основная обучающая функция: Главная
- deepspeed_config: {}
- fsdp_config: {}
Итак, вы можете сделать следующее:

Метод1: выполнить accelerate config и ответить:
Which type of machine are you using? -> мульти-графический процессор
How many GPU(s) should be used for distributed training? -> <Количество графических процессоров> (здесь 2) В остальном настройки по умолчанию в порядке.

Метод2: добавить --multi_gpu в командной строке вашего обучения, например:
accelerate launch --multi_gpu --num_processes=2 train_network.py

Isotr0py · 2023-05-09T09:26:41Z

@nofacedeepfake My answer may be unclear. No need to replace any file manually.

You just need to run accelerate config to reset the config file. Accelerate Config Doc and Accelerate CMD Doc may be helpful.

And if you don't want to change the accelerate config file, you can just change the train command line to run training script like:
accelerate launch --multi_gpu --num_processes=2 train_network.py

nofacedeepfake · 2023-05-13T08:15:48Z

no method worked
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.). [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.).

C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py:249: FutureWarning: logging_dir is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use project_dir instead. warnings.warn( [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.). prepare accelerator [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.). C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py:249: FutureWarning: logging_dir is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use project_dir instead. warnings.warn( Traceback (most recent call last): File "C:\Kohya\kohya_ss\train_network.py", line 773, in [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.). train(args) File "C:\Kohya\kohya_ss\train_network.py", line 139, in train [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [DESKTOP-VTVG7AS]:29500 (system error: 10049 - ╥Ёхсєхь√щ рфЁхё фы ётюхую ъюэЄхъёЄр эхтхЁхэ.). accelerator, unwrap_model = train_util.prepare_accelerator(args)Traceback (most recent call last): File "C:\Kohya\kohya_ss\train_network.py", line 773, in File "C:\Kohya\kohya_ss\library\train_util.py", line 2875, in prepare_accelerator train(args) File "C:\Kohya\kohya_ss\train_network.py", line 139, in train accelerator = Accelerator(accelerator, unwrap_model = train_util.prepare_accelerator(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 346, in init File "C:\Kohya\kohya_ss\library\train_util.py", line 2875, in prepare_accelerator self.state = AcceleratorState( File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 540, in init accelerator = Accelerator(PartialState(cpu, **kwargs) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 346, in init File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 129, in init self.state = AcceleratorState(torch.distributed.init_process_group(backend="nccl", **kwargs) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 540, in init File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 895, in init_process_group PartialState(cpu, **kwargs)default_pg = _new_process_group_helper( File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 129, in init File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 998, in _new_process_group_helper torch.distributed.init_process_group(backend="nccl", **kwargs)raise RuntimeError("Distributed package doesn't have NCCL " "built in") File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 895, in init_process_groupRuntimeError : default_pg = _new_process_group_helper(Distributed package doesn't have NCCL built in File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 998, in _new_process_group_helper raise RuntimeError("Distributed package doesn't have NCCL " "built in") RuntimeError: Distributed package doesn't have NCCL built in ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 45044) of binary: C:\Kohya\kohya_ss\venv\Scripts\python.exe Traceback (most recent call last): File "C:\Users\79255\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\79255\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Kohya\kohya_ss\venv\Scripts\accelerate.exe_main.py", line 7, in File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main args.func(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 914, in launch_command multi_gpu_launcher(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 603, in multi_gpu_launcher distrib_run.run(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\run.py", line 785, in run elastic_launch( File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\launcher\api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "C:\Kohya\kohya_ss\venv\lib\site-packages\torch\distributed\launcher\api.py", line 250, in launch_agent

nofacedeepfake · 2023-05-13T08:18:05Z

There are three video cards in my system

nofacedeepfake · 2023-05-13T08:19:09Z

2 pieces of 3090ti nvlink and 3060

nofacedeepfake · 2023-05-13T08:42:10Z

accelerate launch --gpu_ids=0,1 --multi_gpu --num_processes=2 --num_cpu_threads_per_process=8 "train_network.py"
and gives the same error

Isotr0py · 2023-05-13T08:46:58Z

@nofacedeepfake I think it's an error from windows environment without NCCL.
According to #247 , you can add this code in below to the top of train_network.py:

import os
os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

It's reported to work on windows, but I don't have a windows environment with multiple GPUs to test it :(

nofacedeepfake · 2023-05-13T09:05:23Z

https://ibb.co/48870VD

CCJetWing · 2023-12-16T09:49:23Z

I have set accelerate launch arguements but have another error:

NOTE: Redirects are currently not supported in Windows or MacOs.
Using RTX 3090 or 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\accelerate\commands\launch.py", line 1027, in
main()
File "D:\NEW\lora-scripts\Venv\lib\site-packages\accelerate\commands\launch.py", line 1023, in main
launch_command(args)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\accelerate\commands\launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\accelerate\commands\launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\run.py", line 785, in run
elastic_launch(
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\launcher\api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\launcher\api.py", line 241, in launch_agent
result = agent.run()
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\metrics\api.py", line 129, in wrapper
result = f(*args, **kwargs)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\agent\server\api.py", line 723, in run
result = self._invoke_run(role)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\agent\server\api.py", line 858, in _invoke_run
self._initialize_workers(self._worker_group)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\metrics\api.py", line 129, in wrapper
result = f(*args, **kwargs)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\agent\server\api.py", line 692, in _initialize_workers
self._rendezvous(worker_group)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\metrics\api.py", line 129, in wrapper
result = f(*args, **kwargs)
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\agent\server\api.py", line 546, in _rendezvous
store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous()
File "D:\NEW\lora-scripts\Venv\lib\site-packages\torch\distributed\elastic\rendezvous\static_tcp_rendezvous.py", line 55, in next_rendezvous
self._store = TCPStore( # type: ignore[call-arg]
RuntimeError: unmatched '}' in format string

How to fix this?

kohya-ss · 2023-12-16T11:05:14Z

According to this pytorch/pytorch#100185 , maybe, I don't think this is sd-scripts' issue...

Isotr0py added 9 commits April 25, 2023 20:52

Fix DDP bugs

7561604

Fix DDP bugs for finetune and db

b89b015

refactor model loader

0eee07d

fix DDP network

2059c88

try to fix DDP network in train unet only

5807751

remove unuse DDP import

acba48a

refactor DDP transform

cc8ea52

refactor DDP transform

fb40ef6

fix sample images bugs

702d943

Isotr0py marked this pull request as draft April 25, 2023 17:11

Isotr0py added 3 commits April 26, 2023 10:39

change DDP tranform location

6c3dc27

add autocast to train_db

9c9d21d

support DDP in XTI

97fa7ae

Isotr0py marked this pull request as ready for review April 26, 2023 05:38

Clear DDP import

32ad4d5

kohya-ss changed the base branch from main to dev May 3, 2023 01:23

kohya-ss merged commit e1143ca into kohya-ss:dev May 3, 2023

Isotr0py deleted the DDP_fix branch May 3, 2023 02:19

kohya-ss mentioned this pull request May 3, 2023

[Request] Fix multi-gpu training for train_db.py and fine_tune.py #359

Closed

bmaltais mentioned this pull request May 5, 2023

v21.5.8 bmaltais/kohya_ss#714

Merged

kohya-ss mentioned this pull request Nov 5, 2023

[Bug] Gradients not synchronized #924

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix DDP issues and Support DDP for all training scripts #448

Fix DDP issues and Support DDP for all training scripts #448

Isotr0py commented Apr 25, 2023 •

edited

Loading

Isotr0py commented Apr 25, 2023 •

edited

Loading

alleniver commented Apr 26, 2023

kohya-ss commented Apr 26, 2023

kohya-ss commented May 3, 2023

nofacedeepfake commented May 6, 2023

Isotr0py commented May 6, 2023 •

edited

Loading

nofacedeepfake commented May 9, 2023

Isotr0py commented May 9, 2023 •

edited

Loading

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

Isotr0py commented May 13, 2023 •

edited

Loading

nofacedeepfake commented May 13, 2023

CCJetWing commented Dec 16, 2023

kohya-ss commented Dec 16, 2023

Fix DDP issues and Support DDP for all training scripts #448

Fix DDP issues and Support DDP for all training scripts #448

Conversation

Isotr0py commented Apr 25, 2023 • edited Loading

Isotr0py commented Apr 25, 2023 • edited Loading

alleniver commented Apr 26, 2023

kohya-ss commented Apr 26, 2023

kohya-ss commented May 3, 2023

nofacedeepfake commented May 6, 2023

Isotr0py commented May 6, 2023 • edited Loading

nofacedeepfake commented May 9, 2023

Isotr0py commented May 9, 2023 • edited Loading

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

nofacedeepfake commented May 13, 2023

Isotr0py commented May 13, 2023 • edited Loading

nofacedeepfake commented May 13, 2023

CCJetWing commented Dec 16, 2023

kohya-ss commented Dec 16, 2023

Isotr0py commented Apr 25, 2023 •

edited

Loading

Isotr0py commented Apr 25, 2023 •

edited

Loading

Isotr0py commented May 6, 2023 •

edited

Loading

Isotr0py commented May 9, 2023 •

edited

Loading

Isotr0py commented May 13, 2023 •

edited

Loading