Multi node - Multi GPU capability

[vince62s]

This PR allows to train on several nodes and several GPU:
introducing
-master_ip: ip address of the master node
-master_port: port number of th emaster node
-world_size = total number of processes to be run (total GPUs accross all nodes)
-gpu_ranks = list of indices of processes accross all nodes

-gpuid is deprecated

[sebastianGehrmann]

Hey @vince62s - nice changes! I personally find world_size a little ambiguous. Made me think of the world models paper. Could we change that to something more expressive like total_gpu_process_count?

[vince62s]

Well I agree except that this is exactly the vocabulary used by pytorch distributed (and subsequently Fairseq....)
https://pytorch.org/docs/stable/distributed.html

Sebastian, while you're here :) we really need you on the copy stuff that is broken (see #749 for instance)

[sebastianGehrmann]

I see, haven't really used those. Wonder why they thought it'd be a good idea.

Will have a look at the copy things now that I am back.

[vince62s]

For the records, when using multi node, if like most people you have a regular 1Gbps network, you really need to use accum_count above 8 or 16 to avoid too many inter-node communications.

[mjc14]

hi,
@vince62s ,i think the codes only support multi gpus inter-node, because if i have two nodes and four gpus in each node, the list of indices can't be 0 ~7, the main process can't have 8 child processes in each node, and i think to split data for to different node is needed.

    procs = []
    for device_id in range(nb_gpu):
        procs.append(mp.Process(target=run, args=(
            opt, device_id, error_queue, ), daemon=True))
        procs[device_id].start()
        logger.info(" Starting process pid: %d  " % procs[device_id].pid)
        error_handler.add_child(procs[device_id].pid)
    for p in procs:
        p.join()

i think the Facebook fairseq will give you more ideas. it support multi gpus inter-node and gpus cross node.
https://github.com/pytorch/fairseq
https://fairseq.readthedocs.io/en/latest/getting_started.html#advanced-training-options

[vince62s]

it does work.
on master you run world_size 8 gpu_ranks 0 1 2 3, and on node 2 you run world_size 8 gpu_ranks 4 5 6 7
make sure on each you have cuda_visible_devices 0,1,2,3

[mjc14]

sorry, it is my mistake. i read your code carefully in the morning .it did work . because i use slurm to control my jobs ,so i make some mistake. your codes is mostly perfect for my work. thanks very much.

[vince62s]

FYI one thing:
if your network is slow (eg 1Gbps) it will take time to update accross the 2 nodes depending on the size of your model.
For instance if your model is about 500MB it takes 5-6 sec for each update.
so you'd better use a high -accum (like 16 or 32)
but it all depends on your use case.

[mjc14]

thanks, reduce update times is needs. i have a test result about NCCL2 （all_reduce）on my nodes, it is about 5.09 GB/s on four nodes which has 4 gpus in each node.

[mpatwary]

Does anyone has sbatch/slurm example to run experiments on multinode multiGPU. I am not sure how to set the master_ip and port when slurm will give nodes on the fly? Helps appreciated.

[mjc14]

@mpatwary https://fairseq.readthedocs.io/en/latest/getting_started.html#advanced-training-options

[mpatwary]

@mjc14 That is a different codebase, right?

I am interested running this codebase in multinode environment using slurm. I tried the following one, but it doesn't produce anything.

sbatch -N 2 -p TitanXx8 --ntasks 2 --gres=gpu:8 --cpus-per-task 2 --job-name=smt --signal=USR1@600 --wrap "srun stdbuf -i0 -o0 -e0 python train.py -data data/demo -save_model demo-model-16 -world_size 16 -gpu_ranks 0 1 2 3 4 5 6 7" --output output-16.out

For single node, using the following command works fine.

sbatch -N 1 -p TitanXx8 --ntasks 1 --gres=gpu:8 --cpus-per-task 2 --job-name=smt --signal=USR1@600 --wrap "srun stdbuf -i0 -o0 -e0 python train.py -data data/demo -save_model demo-model-08 -world_size 8 -gpu_ranks 0 1 2 3 4 5 6 7" --output output-08.out

For multinode, I added the following lines in the onmt/utils/distributed.py so that I don't need to pass master_ip in sbatch submission.

def multi_init(opt, device_id):
node_list = os.environ.get('SLURM_JOB_NODELIST')
hostnames = subprocess.check_output(['scontrol', 'show', 'hostnames', node_list])
host=hostnames.split()[0].decode('utf-8')
dist_init_method = 'tcp://{master_ip}:{master_port}'.format(master_ip=host, #opt.master_ip, master_port=opt.master_port)

[mjc14]

yes, it is different codebase. you can not run this codebase in multinode environment using slrum directly. you need change some codes. you can read the fairseq project. it will be useful.
the following tow files is the key .
https://github.com/pytorch/fairseq/blob/master/train.py
https://github.com/pytorch/fairseq/blob/master/fairseq/distributed_utils.py