CUDA_VISIBLE_DEVICES isn't being respected / hostfile doesn't quite work for one node

[stas00]

I'm trying to experiment with DS/1gpu and it's not respecting CUDA_VISIBLE_DEVICES

I run the script as:

CUDA_VISIBLE_DEVICES=1 deepspeed --num_gpus=1 ./finetune_trainer.py ...

but it runs on GPU 0 ignoring CUDA_VISIBLE_DEVICES=1

Then I tried to use deepspeed launcher flags as explained here: https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node and encountered multiple issues there:

1. I think the --hostfile cl arg in the example are in the wrong place, shouldn't they be right after deepspeed and not in the client's args? that is instead of:

deepspeed <client_entry.py> <client args> \
  --deepspeed --deepspeed_config ds_config.json --hostfile=myhostfile

deepspeed  --hostfile=myhostfile <client_entry.py> <client args> \
  --deepspeed --deepspeed_config ds_config.json

This is a launcher arg and not client arg.

2. it can't handle hostfile with 1 entry:

$ cat hostfile
worker-1 slots=2

$ deepspeed --hostfile hostfile  ./finetune_trainer.py ...
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/bin/deepspeed", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/bin/deepspeed", line 6, in <module>
    main()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 259, in main
    resource_pool = fetch_hostfile(args.hostfile)
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 133, in fetch_hostfile
    raise err
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 127, in fetch_hostfile
    hostname, slots = line.split()
ValueError: not enough values to unpack (expected 2, got 0)

3. it can't handle exclusion or inclusions w/o the hostfile (misleading docs)
   Copy-n-pasting from docs - very last code example:

$ deepspeed --exclude="worker-1:0" ./finetune_trainer.py
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/bin/deepspeed", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/bin/deepspeed", line 6, in <module>
    main()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 272, in main
    active_resources = parse_inclusion_exclusion(resource_pool,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 240, in parse_inclusion_exclusion
    return parse_resource_filter(active_resources,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 187, in parse_resource_filter
    raise ValueError("Hostname '{}' not found in hostfile".format(hostname))
ValueError: Hostname 'worker-1' not found in hostfile

I think the docs are wrong/misleading - they suggest:

You can instead include or exclude specific resources using the --include and --exclude flags. For example, to use all available resources except GPU 0 on node worker-2 and GPUs 0 and 1 on worker-3:

• but they don't specify that the hostfile is actually needed.
• and the error message is misleading since what hostfile is it talking about? I haven't passed it any hostfiles in this experiment and if it found it in the current dir, that hostfile does have worker-1 in it - see cat hostfile earlier. So it should not just say "in hostfile" but in /path/to/hostfile
• I think in this particular situation it should say: "hostfile hasn't been provided and it's required"

4. this is not the right solution since it tries to ssh to worker-1

subprocess.CalledProcessError: Command '['ssh worker-1 hostname -I']' returned non-zero exit status 255.

So how does one configure deepspeed to use a specific GPU on a single node?

Thank you!

[jeffra]

Thanks for reporting this @stas00. There are a few different things going on here, let me try and address all of them here (if I miss one please let me know haha).

If you do not provide the deepspeed launcher with a hostfile (via -H/--hostfile) it will only launch processes within that local node. We will try to discover all the available GPUs on the box via torch.cuda.device_count(), see here for more details on this logic. Going forward the local node would be referred to as localhost.

The deepspeed launcher was primarily written to simplify multi-node training in interactive training environments. It supports arbitrary inclusion/exclusion of gpus/nodes. For example, I may want to exclude just the 3rd gpu on node 5 (maybe it has ECC errors) out of a total of 128 gpus and 32 nodes. This would be achieved via deepspeed --exclude worker-5:3 train.py.

In order to support this arbitrary inclusion/exclusion our launcher sets the appropriate CUDA_VISIBLE_DEVICES at process launch time on each node. This means that if the user sets their own CUDA_VISIBLE_DEVICES on the launching node it's not clear if they want to set this value on the local node or all nodes. We should update our docs to make this more clear though.

If you wanted to do something like CUDA_VISIBLE_DEVICES=1 deepspeed --num_gpus=1 ./finetune_trainer.py ... I would recommend running this as deepspeed --include localhost:1 ./finetune_trainer.py ....

Now if you wanted to use a hostfile to define your node list (even just 1 node) you can do that like you have. However, after seeing your ValueError stack trace I think you may have a trailing new-line at the end of the file. It seems our code was not tested with this case and it causes it to crash. I've submitted a PR to handle this case gracefully (see #669).

I think maybe one of the key missing pieces in our doc is our decision to reference the local node as localhost if no hostfile is given. So in the case of deepspeed --exclude="worker-1:0" ./finetune_trainer.py if you added your above hostfile I think it should work the way you want.

[stas00]

As I don't have access to any clusters as of yet, I'm happy to just focus on the direct necessity of being able to specify which gpu to run on.

I would recommend running this as deepspeed --include localhost:1 ./finetune_trainer.py ....

Yes, that worked! Awesome!

Could we please document that or I could make a PR to add it to the launcher doc if it will save your time.

---

BTW, I tried first:

CUDA_VISIBLE_DEVICES=1  deepspeed --include localhost:1 ./finetune_trainer.py ...

as I had it already in the script and it crashed with:

Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/bin/deepspeed", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/bin/deepspeed", line 6, in <module>
    main()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 272, in main
    active_resources = parse_inclusion_exclusion(resource_pool,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 240, in parse_inclusion_exclusion
    return parse_resource_filter(active_resources,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/launcher/runner.py", line 190, in parse_resource_filter
    raise ValueError("No slot '{}' specified on host '{}'".format(
ValueError: No slot '1' specified on host 'localhost'

So ideally the launcher should detect such situation and tell the user to not set CUDA_VISIBLE_DEVICES.

I also tried to add a hostfile with localhost slots=2 and it tries to ssh to it.

I'd say documenting:

deepspeed --include localhost:1

as a way to run a specific single GPU on a local host is all that is needed.

---

I think you may have a trailing new-line at the end of the file

You're spot on! I used nano and it can't help but leave extraneous new lines. I fixed that and it worked, but your PR is definitely a goodness. Not really needing at the moment but I'd add skipping commented out lines too.

---

let me try and address all of them here (if I miss one please let me know haha).

And as I mentioned in OP item 1. I think the --hostfile cl arg in the documentation example is in the wrong place

[skpig]

@jeffra Thanks. --include local:3 can set CUDA_VISIBLE_DEVICES=3 properly. However, the engine returned by deepspeed.initialize is on the wrong device.

Here is my code:

def debug():
    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", default=0, type=int,
                        help="local_rank for distributed training on gpus")
    parser = deepspeed.add_config_arguments(parser)
    args = parser.parse_args()
    os.environ['MASTER_ADDR'] = 'localhost'  #
    os.environ['MASTER_PORT'] = '12345'  #
    os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'
    args.local_rank = int(os.environ['LOCAL_RANK'])

    model = torch.nn.Linear(10,10)
    engine, _, _, _ = deepspeed.initialize(args=args, model=model, model_parameters=model.parameters())
    print("Engine is on device: ", next(engine.parameters()).device)

And I launch the code above with deepspeed --include="localhost:3" train.py --deepspeed --deepspeed_config config.json
I'm willing to run the model on device 3, but the engine is on device 0.

[stas00]

@skpig, if I may recommend opening a new Issue - it'd help with tracking and making sure your bug report doesn't fall between the cracks - as this Issue has been closed.