HPUAccelerator: remove support in set_visible_devices_envs

[nelyahu]

The way deepspeed sets it is not correct with all HPU instances and may lead to incorrect behavior.

[delock]

Hi @nelyahu in the link https://docs.habana.ai/en/latest/PyTorch/Reference/PT_Multiple_Tenants_on_HPU/Multiple_Workloads_Single_Docker.html specificed in the code. It mentioned HABANA_VISIBLE_MODULES is used for multi-tenant on HPU, how does this usage conflict with set_visible_devices_envs() in DeepSpeed? Thanks!

[nelyahu]

@delock on Intel Gaudi accelerator (HPU), the visible module ids are determines by the hl-smi -L command, it does not always 0,1,2,3,4,5,6,7 and specially when running multiple VMs on the same server each VM can get different module
For example, the case of 2 VMs
VM#1 can get module ids 1,4,5,2
VM#2 can get 7,0,3,6

[delock]

Hi @nelyahu From the title and description this PR intends to avoid set_visible_devices_envs on this line:

DeepSpeed/deepspeed/launcher/launch.py

Line 166 in ffe0af2

get_accelerator().set_visible_devices_envs(current_env, local_accelerator_ids)

This PR also nullify this line which does not set_visible_devices_envs, is it also intended?

DeepSpeed/deepspeed/launcher/runner.py

Line 408 in ffe0af2

visible_devices = os.environ.get(visible_devices_env, "")

@delock on Intel Gaudi accelerator (HPU), the visible module ids are determines by the hl-smi -L command, it does not always 0,1,2,3,4,5,6,7 and specially when running multiple VMs on the same server each VM can get different module For example, the case of 2 VMs VM#1 can get module ids 1,4,5,2 VM#2 can get 7,0,3,6

[nelyahu]

@delock i want to avoid DeepSpeed accessing to this env var at all.
indeed causing index out of range issue there.
Maybe should replicate the CPU Accelerator behavior for now - which uses CUDA_VISIBLE_DEVICES

[delock]

Hi @nelyahu, the latest push added cosmetic change from other 8 files which is unrelated.

[nelyahu]

Hi @nelyahu, the latest push added cosmetic change from other 8 files which is unrelated.

Yes, formatter version issue. fixed