os.environ['CUDA_VISIBLE_DEVICES'] = '1' does not work in jupyter

[imhuay]

As the title says, it is invalid to specify the GPU through CUDA_VISIBLE_DEVICES in jupyter, and only 'GPU:0' will still be used; but it is effective when using CUDA_VISIBLE_DEVICES=1 python *.py

[zhuohan123]

Did you set CUDA_VISIBLE_DEVICES before you import any vLLM/Pytorch/Ray related packages? Maybe it's because you set the environment variable too late so it will not be effective.