How to use multi-gpu quantization

[rexmxw02]

How to use multi-gpu quantization
when i operate :
export MODEL_NAME_OR_PATH="/llama2_70B"
export OUTPUT_DIR="/llama2_70B_quantize"
export QUANTIZATION_SCHEME="fp8"
export DEVICE="cuda:0,1,2,3,4"
export DEVICE="cuda:0,1,2,3" # Device to run the quantization process. Defaults to "cuda:0".
fmo quantize --dataset-name-or-path $DATASET_NAME_OR_PATH --model-name-or-path $MODEL_NAME_OR_PATH --output-dir $OUTPUT_DIR --mode $QUANTIZATION_SCHEME --device $DEVICE

it shows fmo.main ERROR: Invalid device string: 'cuda:0,1,2,3',so how can i do?
Loading checkpoint shards: 100%|███████████████████████████████████████| 4/4 [00:00<00:00, 13.39it/s]
2024-11-07 12:39:35,853.00853: fmo.main ERROR: Invalid device string: 'cuda:0,1,2,3'

[seungduk-yanolja]

+1
it seems a bit slow with a single GPU