demo.py: running on cuda:0 and cuda:1 gives different results

If I add
DEVICE: 'cuda:1'
https://github.com/facebookresearch/Mask2Former/blob/main/configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml#L3

I get very different (worse) results than cuda:0 or simply comment it.
What's the possible place that caused this issue?

This is the function block that creates different results:

https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/pixel_decoder/msdeformattn.py#L324

https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py#L117

always returns to all 0 values to output if set cuda to none-zero ids (e.g. cuda:1)