device_map bug

[wenhuach21]

with device_map, the module should not move to cuda first

[xin3he]

@wenhuach21 I'm not sure why the block is not allowed to load, I think it's not breaking the block-wise mechanism.

[wenhuach21]

With a device map, different layers are expected to be assigned to different devices, so moving the entire block to the first device doesn’t make sense.

[xin3he]

I checked it and think it's not a bug.
device_map happend after this moving, which overwrites this behavior and moves layers to different device.
m = m.to(device) ensure that norm op is on the first device?

[wenhuach21]

what if moving to the device causes out of vram