fp16 support

[ffuuugor]

(possibly with autocast and loss scaling, but in our experience with Alex this may result in training instabilities). On the other hand, it reduces memory usage and increases speed (roughly a factor 2 for both).

[yuxiang-guo]

Hi, I wonder whether opacus support fp16. In my experiment, I use the fp16 but get the error:

File "/home/yxguo/anaconda3/envs/ditto/lib/python3.7/site-packages/apex/amp/handle.py", line 110, in scale_loss
if not optimizer._amp_stash.params_have_scaled_gradients:
AttributeError: 'DPOptimizer' object has no attribute '_amp_stash'

[ffuuugor]

Hi
As of this moment opacus doesn't support fp16.

This issue is created to track progress of adding the support. As of today it's not something planned for the near future (but we always welcome external pull requests)