fp16 support

[ffuuugor]

(possibly with autocast and loss scaling, but in our experience with Alex this may result in training instabilities). On the other hand, it reduces memory usage and increases speed (roughly a factor 2 for both).

[yuxiang-guo]

Hi, I wonder whether opacus support fp16. In my experiment, I use the fp16 but get the error:

File "/home/yxguo/anaconda3/envs/ditto/lib/python3.7/site-packages/apex/amp/handle.py", line 110, in scale_loss
if not optimizer._amp_stash.params_have_scaled_gradients:
AttributeError: 'DPOptimizer' object has no attribute '_amp_stash'

[ffuuugor]

Hi
As of this moment opacus doesn't support fp16.

This issue is created to track progress of adding the support. As of today it's not something planned for the near future (but we always welcome external pull requests)

[alidadsetan]

Hi. Any roadmap for this yet? This seems to be the main bottleneck, now that you have the "ghost clipping" in place.

[iden-kalemaj]

Hi @alidadsetan we are enabling support for mixed and fully low precision training with Opacus within the next two weeks. Please see this in progress PR: #764