Mixed precision training.

[KristofferC]

It would be interesting to see what we need to do to be able to do mixed precision training with Flux (https://forums.fast.ai/t/mixed-precision-training/20720, http://on-demand.gputechconf.com/gtc/2018/video/S81012/). With the way new hardware are moving it seems it will likely be important to support.

Right now, I think the major obstacle to overcome is the implementation of Float16 in Julia. AFAIU, these get sent to LLVM as Int16 and things are implemented by widening to Float32 and then truncating.

I think Valentin worked a bit on this (JuliaLang/julia#26381) so perhaps he (you) could give some input, cc @vchuravy, also cc @Keno.

[MikeInnes]

Note that mixed precision training requires having new kernels. We want to operate on CuArray{Float16}, but every does an internal conversion to f32, computes, and then converts back to f16 for intermediate storage. But to experiment with a simple NN we only need broadcast, reduce and some CUDNN wrappers.

[KristofferC]

AFAIU, the full forward and backward pass are done in half precision, and the update to the existing weights (using a single precision "master" copy of the weights) is in single precision.

[MikeInnes]

See for example this slide deck which discusses keeping weights, activations and parameters in f16 while doing on-device calculation and accumulation in f32 (particularly slide 19).

Half precision training would be a useful first step for this though.

[AriMKatz]

https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/

https://docs.fast.ai/callback.fp16.html

[CarloLucibello]

closing in favor of #2291 with more up-to-date discussion