Does opencl for caffe support fp16?

[sixsamuraisoldier]

Hi there, I'm trying to figure out if this branch supports fp16 compute for the rx 480.
Also if anyone could tell me if Polaris (the rx480) supports a 2:1 ratio for fp16 that'd be great.
This person seems to think this is fp16, any confirmation?
https://techaltar.com/amd-rx-480-gpu-review/2/

Thanks in advance

[naibaf7]

@sixsamuraisoldier
Polaris RX 480 has a 1:1 ratio for FP16:FP32. FP16 on Polaris is useful to reduce memory usage and put less pressure on the memory interface.

But on the nVidia side it does not look much better for desktop GPUs at the moment:
GTX 1060, 1070 and 1080 do have a FP16:FP32 ratio of 1/64:1
For the Titan X Pascal, we don't quite know yet, there it could be either 2:1 or 1/64:1
(EDIT: The Titan X Pascal has 1/64:1 as well, making GP102, GP104 and GP106 architecturally identical, but different from P100).

FP16 for OpenCL Caffe is planned & work in progress.

For optimal performance on this OpenCL branch, enable LIBDNN when building Caffe.

[sixsamuraisoldier]

@naibaf7 Thanks so much for the info, I've been looking for that info for a long time.
Do you have any idea what's going on in the benchmark i linked to?
Is that just the increased bandwidth due to fp16?
Also, on older gpus like the r9 fury, can i reduce it to fp16 and reap similar memory size increases?
Thanks

[naibaf7]

@sixsamuraisoldier
I think that benchmark is with AMD's batched GEMM FP32. It's pretty fast but uses quite a lot of memory.
LibDNN FP32 on the RX 480 should even be a little bit faster than that, and is also applicable for backward/training pass while using NO additional working memory.

To my knowledge, Fiji, Tonga and Polaris (R9 Fury, R9 Fury X, R9 285, R9 380, RX 480) should have extended support for FP16 (meaning they can store in FP16 and also do some native commands in FP16). Just storing in FP16 should also be possible on older GPUs, such as Tahiti and Hawaii.

Give it a try :)

By the way I am a little bit concerned about that benchmark:

• First off, if they tested the classical way, this will not be the correct training time result.
• A corrected benchmark for minibatch size 128 is given with this command: ./build/tools/caffe time -model models/bvlc_alexnet/benchmark128.prototxt -gpu=0 -iterations=5
• With cuDNN properly enabled, the GTX 980 should still be about twice to three times as fast as an RX 480 or R9 390X (and not the other way around, as this bench is suggesting).
• They either tested 970/980 on cuBLAS or even on OpenCL. Not too sure which one it is, but this looks nowhere near cuDNN performance.

[gongzg]

@naibaf7 what's the status of fp16 support for OpenCL branch? Is there already any ongoing effort?

[naibaf7]

@gongzg
I am working a bit on it (and INT8) but it's difficult... i.e. it would be nice to have a better kernel code style (than what it is now with OpenCL kernels in .cl files and then string-embedding them) before adding so many new data types - just because otherwise the amount of code and compile problems will literally explode.
Ideas include:

• Switch to a structure where we we write not only kernels but also a compatibility metacode, which manifests device requirements for kernels, and if the code can be cross-compiled either from CUDA to OpenCL or vice versa. Many simple kernels would only have to be written once, and we could provide and filter multiple kernels where required.
• Switch to a single kernel code path for CUDA and OpenCL where possible, use NVRTC instead of NVCC always.
• Cache all kernels (NVRTC and OpenCL-RTC) as precompiled for each device after the first run, to still keep the speed of fixed compilation while still doing RTC.
• Unify the memory model with CUDA and OpenCL pointers in polymorphic virtual pointers (I've not forgotten your pull request here, but still think it's not the best option when considering the other changes we need to go through).
• Unify the kernel launch code between CUDA and OpenCL (with runtime polymorphy)
• Call math_functions.cpp polymorphic between OpenCL, CUDA and CPU.

If we don't do that, and for example just merge this: https://github.com/NVIDIA/caffe/tree/experimental/fp16 the compile time will go up by a significant amount, slowing development further. This is due to increased templation from <Dtype> to <Dtype, Mtype>, or in our case it will likely have to be <Dtype, Mtype, Ctype> (CPU type, device store type, device compute type).

Another problem I ran into was finding a BLAS with full FP16 support (OpenCL HGEMM/HGEMV).

But yeah, if we think about full FP16 support, we also have to find supporting libraries like the BLAS, otherwise it's a no-go. And we should, from beginning, code it in a way that will support more data types than just FP16, such as fixed-point (with INT8, http://www.jmlr.org/proceedings/papers/v37/gupta15.pdf, https://arxiv.org/abs/1605.06402) and binary (XOR-NET http://arxiv.org/abs/1511.00363)

If you ask me where you could help the most, I'd say it's writing the BLAS code for HGEMM, HGEMV, fixed point GEMM/GEMV and binary (XOR)-GEMM/GEMV.

Any suggestions on that? :) It's certainly a lot to do, possibly too much for just one or two developers to get it all done this year.

[gongzg]

@naibaf7 I agree most of your comments here. And I think we may add fp16 support step by step, here are some of my comments:

1. I would like to see a unify memory model get merged sooner than latter. I agree that the current implementation is not ideal, but it's the simplest way to merge with our current code base. :)
2. Merge Nvidia's fp16 branch may be a good direction to go as we can reuse some of the framework and save some duplicate effort. I think we can continue to solve the compilation speed issue after we finish the functional developement.
3. About the OpenCL BLAS library, we are working on integrate GEMV/GEMM for floating point data type to ISAAC currently. After that, we will continue to focus on ISAAC to optimize those half floating point kernels for Intel Gen graphics. I would like to set the binary and fixed point support to a lower priority. On intel Gen graphics platform, there is no fixed point MAD support currently, so basiclly, fixed int8 could only get identical computation capability with half floating point.
4. At last, given the fact that the NVIDIA's fp16 hasn't been accepted by the master branch yet. Does it mean that this feature is unlikely be merged into master branch? What's your thoughts about to make the OpenCL branch merged into master branch eventually? What's the major gap? Will the gap become even wider if we implement fp16 into this branch?

[naibaf7]

@gongzg

1. I understand, we'll have to re-evaluate this.
2. Agreed, however we certainly need to extend it a bit
3. Good. So my proposal is to go ahead with it as soon as ISAAC supports HGEMV and HGEMM on a level where we can run unit-tests in Caffe.
4. Yes it's quite unlikely, due to the conservative way of BVLC working with Caffe. The OpenCL branch will likely never be merged, just because it adds too much code and potential issues for users. On the good side, this also means we can be less conservative about how OpenCL Caffe works internally, so long as networks and trained networks remain 100% backward compatible, and the interface remains compatible except for device initialization.