Cuda 3.0 support

[DhairyaLGandhi]

cc @maleadt , @jonathan-laurent

[DhairyaLGandhi]

@jonathan-laurent @maleadt could you confirm that this branch works fine for AlphaZero and CUDA 3. And also whether you've tested the subpackage from FluxML/NNlib.jl#286

[CarloLucibello]

you can test the NNlibCUDA´s subpackage with

(@v1.6) pkg> add https://github.com/FluxML/NNlib.jl:lib/NNlibCUDA

[CarloLucibello]

It´s a bit tricky to support both julia 1.5 and julia 1.6 at this point (NNlibCUDA requires julia 1.6). Should we drop julia 1.6 support?

[CarloLucibello]

the Manifest here is adding CUDA 2.6 actually

[maleadt]

Now that we have device overrides, Base.exp etc are GPU compatible, so there's a couple of GPU-specific invocations of exp, log, ... that can go.

[jonathan-laurent]

What exactly should I do to test this? I tried the following without success:

]add Flux#dg/cuda16 NNlib#master CUDA@3.0.3

[DhairyaLGandhi]

You'd need to check out a local copy and instantiate the manifest in the branch. You also need https://github.com/FluxML/NNlibCUDA.jl

[jonathan-laurent]

AlphaZero.jl works fine with CUDA 3.0 and this branch.

[DhairyaLGandhi]

I think this is good to go, anyone have thoughts?

[CarloLucibello]

the NNlibCUDA compat bound is missing and the Manifest could be removed. Beside that, looks good!

[DhairyaLGandhi]

Thanks, added the bound, and I'd like to keep the manifest around for now, so I think we are good to go here.

[jonathan-laurent]

Is the plan to merge this in the next minor release of Flux? (I imagine this cannot be part of a patch release if you are dropping support for Julia <1.6).

[DhairyaLGandhi]

Yeah, I was considering merging and releasing it now. Its allowed within semver to limit Julia versions, and we aren't breaking any Flux API so we are good. Any downstream breaking changes would need depending on CUDA explicitly so we are covered with a patch release.

[DhairyaLGandhi]

ref FluxML/NNlib.jl#316

Its not a blocker, but good to have it be complete. On 1.6 (which is what Flux will bump Julia to), we should see the correct bounds.