Incomplete CUDNN wrappers

[denizyuret]

It doesn't look like RNNDesc has information about number of layers, bidirectional, etc. params() does not seem to be aware of multiple layers if I am not mistaken. Is this an incomplete interface? What does Flux use? (I can help complete it)

[denizyuret]

I think we should have a general discussion about the CUDNN interface and the NNlib interface for e.g. conv, rnn, and batchnorm. Currently the NNlib conv interface is missing the group parameter, the pool interface does not support mode=2, there is no rnn interface as far as I can see. Before I go in and help complete some of this, I would like to find what the general design philosophy is so I don't break it.

[maleadt]

Bit of a late comment, and we've discussed some of this on the call already. Basically there's not much of a design philosophy from the CUDA.jl side of things; we just aim to expose the CUDNN functionality (staying close to the CUDNN interface, but raising the abstraction level where possible), and use it to implement interfaces like NNlib's. So I guess the real discussion would be how to structure the NNlib interface.

Concerning the CUDNN wrappers here: I'd like them to be usable by themselves, i.e. without having to use NNlib or Flux, but that isn't currently the case because they were developed together with Flux and moved them out at some point (hence the coupling). I'd like to undo that and provide isolated tests, as we've done with the other libraries, but this hasn't been a high-priority item.

[denizyuret]

Bit of a late comment, and we've discussed some of this on the call already. Basically there's not much of a design philosophy from the CUDA.jl side of things; we just aim to expose the CUDNN functionality (staying close to the CUDNN interface, but raising the abstraction level where possible), and use it to implement interfaces like NNlib's. So I guess the real discussion would be how to structure the NNlib interface.

Who is the decider on this?

Concerning the CUDNN wrappers here: I'd like them to be usable by themselves, i.e. without having to use NNlib or Flux, but that isn't currently the case because they were developed together with Flux and moved them out at some point (hence the coupling). I'd like to undo that and provide isolated tests, as we've done with the other libraries, but this hasn't been a high-priority item.

I assume you mean the intermediate level between the automatically generated low-level cudnn calls and the NNlib functions? From existing examples I see the same function names kept, but long argument lists replaced with CuArrays etc. If we could agree on a standard I can work on this.

[maleadt]

Who is the decider on this?

NNlib maintainers?

I assume you mean the intermediate level between the automatically generated low-level cudnn calls and the NNlib functions? From existing examples I see the same function names kept, but long argument lists replaced with CuArrays etc. If we could agree on a standard I can work on this.

Yeah, the CUBLAS wrappers are a better example here: libcublas.jl for the autogenerated stuff, wrappers.jl with low-level but much more user-friendly functionality (using multiple dispatch, array types, e.g., scal!(..., ::CuArray{T}) dispatching to cublasSscal_v2 or cublasDscal_v2), and then various integrations like linearalgebra.jl which use said low-level wrappers to implement various interfaces.

With CUDNN, the two are heavily intertwined. Maybe it doesn't make sense to separate them out as strongly as with CUBLAS, but it would benefit testability, and not constrain users to the NNlib interfaces.

[maleadt]

@denizyuret any update on the CUDNN rework?

[denizyuret]

Still incomplete at denizyuret/Knet.jl#614. If you'd like I can strip Knet specific stuff and create a CUDA.jl pull request within a week, at least for feedback.

[maleadt]

Yeah that might be interesting, that way we can do development in the place the code will actually be (assuming Knet will use the CUDNN submodule from CUDA.jl).

[denizyuret]

I have the following problem: We have the low level library call and the high level user API. Neither is appropriate as an AutoGrad primitive. For a function defined as an AutoGrad primitive:

• It must return an output value (low level cudnn calls do not).
• Preferably the positional args should be the ones that have gradients, the rest should be delegated to keyword args (low level cudnn calls have many other positional args, high level API may make gradient args optional keywords).
• Some cudnn functions have multiple backward functions that need to be called in a particular order, which may not match the order of function arguments.

Anyway, details aside I solve the problem by defining an intermediate function (that gets called by the high level API and does the low level library call) that satisfies the AutoGrad requirements. If I remove this intermediate function, Knet will not be able to take advantage of the new API. If I leave it in, it may not satisfy all the other autograd packages out there. How do you think I should proceed?

[denizyuret]

I am preparing a PR at https://github.com/denizyuret/CUDA.jl/tree/dy/cudnn.

[denizyuret]

@maleadt @DhairyaLGandhi : I worked on CUDA/lib/cudnn/nnlib.jl and have a few observations, suggestions:

• I think this file should not be part of / tested in CUDA.jl. Flux.jl may be a better home for it.
• CUDA/test/cudnn.jl should test the actual cudnn calls rather than NNlib functions.
• softmax with dims other than 1 were buggy, fixed now.
• softmax is THE most expensive operation in many language models and the current NNlib interface is forcing ∇softmax to recompute softmax(x). This can be fixed if ∇softmax accepts y as an argument instead of x.

[maleadt]

• I think this file should not be part of / tested in CUDA.jl. Flux.jl may be a better home for it.

• CUDA/test/cudnn.jl should test the actual cudnn calls rather than NNlib functions.

Other libraries, like CUSOLVER similarly integrate with, say, LinearAlgebra, so the tests are two-fold: first the low-level API calls, then the high-level interface integrations. Or does the NNlib integration (or tests thereof) depend on Flux-like functionality?