CUDNN: STATUS_BAD_PARAM; Reason: dimA[i] <= 0

[conormcmenamin]

Hi guys, new to Julia and using my GPU to compute ML tasks so bare with me. CUDA seems to not recognise my GPU (at least I think that is the problem) and I have a codebase) that uses a CNN to perform image classification. I have attached the script and a PDF outlining what errors I get exactly. I have the errors pasted here:

Errors:

CuDNN (v8302) function cudnnConvolutionBackwardFilter() called:
    Info: Traceback contains 78 message(s)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
        Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
        Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (dyDesc->dimA[2] != 1) && (dyDesc->dimA[3] != 1)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (dyDesc->dimA[2] != 1) && (dyDesc->dimA[3] != 1)
        Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
Time: 2022-04-14T20:13:49.743134 (0d+0h+1m+23s since start)
Process=6408; Thread=12332; GPU=NULL; Handle=NULL; StreamId=NULL.

lecture3.jl.txt

[maleadt]

CUDA seems to not recognise my GPU (at least I think that is the problem)

I'm not sure I'm following; how to you figure that out from the error you run into?

Could you post an MWE or a shorted way to reproduce this instead of the entire lecture source?

[conormcmenamin]

Hi,

I'm not sure I'm following; how to you figure that out from the error you run into?

It is an assumption based on the GPU=NULL part of the error msg. I have attached a MWE pluto notebook that reproduces the same error.
MWE.txt

[conormcmenamin]

@maleadt Is there any progress with this particular issue?

[maleadt]

It is an assumption based on the GPU=NULL part of the error msg.

That's just you using device 0, rendered as NULL by CUDNN's logging system.

And no, no progress yet. I also don't expect to work on this soon.

FWIW, the code you attached doesn't exactly constitute an MWE, given how it depends on Flux, MLDatasets, Images, etc.

If this blocks your work, I recommend you dev CUDA.jl and add some code right before the failing CUDNN API call to better figure out the exact conditions that cause the error.

[Octogonapus]

This has been showing up in a number of our packages so I could spend some time investigating. WRT to:

add some code right before the failing CUDNN API call to better figure out the exact conditions that cause the error.

Unfortunately I'm not familiar with this package and I'm not sure how to do this. Could you please share some candidate code locations where I should start looking?

[maleadt]

You presumably get a Julia backtrace when the error happens? You can also just look for the function mentioned in the error message, cudnnConvolutionBackwardFilter. It appears that function isn't even being called here, but from NNlibCUDA, so this issue should arguably move there.

[Octogonapus]

I don't get much. Just logs like

┌ Error: 2022-05-25 18:38:04 tid:1 CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2022-05-25T14:01:38.985533 (0d+0h+3m+52s since start)
│ Process=136010; Thread=136010; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/fAEDi/lib/cudnn/CUDNN.jl:140

I didn't know this came from NNlibCUDA, thanks. Will start looking there

[maleadt]

So it's just an error log message from CUDNN that otherwise doesn't hurt execution (i.e. the BAD_PARAM error isn't caught by the caller)? Still a bad sign, of course.

[Octogonapus]

Indeed it seems to have no effect on execution. They do sort of spam our logs, though, so I wanted to get to the bottom of it.

[maleadt]

The fact that the library wrapper (which I presume is the path to this API call that's being used; you could try confirming that by adding some print statements surrounding the ccall), uses the @checked macro, https://github.com/JuliaGPU/CUDA.jl/blob/ff321312c8cdf47d7f9d21e72e40496639cc59dd/lib/cudnn/libcudnn.jl#L599-L610=, should imply that the API call's return code is checked. So it's weird that the logs mention an ERROR_BAD_PARAM, but that this is never returned to the outside.

Again, I'd suggest adding some debug output surrounding the API call so you can reproduce it in isolation. If you can then show that invoking CUDNN in a certain manner generates a BAD_PARAM log message without signalling error, that's arguably a CUDNN bug that could be filed with NVIDIA.