From the terminal:

$ julia --math-mode=fast

(v1.1) pkg> status
    Status `~/.julia/environments/v1.1/Project.toml`
  [c5f51814] CUDAdrv v4.0.4
  [be33ccc6] CUDAnative v2.5.5
  [3a865a2d] CuArrays v1.4.7
  [...]

using CuArrays
CuArrays.CUBLAS.handle()

gives me this error:

ERROR: MethodError: Cannot `convert` an object of type Ptr{Nothing} to an object of type CuArrays.CUBLAS.cublasMath_t
...

The simple fix is to reorder arguments in this line:
cublasSetMathMode(handle, CUBLAS_TENSOR_OP_MATH)

However, the existence of this error seems to indicate that nobody is relying on the behavior that "fast-math" mode enables CUBLAS_TENSOR_OP_MATH. I would suggest making the "fast-math" independent of CUBLAS_TENSOR_OP_MATH by deleting the highlighted code here:
https://github.com/JuliaGPU/CuArrays.jl/blob/master/src/blas/CUBLAS.jl#L55-L59

If the user wants CUBLAS_TENSOR_OP_MATH, then they really want it, and they can make that explicit by calling:
cublasSetMathMode(handle(), CUBLAS_TENSOR_OP_MATH)