Leaky relu failing tests

[AdarshKumar712]

When I am trying to check for gradient float preserving tests seperately, it is failing tests:

julia> using Test

julia> using Zygote

julia> using NNlib

julia> typeof(leakyrelu'(Float32(0.5)))
Float64

julia> function test_gradient_float_precision_preserving(a)
           @testset "$(a): " begin
               for T in [Float32, Float64]
                   for val in [-10, -1, 0, 1, 10]
                       val = @inferred a'(T(val))
                       @test typeof(val) == T
                   end
               end
           end
       end
test_gradient_float_precision_preserving (generic function with 1 method)

julia> test_gradient_float_precision_preserving(leakyrelu)
leakyrelu: : Test Failed at REPL[5]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[5]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::Function) at ./REPL[5]:3
leakyrelu: : Test Failed at REPL[5]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[5]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::Function) at ./REPL[5]:3
leakyrelu: : Test Failed at REPL[5]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[5]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::Function) at ./REPL[5]:3
leakyrelu: : Test Failed at REPL[5]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[5]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::Function) at ./REPL[5]:3
leakyrelu: : Test Failed at REPL[5]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[5]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::Function) at ./REPL[5]:3
Test Summary: | Pass  Fail  Total
leakyrelu:    |    5     5     10
ERROR: Some tests did not pass: 5 passed, 5 failed, 0 errored, 0 broken.

PS. I understood the reason for this. This might be because of version difference between master branch and released version of NNlib

[AdarshKumar712]

But test seems to be pass, if I redefine Leakyrelu from the master branch code.

julia> NNlib.leakyrelu(x::Real, a = oftype(x / 1, 0.01)) = max(a * x, x / one(x))

julia> test_gradient_float_precision_preserving(NNlib.leakyrelu)
Test Summary: | Pass  Total
leakyrelu:    |   10     10
Test.DefaultTestSet("leakyrelu: ", Any[], 10, false)

However, the above definition would still fail in the following case:

julia> function test_gradient_float_precision_preserving(a)
           @testset "$(a): " begin
               for T in [Float32, Float64]
                   for val in [-10, -1, 0, 1, 10]
                       val = @inferred gradient(x->a(x,0.01),T(val))[1]
                       @test typeof(val) == T
                   end
               end
           end
       end
test_gradient_float_precision_preserving (generic function with 1 method)

julia> test_gradient_float_precision_preserving(NNlib.leakyrelu)
leakyrelu: : Test Failed at REPL[10]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[10]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::typeof(leakyrelu)) at ./REPL[10]:3
leakyrelu: : Test Failed at REPL[10]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[10]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::typeof(leakyrelu)) at ./REPL[10]:3
leakyrelu: : Test Failed at REPL[10]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[10]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::typeof(leakyrelu)) at ./REPL[10]:3
leakyrelu: : Test Failed at REPL[10]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[10]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::typeof(leakyrelu)) at ./REPL[10]:3
leakyrelu: : Test Failed at REPL[10]:6
  Expression: typeof(val) == T
   Evaluated: Float64 == Float32
Stacktrace:
 [1] macro expansion at ./REPL[10]:6 [inlined]
 [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
 [3] test_gradient_float_precision_preserving(::typeof(leakyrelu)) at ./REPL[10]:3
Test Summary: | Pass  Fail  Total
leakyrelu:    |    5     5     10
ERROR: Some tests did not pass: 5 passed, 5 failed, 0 errored, 0 broken.

Difference: I passed the second parameter of leakyrelu as well for gradient float precision test

[matsueushi]

I think the parameter should be typecasted to T

val = @inferred gradient(x->a(x,T(0.01)),T(val))[1]

[AdarshKumar712]

No, actually we don't usually give the type of paramter. Moreover I think this can be handelled if change the leakyrelu to automatically adjust the type of parameter.

julia> NNlib.leakyrelu(x::Real, a = oftype(x / 1, 0.01)) = max(oftype(x/1,a) * x, x / one(x))

julia> function test_gradient_float_precision_preserving(a)
           @testset "$(a): " begin
               for T in [Float32, Float64]
                   for val in [-10, -1, 0, 1, 10]
                       val = @inferred gradient(x->a(x,0.01),T(val))[1]
                       @test typeof(val) == T
                   end
               end
           end
       end
test_gradient_float_precision_preserving (generic function with 1 method)

julia> test_gradient_float_precision_preserving(NNlib.leakyrelu)
Test Summary: | Pass  Total
leakyrelu:    |   10     10
Test.DefaultTestSet("leakyrelu: ", Any[], 10, false)

[matsueushi]

I missed the point (because I have been using activation functions with typed parameters, like lealyrelu(x, 0.2f0). It might be worth to create a PR if many people use in that way, I think.

[AdarshKumar712]

Yes, I guess changing the Leakyrelu function definition for that would ensure float precision, independent of how the user defines it.
Would it better if I make this change in #175 itself or should I create a seperate PR on that?