NaN caused by tanh_fast

[zhubonan]

FluxML/Flux.jl#1776 subsitutes tanh by tanh_fast, howeer, the latter gives NaN for large numbers:

I am on Julia 1.7.2 and Flux 0.13.

using Flux

julia> tanh_fast(1e10)
NaN

julia> tanh_fast(-1e10)
-1.0

julia> d = Dense(1=>1, tanh)
Dense(1 => 1, tanh)  # 2 parameters

julia> d([-1.e16])
1-element Vector{Float64}:
 NaN

julia> d([1.e16])
1-element Vector{Float64}:
 -1.0

NaN would then propagate down the network....

[zhubonan]

Err may be this should be raised to NNLib.jl instead?

[mcabbott]

I think that's a stupid bug: There's a test which should use a constant for x > sqrt(900) to avoid this, but it uses sign(y) not sign(x).

NNlib.jl/src/activations.jl

Lines 776 to 784 in 886b34c

	@inline function tanh_fast(x::Float64)
	exp2x = @fastmath exp(x + x)
	y = (exp2x - 1) / (exp2x + 1)
	# That has large errors near zero; using `expm1` would more accurate, but about as slow as `tanh`.
	# Instead, we switch to a polynomial, which is very accurate within its range:
	x2 = x * x
	ypoly = x * evalpoly(x2, (1.0, -0.33333333333324583, 0.13333333325511604, -0.05396823125794372, 0.02186660872609521, -0.008697141630499953))
	ifelse(x2 > 900.0, sign(y), ifelse(x2 < 0.017, oftype(y, ypoly), y))
	end