Issue with `logsumexp` with ForwardDiff (and -Inf values)

[theogf]

Here is a MWE:

julia> using ForwardDiff, LogExpFunctions

julia> x = log.([0.0; rand(10)]);

julia> ForwardDiff.gradient(logsumexp, x)
11-element Vector{Float64}:
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN
 NaN

[devmotion]

It works fine with ForwardDiff 0.10.32:

julia> using ForwardDiff, LogExpFunctions

julia> x = log.([0.0; rand(10)]);

julia> ForwardDiff.gradient(logsumexp, x)
11-element Vector{Float64}:
 0.0
 0.1378738196703953
 0.12866297288002634
 0.1500857601549507
 0.11279311035352795
 0.06729907609156742
 0.1124401188225104
 0.002025935493071411
 0.15585691506594368
 0.09113390666845002
 0.04182838479955667

So it seems this might another example that was broken by JuliaDiff/ForwardDiff.jl#481. This PR has created far too many downstream issues by now and should not have been included in a non-breaking release.