Inference performance regression vs 1.2 (NeuralNetDiffEq)

[KristofferC]

Using NeuralNetDiffEq with the code

using NeuralNetDiffEq, Flux
linear = (u,p,t) -> @. t^3 + 2*t + (t^2)*((1+3*(t^2))/(1+t+(t^3))) - u*(t + ((1+3*(t^2))/(1+t+t^3)))
linear_analytic = (u0,p,t) -> [exp(-(t^2)/2)/(1+t+t^3) + t^2]
prob = ODEProblem(ODEFunction(linear,analytic=linear_analytic),[1f0],(0.0f0,1.0f0))
chain = Flux.Chain(Dense(1,5,σ),Dense(5,1))
opt = Flux.ADAM(0.1, (0.9, 0.95))
sol  = solve(prob,NeuralNetDiffEq.NNODE(chain,opt),verbose = true, dt=1/5f0)

it finishes in about 8 seconds on 1.2 while on 1.3 it never seems to start. Interrupting it gives the following stacktrace but not sure how useful that is:

julia> @time sol  = solve(prob,NeuralNetDiffEq.NNODE(chain,opt),verbose = true, dt=1/5f0)
j^C^C^C^C^CWARNING: Force throwing a SIGINT
Internal error: encountered unexpected error in runtime:
InterruptException()
^C^C^C^Chas_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:162
has_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:159
has_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:0
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1054
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:701
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1145 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1298
forall_exists_equal at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1350
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1312
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1063
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1145 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1298
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:701
subtype_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:836
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:732
subtype_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:836
exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1374 [inlined]
forall_exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1402
jl_subtype_env at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1767
ml_matches_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2631
jl_typemap_intersection_node_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:0
jl_typemap_intersection_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/typemap.c:572
jl_typemap_intersection_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/typemap.c:563
ml_matches at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2701
jl_matching_methods at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:1803
abstract_call_gf_by_type at ./reflection.jl:821
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:21
abstract_eval_call at ./compiler/abstractinterpretation.jl:873
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
jfptr_abstract_call_606.clone_1 at /Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
abstract_apply at ./compiler/abstractinterpretation.jl:581
abstract_call at ./compiler/abstractinterpretation.jl:637
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_ext at ./compiler/typeinfer.jl:574
typeinf_ext at ./compiler/typeinfer.jl:605
jfptr_typeinf_ext_1.clone_1 at /Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2130 [inlined]
jl_apply_generic at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2300 [inlined]
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1631 [inlined]
jl_type_infer at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:209
jl_compile_method_internal at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:1877
_jl_invoke at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2300
ODEFunction at /Users/kristoffer/.julia/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230
#9 at ./none:0
mapfoldl_impl at ./generator.jl:47
#mapfoldl#186 at ./reduce.jl:72 [inlined]
mapfoldl at ./reduce.jl:72 [inlined]
#mapreduce#194 at ./reduce.jl:200 [inlined]
mapreduce at ./reduce.jl:200 [inlined]
sum at ./reduce.jl:395 [inlined]
#8 at /Users/kristoffer/Regressions13/dev/NeuralNetDiffEq/src/ode_solve.jl:47 [inlined]
#15 at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:72 [inlined]
gradient_ at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:97
#gradient#24 at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:164
unknown function (ip: 0x1215ac2ad)
gradient at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:164 [inlined]
macro expansion at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:71 [inlined]
macro expansion at /Users/kristoffer/.julia/packages/Juno/oLB1d/src/progress.jl:134 [inlined]
#train!#12 at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:69
#train! at ./none:0
unknown function (ip: 0x1215a0ae4)
#solve#2 at /Users/kristoffer/Regressions13/dev/NeuralNetDiffEq/src/ode_solve.jl:55
#solve at ./none:0

[JeffBezanson]

Looking into this.

[JeffBezanson]

Bisect points to

commit 06b4af4d57182de8d5c3ea21525924cc62c3bfaa
Author: Jameson Nash <vtjnash@gmail.com>
Date:   Fri Jun 28 12:20:33 2019 -0400

    dispatch: tolerate un-sorted tables and support non-transitive specificity

[JeffBezanson]

We are in a subtyping problem that takes ~forever inside

julia/src/gf.c

Line 2631 in bc36752

if (jl_subtype(closure->match.ti, sig2)) {

.

It's something like

julia> B = Tuple{typeof(Base.convert), Type{Tuple{Vararg{V, N} where N}}, Tuple{Vararg{V, N} where N}} where V
Tuple{typeof(convert),Type{Tuple{Vararg{V,N} where N}},Tuple{Vararg{V,N} where N}} where V

julia> struct Tracked{T}
       end

julia> A = Tuple{typeof(Base.convert), Type{T}, T} where T<:Tuple{Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}}
Tuple{typeof(convert),Type{T},T} where T<:NTuple{30,Union{Nothing, Tracked}}

julia> A<:B