Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NewRecur experimental interface #11

Merged
merged 13 commits into from
Aug 9, 2023
2 changes: 2 additions & 0 deletions src/Fluxperimental.jl
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,6 @@ include("chain.jl")

include("compact.jl")

include("new_recur.jl")

end # module Fluxperimental
105 changes: 105 additions & 0 deletions src/new_recur.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@



##### Helper scan funtion which can likely be put into NNLib. #####
"""
scan

Recreating jax.lax.scan functionality in julia.
"""
function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
# xs = Flux.eachlastdim(x_block)
x_init, x_rest = Iterators.peel(xs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
x_init, x_rest = Iterators.peel(xs)
x_init = first(xs)

Since x_rest isn't used. Or is peel more efficient in the contexts we care about?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use x_rest in the foldl call to go through the rest of the sequence.

The reason peel was chosen here was not efficiency, but oddly gradient related. If instead I did

x_init = first(xs)
x_rest = xs[begin+1:end]

the resulting gradients were wrong. I've since documented this in the function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems very concerning. Do you have a MWE?

Copy link
Contributor Author

@mkschleg mkschleg Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, no mwe. Commenting out the peel and replacing with the code block, I haven't been able to figure out where the gradients are going wrong here. Only know the tests fail.


(carry, out_) = func(init_carry, x_init)

init = (typeof(out_)[out_], carry)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also didn't quite understand why this is using carry instead of init_carry, can you walk me through that? AIUI this requires xs to be non-empty

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. xs is currently assumed to be non-empty. I've played around with only using foldl for this interface, but honestly I can't figure it out (i.e. Zygote hates it). I'm not sure why, and the error message is unfortunate. For this first pass, I think we should just do this, and then maybe bisect this choice and try and figure out why Zygote is not liking the other version in another issue/PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that Lux handles this with less code. Can we do the same? I don't think a dedicated rule is required for foldl like theye have, however.

Copy link
Contributor Author

@mkschleg mkschleg Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, so if we try and use a similar implementation to the current Lux implementation it would look like the below.

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
  function __recurrence_op(::Nothing, input)
    carry, out = func(init_carry, input)
    return carry, [out]
  end

  # recurrence operation used in the fold. Takes the state  of the
  # folde and the next input, returns the new state.
  function __recurrence_op((carry, outputs), input)
    carry, out = func(carry, input)
    return carry, vcat(outputs, [out])
  end
  # Fold left to right.
  foldl(__recurrence_op, xs; init=nothing)
end

Unfortunately, gradients won't compile afaict. I'm not sure why we don't get the same behavior as Lux.jl. Also preemptively Lux's foldl_init just calls an inline for foldl. Using foldl_init also fails in a similar way.

Here is the stack trace when running the tests. I have no idea what to takeaway from this error, and any help would be great.

Stack Trace

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
  Got exception outside of a @test
  MethodError: Cannot `convert` an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}}

  Closest candidates are:
    convert(::Type{T}, ::T) where T
     @ Base Base.jl:64

  Stacktrace:
    [1] cvt1
      @ ./essentials.jl:418 [inlined]
    [2] ntuple
      @ ./ntuple.jl:49 [inlined]
    [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}})
      @ Base ./essentials.jl:419
    [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64)
      @ Base ./array.jl:969
    [5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
      @ Base ./accumulate.jl:432
    [6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Nothing, Nothing}})
      @ Base ./accumulate.jl:361
    [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Nothing, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Nothing, Nothing}}}})
      @ Base ./accumulate.jl:342
    [8] accumulate!
      @ ./accumulate.jl:337 [inlined]
    [9] #accumulate#872
      @ ./accumulate.jl:285 [inlined]
   [10] accumulate
      @ ./accumulate.jl:272 [inlined]
   [11] #rrule#1714
      @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined]
   [12] rrule
      @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined]
   [13] chain_rrule_kw
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined]
   [14] macro expansion
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
   [15] _pullback
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
   [16] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:56 [inlined]
   [17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [18] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:70 [inlined]
   [19] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:142 [inlined]
   [20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [21] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:131 [inlined]
   [22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [23] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
   [24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
   [26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
   [27] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
   [28] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [29] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
   [30] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [31] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
   [32] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [33] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
   [34] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [35] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
   [36] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [37] top-level scope
      @ none:6
   [38] eval
      @ ./boot.jl:370 [inlined]
   [39] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [40] _start()
      @ Base ./client.jl:522

Copy link
Member

@ToucheSir ToucheSir Jul 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha I think I know what this is. Long story short, ChainRules tries to store an array of pullbacks, but since both method overloads have slightly different captured variables, they result in different pullbacks. Now I know why Lux has custom rrules for this: they erase the type of the pullback array to avoid the conversion error.

Does the following change work?

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
  function __recurrence_op(::Tuple{Nothing, Nothing}, input)
    carry, out = func(init_carry, input)
    return carry, [out]
  end

  # recurrence operation used in the fold. Takes the state  of the
  function __recurrence_op((carry, outputs), input)
  ...
  end

  # Fold left to right.
  foldl(__recurrence_op, xs; init=(nothing, nothing))
end

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error do you get using the second one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems similar to the previous error.

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
  Got exception outside of a @test
  MethodError: Cannot `convert` an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}}

  Closest candidates are:
    convert(::Type{T}, ::T) where T
     @ Base Base.jl:64

  Stacktrace:
    [1] cvt1
      @ ./essentials.jl:418 [inlined]
    [2] ntuple
      @ ./ntuple.jl:49 [inlined]
    [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}})
      @ Base ./essentials.jl:419
    [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64)
      @ Base ./array.jl:969
    [5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
      @ Base ./accumulate.jl:432
    [6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Nothing, Nothing}, Nothing}})
      @ Base ./accumulate.jl:361
    [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Nothing, Nothing}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Nothing, Nothing}, Nothing}}}})
      @ Base ./accumulate.jl:342
    [8] accumulate!
      @ ./accumulate.jl:337 [inlined]
    [9] #accumulate#872
      @ ./accumulate.jl:285 [inlined]
   [10] accumulate
      @ ./accumulate.jl:272 [inlined]
   [11] #rrule#1714
      @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined]
   [12] rrule
      @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined]
   [13] chain_rrule_kw
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined]
   [14] macro expansion
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
   [15] _pullback
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
   [16] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:50 [inlined]
   [17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [18] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:106 [inlined]
   [19] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:178 [inlined]
   [20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [21] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:167 [inlined]
   [22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [23] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
   [24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
   [26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
   [27] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
   [28] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [29] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
   [30] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [31] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
   [32] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [33] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
   [34] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [35] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
   [36] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [37] top-level scope
      @ none:6
   [38] eval
      @ ./boot.jl:370 [inlined]
   [39] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [40] _start()
      @ Base ./client.jl:522

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/rulesets/Base/mapreduce.jl:440

This is https://github.com/JuliaDiff/ChainRules.jl/blob/main/src/rulesets/Base/mapreduce.jl#L440 which is part of the rule for foldl. Which I wrote and is a bit of a monster.

If it's easy to do so, trying with JuliaDiff/ChainRules.jl#569 might be worthwhile This re-writes the rule to solve various problems, esp. to do with init.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using mapfoldl_impl would work well enough, here's a rough sketch:

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
   # Recurrence operation used in the fold. Takes the state of the
   # fold and the next input, returns the new state.
   function recurrence_op((carry, outputs), input)
       carry, out = func(carry, input)
       return carry, vcat(outputs, [out])
   end
   # Fold left to right.
   return Base.mapfoldl_impl(identity, recurrence_op, (init_carry, empty(xs)), xs)
end

function rrule(
        config::RuleConfig{>:HasReverseMode}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::G, init, x::Union{AbstractArray, Tuple};
    ) where {G}
    hobbits = accumulate(x; init=(init, nothing)) do (a, _), b
        c, back = rrule_via_ad(config, op, a, b)
    end
    y = first(last(hobbits))
    axe = axes(x)
    project = ProjectTo(x)
    function unfoldl(dy)
        trio = accumulate(Iterators.reverse(hobbits); init=(0, dy, 0)) do (_, dc, _), (_, back)
            ds, da, db = back(dc)
        end
        dop = sum(first, trio)
        dx = map(last, Iterators.reverse(trio))
        d_init = trio[end][2]
        return (NoTangent(), NoTangent(), dop, d_init, project(reshape(dx, axe)))
    end
    return y, unfoldl
end

I think the problem is that the rrule which takes an Array in that PR doesn't provide a gradient for the init value. I don't think it would be too hard to add (see simplified method above) but I may be missing something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a similar error still, and it is in the line hobbits = accumulate.... I'm not too sure how to debug this, as it is a convert issue when accumulating the gradient. When I get a bit of time I will try and debug a bit more, but I'm at a loss with the current error, so any help would be appreciated.

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
  Got exception outside of a @test
  MethodError: Cannot `convert` an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}} to an object of type
    Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}

  Closest candidates are:
    convert(::Type{T}, ::T) where T
     @ Base Base.jl:64

  Stacktrace:
    [1] cvt1
      @ ./essentials.jl:418 [inlined]
    [2] ntuple
      @ ./ntuple.jl:49 [inlined]
    [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}})
      @ Base ./essentials.jl:419
    [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}}, i1::Int64)
      @ Base ./array.jl:969
    [5] _accumulate1!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{Any}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
      @ Base ./accumulate.jl:432
    [6] _accumulate!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}})
      @ Base ./accumulate.jl:361
    [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}})
      @ Base ./accumulate.jl:342
    [8] accumulate(op::Function, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}})
      @ Base ./accumulate.jl:285
    [9] accumulate
      @ ./accumulate.jl:272 [inlined]
   [10] rrule(config::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, init::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, x::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
      @ Fluxperimental ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:48
   [11] chain_rrule(::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::Function, ::Function, ::Vararg{Any})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:223
   [12] macro expansion
      @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
   [13] _pullback(::Zygote.Context{true}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), ::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101
   [14] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:42 [inlined]
   [15] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [16] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:137 [inlined]
   [17] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:209 [inlined]
   [18] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [19] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:198 [inlined]
   [20] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [21] _pullback
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
   [22] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
   [23] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
   [24] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
      @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
   [25] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
   [26] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [27] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
   [28] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [29] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
   [30] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [31] macro expansion
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
   [32] macro expansion
      @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
   [33] top-level scope
      @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
   [34] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [35] top-level scope
      @ none:6
   [36] eval
      @ ./boot.jl:370 [inlined]
   [37] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [38] _start()
      @ Base ./client.jl:522


function recurrence_op(input, (outputs, carry))
carry, out = func(carry, input)
return vcat(outputs, typeof(out)[out]), carry
end
results = foldr(recurrence_op, xs[(begin+1):end]; init)
results[2], results[1]
mkschleg marked this conversation as resolved.
Show resolved Hide resolved
end

function scan_full(func, init_carry, x_block)
xs_ = Flux.eachlastdim(x_block)
xs = if xs_ isa Base.Generator
collect(xs_) # eachlastdim produces a generator in non-gradient environment
else
xs_
end
scan_full(func, init_carry, xs)
end

function scan_partial(func, init_carry, xs::AbstractVector{<:AbstractArray})
x_init, x_rest = Iterators.peel(xs)
(carry, y) = func(init_carry, x_init)
for x in x_rest
(carry, y) = func(carry, x)
end
# carry, y
carry, y
end

function scan_partial(func, init_carry, x_block)
xs_ = Flux.eachlastdim(x_block)
xs = if xs_ isa Base.Generator
collect(xs_) # eachlastdim produces a generator in non-gradient environment
else
xs_
end
scan_partial(func, init_carry, xs)
end


"""
NewRecur
New Recur. An experimental recur interface for removing statefullness in recurrent architectures for flux.
"""
struct NewRecur{RET_SEQUENCE, T}
cell::T
# state::S
function NewRecur(cell; return_sequence::Bool=false)
new{return_sequence, typeof(cell)}(cell)
end
function NewRecur{true}(cell)
new{true, typeof(cell)}(cell)
end
function NewRecur{false}(cell)
new{false, typeof(cell)}(cell)
end
end

Flux.@functor NewRecur
Flux.trainable(a::NewRecur) = (; cell = a.cell)
Base.show(io::IO, m::NewRecur) = print(io, "Recur(", m.cell, ")")
NewRNN(a...; return_sequence::Bool=false, ka...) = NewRecur(Flux.RNNCell(a...; ka...); return_sequence=return_sequence)


(l::NewRecur)(init_carry, x_mat::AbstractMatrix) = MethodError("Matrix is ambiguous with NewRecur")
(l::NewRecur)(init_carry, x_mat::AbstractVector{T}) where {T<:Number} = MethodError("Vector is ambiguous with NewRecur")

(l::NewRecur)(xs) = l(l.cell.state0, xs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(l::NewRecur)(xs) = l(l.cell.state0, xs)
(l::NewRecur)(xs::AbstractArray) = l(l.cell.state0, xs)

For your consideration

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likely this would be a good idea. But I didn't restrict this in this initial pass.



function (l::NewRecur{false})(init_carry,
xs)
results = scan_partial(l.cell, init_carry, xs)
results[2]
end
mkschleg marked this conversation as resolved.
Show resolved Hide resolved

# From Lux.jl: https://github.com/LuxDL/Lux.jl/pull/287/
function (l::NewRecur{true})(init_carry,
xs,)

results = scan_full(l.cell, init_carry, xs)

h = results[2]
sze = size(h[1])
reshape(reduce(hcat, h), sze[1], sze[2], length(h))
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
function (l::NewRecur{true})(init_carry,
xs,)
results = scan_full(l.cell, init_carry, xs)
h = results[2]
sze = size(h[1])
reshape(reduce(hcat, h), sze[1], sze[2], length(h))
end
function (l::NewRecur{true})(init_carry, xs)
results = scan_full(l.cell, init_carry, xs)
return results[1], stack(results[2])
end

Similar story here. stack vs reduce(hcat are more or less equally efficient so feel free to use either.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this to stack like you suggested.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize stack is not available in 1.6. This function was added to 1.9. Are we targeting only 1.9?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's available via Compat.jl, which Flux already has as a transitive dep so there's zero additional import overhead.




182 changes: 182 additions & 0 deletions test/new_recur.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@


@testset "NewRecur RNN" begin
@testset "Forward Pass" begin
cell = Flux.RNNCell(1, 1, identity)
layer = Fluxperimental.NewRecur(cell; return_sequence=true)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = reshape([2.0f0, 3.0f0], 1, 1, 2)

# @show layer(x)
@test eltype(layer(x)) <: Float32
@test size(layer(x)) == (1, 1, 2)

@test_throws MethodError layer([2.0f0])
@test_throws MethodError layer([2.0f0;; 3.0f0])
end


@testset "gradients-implicit" begin
cell = Flux.RNNCell(1, 1, identity)
layer = Flux.Recur(cell)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = [[2.0f0], [3.0f0]]

# theoretical primal gradients
primal =
layer.cell.Wh .* (layer.cell.Wh * layer.cell.state0 .+ x[1] .* layer.cell.Wi) .+
x[2] .* layer.cell.Wi
∇Wi = x[1] .* layer.cell.Wh .+ x[2]
∇Wh = 2 .* layer.cell.Wh .* layer.cell.state0 .+ x[1] .* layer.cell.Wi
∇b = layer.cell.Wh .+ 1
∇state0 = layer.cell.Wh .^ 2

nm_layer = Fluxperimental.NewRecur(cell; return_sequence = true)
ps = Flux.params(nm_layer)
x_block = reshape(vcat(x...), 1, 1, length(x))
e, g = Flux.withgradient(ps) do
out = nm_layer(x_block)
sum(out[1, 1, 2])
end

@test primal[1] ≈ e
@test ∇Wi ≈ g[ps[1]]
@test ∇Wh ≈ g[ps[2]]
@test ∇b ≈ g[ps[3]]
@test ∇state0 ≈ g[ps[4]]
end


@testset "gradients-explicit" begin

cell = Flux.RNNCell(1, 1, identity)
layer = Flux.Recur(cell)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = [[2.0f0], [3.0f0]]

# theoretical primal gradients
primal =
layer.cell.Wh .* (layer.cell.Wh * layer.cell.state0 .+ x[1] .* layer.cell.Wi) .+
x[2] .* layer.cell.Wi
∇Wi = x[1] .* layer.cell.Wh .+ x[2]
∇Wh = 2 .* layer.cell.Wh .* layer.cell.state0 .+ x[1] .* layer.cell.Wi
∇b = layer.cell.Wh .+ 1
∇state0 = layer.cell.Wh .^ 2


x_block = reshape(vcat(x...), 1, 1, length(x))
nm_layer = Fluxperimental.NewRecur(cell; return_sequence = true)
e, g = Flux.withgradient(nm_layer) do layer
out = layer(x_block)
sum(out[1, 1, 2])
end
grads = g[1][:cell]

@test primal[1] ≈ e
@test ∇Wi ≈ grads[:Wi]
@test ∇Wh ≈ grads[:Wh]
@test ∇b ≈ grads[:b]
@test ∇state0 ≈ grads[:state0]

end
end

@testset "New Recur RNN Partial Sequence" begin

@testset "Forward Pass" begin
cell = Flux.RNNCell(1, 1, identity)
layer = Fluxperimental.NewRecur(cell)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = reshape([2.0f0, 3.0f0], 1, 1, 2)

@test eltype(layer(x)) <: Float32
@test size(layer(x)) == (1, 1)

@test_throws MethodError layer([2.0f0])
@test_throws MethodError layer([2.0f0;; 3.0f0])

end

@testset "gradients-implicit" begin
cell = Flux.RNNCell(1, 1, identity)
layer = Flux.Recur(cell)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = [[2.0f0], [3.0f0]]

# theoretical primal gradients
primal =
layer.cell.Wh .* (layer.cell.Wh * layer.cell.state0 .+ x[1] .* layer.cell.Wi) .+
x[2] .* layer.cell.Wi
∇Wi = x[1] .* layer.cell.Wh .+ x[2]
∇Wh = 2 .* layer.cell.Wh .* layer.cell.state0 .+ x[1] .* layer.cell.Wi
∇b = layer.cell.Wh .+ 1
∇state0 = layer.cell.Wh .^ 2

nm_layer = Fluxperimental.NewRecur(cell; return_sequence = false)
ps = Flux.params(nm_layer)
x_block = reshape(vcat(x...), 1, 1, length(x))
e, g = Flux.withgradient(ps) do
out = (nm_layer)(x_block)
sum(out)
end

@test primal[1] ≈ e
@test ∇Wi ≈ g[ps[1]]
@test ∇Wh ≈ g[ps[2]]
@test ∇b ≈ g[ps[3]]
@test ∇state0 ≈ g[ps[4]]
end

@testset "gradients-explicit" begin


cell = Flux.RNNCell(1, 1, identity)
layer = Flux.Recur(cell)
layer.cell.Wi .= 5.0
layer.cell.Wh .= 4.0
layer.cell.b .= 0.0f0
layer.cell.state0 .= 7.0
x = [[2.0f0], [3.0f0]]

# theoretical primal gradients
primal =
layer.cell.Wh .* (layer.cell.Wh * layer.cell.state0 .+ x[1] .* layer.cell.Wi) .+
x[2] .* layer.cell.Wi
∇Wi = x[1] .* layer.cell.Wh .+ x[2]
∇Wh = 2 .* layer.cell.Wh .* layer.cell.state0 .+ x[1] .* layer.cell.Wi
∇b = layer.cell.Wh .+ 1
∇state0 = layer.cell.Wh .^ 2


x_block = reshape(vcat(x...), 1, 1, length(x))
nm_layer = Fluxperimental.NewRecur(cell; return_sequence = false)
e, g = Flux.withgradient(nm_layer) do layer
out = layer(x_block)
sum(out)
end
grads = g[1][:cell]

@test primal[1] ≈ e
@test ∇Wi ≈ grads[:Wi]
@test ∇Wh ≈ grads[:Wh]
@test ∇b ≈ grads[:b]
@test ∇state0 ≈ grads[:state0]

end
end

2 changes: 2 additions & 0 deletions test/runtests.jl
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ using Flux, Fluxperimental

include("compact.jl")

include("new_recur.jl")

end