ValSplit broken on Julia 1.10?

[svilupp]

This is a really nifty utility, however, I keep getting errors on Julia 1.10.
It breaks Transformers.jl on Julia 1.10 as well (see link)

Is there a way to fix it? I've tried to narrow it down, but it's over my head!

Running the example in README:

using ValSplit

soundof(animal::Val{:dog}) = "woof"
soundof(animal::Val{:cat}) = "nyan"

@valsplit function soundof(Val(animal::Symbol))
    error("Sound not defined for animal: \$animal")
end

soundof(:dog) # Error!

Stacktrace:

ERROR: MethodError: no method matching iterate(::Nothing)

Closest candidates are:
  iterate(::Base.MethodSpecializations)
   @ Base reflection.jl:1146
  iterate(::Base.MethodSpecializations, ::Nothing)
   @ Base reflection.jl:1152
  iterate(::Base.MethodSpecializations, ::Int64)
   @ Base reflection.jl:1153
  ...

Stacktrace:
  [1] iterate(::Base.Iterators.Filter{Tricks.var"#7#9"{Nothing}, Nothing})
    @ Base.Iterators ./iterators.jl:516
  [2] iterate(::Base.Generator{Base.Iterators.Filter{Tricks.var"#7#9"{Nothing}, Nothing}, Tricks.var"#6#8"})
    @ Base ./generator.jl:44
  [3] _collect(::Type{Method}, itr::Base.Generator{Base.Iterators.Filter{Tricks.var"#7#9"{Nothing}, Nothing}, Tricks.var"#6#8"}, isz::Base.SizeUnknown)
    @ Base ./array.jl:698
  [4] collect(::Type{Method}, itr::Base.Generator{Base.Iterators.Filter{Tricks.var"#7#9"{Nothing}, Nothing}, Tricks.var"#6#8"})
    @ Base ./array.jl:692
  [5] _methods(f_type::Any, t_type::Any, mod::Nothing)
    @ Tricks ~/.julia/packages/Tricks/7oAyo/src/Tricks.jl:146
  [6] _methods(f_type::Any, t_type::Any, mod::Nothing)
    @ Tricks ~/.julia/packages/Tricks/7oAyo/src/Tricks.jl:143 [inlined]
  [7] #s3#3
    @ ValSplit ~/.julia/packages/ValSplit/MMCz3/src/ValSplit.jl:42 [inlined]
  [8] var"#s3#3"(T::Any, N::Any, P::Any, ::Any, f::Any, types::Any, idx::Any, ptype::Any)
    @ ValSplit ./none:0
  [9] (::Core.GeneratedFunctionStub)(::UInt64, ::LineNumberNode, ::Any, ::Vararg{Any})
    @ Core ./boot.jl:600
 [10] valarg_params(f::typeof(soundof), types::Type{Tuple{Val}}, idx::Val{1}, ptype::Type{Symbol})
    @ ValSplit ~/.julia/packages/ValSplit/MMCz3/src/ValSplit.jl:60 [inlined]
 [11] soundof(animal::Symbol)
    @ Main ~/.julia/packages/ValSplit/MMCz3/src/ValSplit.jl:141
 [12] top-level scope
    @ ~/Documents/xyz/transformers/valsplit.jl:12

@code_typed soundof(:dog) returns (different from the README)

CodeInfo(
1 ─ %1 = Main.soundof::typeof(soundof)
│   %2 = Main.Symbol::Type{Symbol}
│   %3 = invoke ValSplit.valarg_params(%1::typeof(soundof), Tuple{Val}::Type{Tuple{Val}}, $(QuoteNode(Val{1}()))::Val{1}, %2::Type{Symbol})::Any
│   %4 = %new(Main.:(var"#3#default_f#5"))::var"#3#default_f#5"
│   %5 = Core.apply_type(Base.Val, %3)::Type{Val{_A}} where _A
│   %6 = %new(%5)::Val
│   %7 = ValSplit._valswitch(%6, $(QuoteNode(Val{1}())), Main.soundof, %4, animal)::Any
└──      return %7
) => Any

Versioninfo:

Julia Version 1.10.0-beta1
Commit 6616549950e (2023-07-25 17:43 UTC)
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 on 6 virtual cores
Environment:
JULIA_EDITOR = code

[ztangent]

Thanks for catching this!!

I'm not sure what's going on yet, but this might be related to changes in Tricks.jl in Julia 1.10 - see this issue: oxinabox/Tricks.jl#38

@oxinabox might have a bit more insight.

[oxinabox]

I suspect it is related. Tricks.jl doesn't really work right on 1.10, I need to find time to look at it properly.

1.10 got a lot more fussy about doing things in generated functions that involved looking at method tables.
And started disallowing a lot (all?) of them.

[ztangent]

So it looks like what ValSplit.jl is relying on is calling Tricks._methods in an @generated function, and Tricks._methods looks like this currently:

function _methods(@nospecialize(f_type), @nospecialize(t_type),
                 mod::Union{Tuple{Module},AbstractArray{Module},Nothing}=nothing)
    tt = _combine_signature_type(f_type, t_type)
    lim, world = -1, typemax(UInt)
    mft = Core.Compiler._methods_by_ftype(tt, lim, world)
    ms = Base.Method[_get_method(m) for m in mft if (mod === nothing || m.method.module ∈ mod)]
    return Base.MethodList(ms, f_type.name.mt)
end

https://github.com/oxinabox/Tricks.jl/blob/019aeb9c9d471c09ddd753e3e89621a69aba9c69/src/Tricks.jl#L140C1-L148

Tricks._methods is in turn doing collect on a Base.Generator (which is the list comprehension on the 4th line, I believe), and it looks like somehow that generator is returning nothing. It seems to me like this is because mft is returned as nothing by Core.Compiler._methods_by_ftype(tt, lim, world), which is causing it to be non-iterable.

[oxinabox]

that sounds correct.
and _methods_by_ftype is harder to use from generated functions in 1.10.
But we did get it to work in Diffractor i think

[ztangent]

I have a bit more time to look into this now -- @oxinabox do you happen to have an example of using _methods_by_ftype in Diffractor? I want to make sure this gets fixed before 1.10 comes out, since a number of my packages (and others) depend on ValSplit.

[oxinabox]

I think it is this
https://github.com/JuliaDiff/Diffractor.jl/blob/4df70dadf8c0ce8e03bd416556277a08e15b8e7d/src/debugutils.jl#L5-L8

I think it might also relate to this PR as to the cause
JuliaLang/julia#48766

[singularitti]

I work on 1.10.0-beta3 and also have the same issue.

[ztangent]

Fix for this has been released! :) Thanks to @chengchingwen for the fix.

https://github.com/ztangent/ValSplit.jl/releases/tag/v0.1.1