failure on 0.6 in anonymous callback

[stevengj]

In Julia 0.6 master, I can successfully convert an ordinary function like sum and call it from Python: PyObject(sum)(3) works (and returns 3).

However, the same code no longer works for anonymous functions: PyObject(x -> x+1)(3) fails. I inserted some println statements into the callback hook, and I verified that f = unsafe_pyjlwrap_to_objref(self_)::Function is successfully getting a reference to f = x -> x+1 (display(f) gives the expected output). However, for some reasons f(3) is then throwing a MethodError.

@vtjnash, @yuyichao, has something changed at a low-level with Function objects, e.g. the new dependency-tracking stuff, that would break this?

(I'm stashing a jl_value_t* pointer in the Python object and pulling it out in the callback to call; a reference to the function object is saved in a global dictionary to prevent it from being garbage-collected.)

[yuyichao]

This is due to cfunction fixing the world age.

[stevengj]

Is there a workaround?

[yuyichao]

Before there's a better way to do it, the only way to switch to a newer world is to use eval.

[stevengj]

Ouch. Well, I guess for Python callbacks into Julia, the overhead is already pretty high.

[yuyichao]

Note that the world age is captured at construction time (when cfunction is called) so another way is to construct a new cfunction everytime and rely on codegen optimization and builtin caching. This may not be applicable to PyCall.jl but is likely the right solution for Cubature.jl.

julia> function calling(f)
           f()
       end
calling (generic function with 1 method)

julia> cf1 = cfunction(calling, Any, (Any,))
Ptr{Void} @0x00007f0131a86170

julia> f = ()->1
(::#3) (generic function with 1 method)

julia> ccall(cf1, Any, (Any,), f)
ERROR: MethodError: no method matching (::##3#4)()
The applicable method may be too new: running in world age 20395, while current world is 20396.
Closest candidates are:
  #3() at REPL[3]:1 (method too new to be called from this world context.)
 in calling(::##3#4) at ./REPL[1]:2
 in anonymous at ./<missing>:?

julia> cf2 = cfunction(calling, Any, (Any,))
Ptr{Void} @0x00007f0131a9aa90

julia> cf3 = cfunction(calling, Any, (Any,))
Ptr{Void} @0x00007f0131a9aa90

julia> cf2 == cf3
true

julia> ccall(cf2, Any, (Any,), f)
1