add variable interpolation #262

marius311 · 2019-04-01T05:43:19Z

From the docs included in this PR:

Python variables can be passed to Julia using either $foo or {foo}.

In [3]: foo = [[1,2],[3,4]]

In [4]: %julia $foo .+ 1
Out[4]: 
array([[2, 3],
       [4, 5]], dtype=int64)

The {} style supports passing arbitrary Python expressions inside.

In [5]: %julia {[x for x in range(3)]}
Out[5]: array([0, 1, 2], dtype=int64)

Variables are automatically converted between equivalent Python/Julia types (should they exist) using PyCall. Note below that 1, x, and the tuple itself are converted to Julia Int64, String, and Tuple, respectively, although the function abs and the result of typeof are not.

In [6]: %julia typeof({(1,"x",abs)})
Out[6]: <PyCall.jlwrap Tuple{Int64,String,PyObject}>

Closes #205

Happy to take any feedback or requests for changes. I didn't write any tests since I couldn't figure out how to make pytest work with python-jl (which my system needs) so maybe someone could help with creating tests, perhaps in a separate PR? (there we'll need to figure out how testing interacts with the fact that interpolation is using the IPython shell global namespace)

stevengj · 2019-04-01T12:12:39Z

Why {} instead of $(...)?

marius311 · 2019-04-01T16:38:28Z

Why {} instead of $(...)?

It would require matching balanced parenthesis with regex (e.g. $( 1 + (2+3) )) which afaik is not possible. I guess I could do something that works to finite nesting, or go beyond regex, but it seemed too complex, whereas $foo and {foo expression} are already implemented in IPython in DollarFormatter hence why the code in this PR was quite simple. Up to you guys what you think should be done, I do agree $(...) would be nice for consistency with Julia, but it might take me a while.

I'll also work on fixing the test errors.

marius311 · 2019-04-01T16:43:44Z

Actually I realize now {} is not going to work either since its valid Julia code, so definitely something else needs to be done...

tkf · 2019-04-01T21:32:43Z

I realize now {} is not going to work either since its valid Julia code

I thought this PR was awesome until I saw this comment. Too bad...

See also #205 (comment)

how to make pytest work with python-jl

You can install pytest with python-jl -m pip install pytest and run it with, e.g., python-jl -m pytest PATH/TO/TEST.py. Also, after #261 you should be able to do tox -- --julia-compiled-modules=no to run tests with statically linked Python.

marius311 · 2019-04-01T22:10:36Z

From @tkf's comment in the other thread.

In [3]: %%julia
   ...: using PyCall
   ...: py"x + 1"

Ha, hadn't thought of that. Thinking about it more, in some sense that's the best syntactically-sound way to do this.

The problem with either $foo or {foo} from my PR is both are sometimes valid Julia code. Consider e.g.

%julia Vector{Int}

where {Int} not meant to be interpolated from Python, or even worse,

x = 1

%%julia
y = $x
@eval $y

where $x is meant to be interpolated but $y is not. If you don't want to implement more complex expression parsing than regex, the way to catch these cases is use some characters other than $ or {} that aren't valid Julia code.

You could also in theory syntactically parse the expression, but actually that to me seems like exactly what using py".." does, its just that Julia itself is doing the parsing for you. So maybe that is the best solution?

The limitation is that py"..." can only grab stuff in the global scope. Its also slightly wordy / unintuitive, so what about the following proposal for modifying this PR:

I get rid of {} interpolation entirely.
I add a %julia --raw option which turns off interpolation, so you can do %julia y=$x then %julia --raw @eval $y.
We document that for expression substitutions, you can do py"...".

I gather this will work the way people expect it to, be flexible enough to let people do what they want by splitting expressions between --raw and non---raw cells, and also leaves us open to adding $(..) interpolation later modulo finding a clean way to match the balanced parentheses. Thoughts?

tkf · 2019-04-01T22:35:56Z

The limitation is that py"..." can only grab stuff in the global scope.

Ah, yes, that's a good point.

Another idea: How about using $"python code"? It seems that Julia can parse it as-is:

julia> Meta.parse(raw"""
       x = $"x if x is None else 0"
       """)
:(x = $(Expr(:$, "x if x is None else 0")))

So, we don't have to do any PyJulia-specific parsing and we can just look at the AST.

Julia parser can also do $$ which would be useful for code-interpolation (as inpy"..."):

julia> Meta.parse(raw"""
       x = $$"x if x is None else 0"
       """)
:(x = $(Expr(:$, :($(Expr(:$, "x if x is None else 0"))))))

tkf · 2019-04-01T22:39:35Z

The fact that it is parsable means that someone could be using it in their macors. So, --raw in your suggestion still makes sense with my $"python code" suggestion.

marius311 · 2019-04-03T09:59:10Z

I wanted to get feedback on the version I just force-pushed to this PR before finishing it up. What do you think of this approach?

In this version, I got rid of {}, but otherwise the docs now essentially mirror PyCall:

Any $var or $(var) expressions that appear in the Julia code (except in comments or string literals) are evaluated in Python and passed to Julia. Use $$var to insert a literal $var in Julia code.

Some examples:

In [1]: %load_ext julia.magic                                                                                                       
Initializing Julia interpreter. This may take some time...

In [2]: foo = [[1,2],[3,4]]                                                                                                         

In [3]: %julia $foo .+ 1                                                                                                            
Out[3]: 
array([[2, 3],
       [4, 5]], dtype=int64)

In [4]: %julia $(len(foo))
Out[4]: 2

In [5]: %julia bar=1; "$bar"   # not interpolated from Python since its in a string
Out[5]: '1'

In [6]: %julia bar=3; :($$bar)    # not interpolated from Python either
Out[6]: 3

The main limitation is that since I'm using Julia to parse everything, including the interpolated Python code, some small number of valid Python expressions are non-valid Julia expressions and hence don't work, e.g. %julia $(2**3). If we really wanted this I could probably do something based on pyparsing but it's going to get way more complex, so I tend to like the simplicity of the current version. Currently also Python code which raises exceptions or calls a generator seems to crash, I can look into that.

In any case, let me know if you have any major problems with this approach.

marius311 · 2019-04-03T10:06:49Z

Also, since it may not totally be clear what's going on, its all in this macro which parses the Julia/Python code and rewrites things so we can just call it from Python with a globals/locals dicts, e.g.:

julia> @macroexpand1 _PyJuliaHelper.@prepare_for_pyjulia_call begin
           $(x+1)
       end
quote
    #= REPL[1]:74 =#
    (PyCall.pyfunction)(((__globals, __locals)->begin
                #= REPL[1]:74 =#
                begin
                    #= REPL[2]:2 =#
                    (convert)(PyCall.PyAny, (PyCall.pyeval_)("x + 1", __globals, __locals))
                end
            end), PyCall.PyObject, PyCall.PyObject)
end

tkf · 2019-04-03T10:27:33Z

I like the idea of using Julia's macro system.

The main limitation is that since I'm using Julia to parse everything, including the interpolated Python code, some small number of valid Python expressions are non-valid Julia expressions and hence don't work, e.g. %julia $(2**3).

Unfortunately, Julia also normalizes some valid Python code to invalid ones, for example:

julia> :([x for x in xs])
:([x for x = xs])

This is why I suggested to use $"...." as we can put anything in string. (We can also re-use py"...." syntax instead.)

marius311 · 2019-04-03T21:59:25Z

Unfortunately, Julia also normalizes some valid Python code to invalid ones

Ah, hadn't realized that, good catch. Maybe its worthwhile enough to correct a few spot-cases like that, and then directing people to use the $".." or py"..." syntax when needed otherwise. It pretty easily fits in the approach I took here, I can definitely add it (I think of the two btw I prefer the py version since its already used elsewhere).

tkf · 2019-04-03T23:33:53Z

I don't think it's a good idea to have multiple syntaxes to do one thing. If $(...) cannot be used for all purpose, we should not have it.

Regarding $"..." vs py"...", I have a mixed feeling. py"..." sounds good because we already have it in PyCall. However, the semantics of it is a bit different. In PyCall, Python code is always evaluated at global scope while in PyJulia it may be in the local scope if it is used like

def f(x):
    y = %julia py"x + 1" + 1
    return y

But if we treat it as an extension of PyCall's semantics, it probably is OK?

marius311 · 2019-04-04T05:58:55Z

I don't think it's a good idea to have multiple syntaxes to do one thing. If $(...) cannot be used for all purpose, we should not have it.

Yea, I can understand this, but $(...) just seems like such the natural way to do it that I wonder whether its not worth keeping anyway. As an exercise, I tried to explain the three options in the documentation, and it didn't seem to take many words to do it adequately, which I'd say is an indication it could be kept.

In PyCall, Python code is always evaluated at global scope while in PyJulia it may be in the local scope if it is used like

This is a good point to bring up. Actually, in the current version of this PR its totally consistent with PyCall and is always the global scope, i.e.

In [10]: x = "global"
    ...: def f():
    ...:     x = "local"
    ...:     ret = %julia py"x"
    ...:     return ret
    ...: f()
Out[10]: 'global'

I'd say lets keep it this way, and maybe at a later time we could introduce a py_str option to use the local scope, i.e. py"x"l which would return "local" in the above example. Presumably this could be simultaneously introduced in PyCall as well.

marius311 · 2019-04-04T06:03:05Z

Next up is tests and figuring out why Python exception crash the whole thing. Hopefully I didn't go down some doomed path with this pyeval thing...

marius311 · 2019-04-04T07:11:01Z

I think I'll likely need some help here @stevengj @tkf, these are internals that are beyond me. Basically any python code which gives an error crashes the interpreter (MWE below). I get different stack traces python-jl vs. natively.

Hopefully it wasn't a bad idea to have Julia call Python back from inside of %julia...

Dynamically linked Python:

In [1]: %load_ext julia.magic                                                                                                       
Initializing Julia interpreter. This may take some time...

In [2]: %julia py"x"                                                                                                                
fatal: error thrown and no exception handler available.
#<null>
rec_backtrace at /home/marius/src/julia/src/stackwalk.c:94
record_backtrace at /home/marius/src/julia/src/task.c:217 [inlined]
jl_throw at /home/marius/src/julia/src/task.c:417
jl_type_error_rt at /home/marius/src/julia/src/rtutils.c:118
jl_type_error at /home/marius/src/julia/src/rtutils.c:125
jl_f_setfield at /home/marius/src/julia/src/builtins.c:693
pyraise at /home/marius/.julia/packages/PyCall/h14lX/src/exception.jl:183 [inlined]
_pyjlwrap_call at /home/marius/.julia/packages/PyCall/h14lX/src/callback.jl:42
unknown function (ip: 0x7f887c055c93)
jl_fptr_trampoline at /home/marius/src/julia/src/gf.c:1864
jl_apply_generic at /home/marius/src/julia/src/gf.c:2219
pyjlwrap_call at /home/marius/.julia/packages/PyCall/h14lX/src/callback.jl:49
unknown function (ip: 0x7f887c02a56e)
_PyObject_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2331
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4854
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
PyEval_EvalCodeEx at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4180
function_call at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/funcobject.c:604
PyObject_Call at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2261
do_call_core at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:5099 [inlined]
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3397
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
_PyFunction_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:5063
_PyObject_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2310
_PyObject_Call_Prepend at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2373
PyObject_Call at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2261
do_call_core at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:5099 [inlined]
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3397
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
PyEval_EvalCodeEx at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4180
PyEval_EvalCode at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:731
builtin_exec_impl at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/bltinmodule.c:983 [inlined]
builtin_exec at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/clinic/bltinmodule.c.h:283
_PyCFunction_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/methodobject.c:234
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4830
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
gen_send_ex at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:189 [inlined]
_PyGen_Send at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:308
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:2081
gen_send_ex at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:189 [inlined]
_PyGen_Send at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:308
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:2081
gen_send_ex at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:189 [inlined]
_PyGen_Send at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/genobject.c:308
_PyCFunction_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/methodobject.c:209
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4830
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyFunction_FastCall at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4912 [inlined]
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4947 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyFunction_FastCall at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4912 [inlined]
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4947 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3344
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyFunction_FastCall at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4912 [inlined]
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4947 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
_PyFunction_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:5063
_PyObject_FastCallDict at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2310
_PyObject_Call_Prepend at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2373
PyObject_Call at /tmp/python-build.20190403234951.21870/Python-3.6.6/Objects/abstract.c:2261
do_call_core at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:5099 [inlined]
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3397
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
fast_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4971 [inlined]
call_function at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4851
_PyEval_EvalFrameDefault at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4159
PyEval_EvalCodeEx at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:4180
PyEval_EvalCode at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/ceval.c:731
run_mod at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/pythonrun.c:1025 [inlined]
PyRun_FileExFlags at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/pythonrun.c:978
PyRun_SimpleFileExFlags at /tmp/python-build.20190403234951.21870/Python-3.6.6/Python/pythonrun.c:420
run_file at /tmp/python-build.20190403234951.21870/Python-3.6.6/Modules/main.c:340 [inlined]
Py_Main at /tmp/python-build.20190403234951.21870/Python-3.6.6/Modules/main.c:810
main at python3 (unknown line)
__libc_start_main at /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
_start at python3 (unknown line)

and python-jl:

In [2]: %julia py"x"                                                                                                                
ERROR: TypeError: in setfield!, expected Ptr{PyCall.PyObject_struct}, got Ptr{Nothing}
Stacktrace:
 [1] pyraise at /home/marius/.julia/packages/PyCall/h14lX/src/exception.jl:183 [inlined]
 [2] _pyjlwrap_call(::PyCall.FuncWrapper{Tuple{PyObject,PyObject},getfield(Main, Symbol("##3#4"))}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/marius/.julia/packages/PyCall/h14lX/src/callback.jl:42
 [3] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/marius/.julia/packages/PyCall/h14lX/src/callback.jl:49
 [4] macro expansion at /home/marius/.julia/packages/PyCall/h14lX/src/exception.jl:81 [inlined]
 [5] __pycall!(::PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyObject, ::Ptr{Nothing}) at /home/marius/.julia/packages/PyCall/h14lX/src/pyfncall.jl:44
 [6] _pycall!(::PyObject, ::PyObject, ::Tuple{Array{String,1}}, ::Int64, ::Ptr{Nothing}) at /home/marius/.julia/packages/PyCall/h14lX/src/pyfncall.jl:29
 [7] #call#111 at /home/marius/.julia/packages/PyCall/h14lX/src/pyfncall.jl:11 [inlined]
 [8] (::PyObject)(::Array{String,1}) at /home/marius/.julia/packages/PyCall/h14lX/src/pyfncall.jl:89
 [9] top-level scope at none:0
Exception ignored in: <module 'threading' from '/home/marius/.pyenv/versions/3.6.6/lib/python3.6/threading.py'>
Traceback (most recent call last):
  File "/home/marius/.pyenv/versions/3.6.6/lib/python3.6/threading.py", line 1289, in _shutdown
    assert tlock.locked()
SystemError: <built-in method locked of _thread.lock object at 0x7f13a4ded170> returned a result with an error set

tkf · 2019-04-04T07:15:59Z

I'm strongly against to have multiple options because

There should be one-- and preferably only one --obvious way to do it.

Using Julia's string(::Expr) is a good experiment but it seems to be too much of a hack to be a part of a package. I prefer to not release half-baked features especially if there is no one working on it to complete it (e.g., writing a special parser to do this correctly).

totally consistent with PyCall and is always the global scope, i.e.

In [10]: x = "global"
    ...: def f():
    ...:     x = "local"
    ...:     ret = %julia py"x"
    ...:     return ret
    ...: f()
Out[10]: 'global'

Thinking about this more, I think this is not a useful behavior and is potentially a big pitfall for Python users. I started thinking py"..." is a good idea because we can fix this pitfall and then introduce a useful variable interpolation at the same time. You can do PyCall.py"..." if you need PyCall behavior.

tkf · 2019-04-04T07:23:48Z

Re: the error with %julia py"x"; I can't reproduce it. Can you open a new issue with all the relevant version info (julia, python, pyjulia, PyCall)?

marius311 · 2019-04-04T07:41:01Z

I'm strongly against to have multiple options because

That's fine, I can remove it.

Thinking about this more, I think this is not a useful behavior and is potentially a big pitfall for Python users. I started thinking py"..." is a good idea because we can fix this pitfall and then introduce a useful variable interpolation at the same time. You can do PyCall.py"..." if you need PyCall behavior.

I didn't understand this, sorry.

Re: the error with %julia py"x"; I can't reproduce it. Can you open a new issue with all the relevant version info (julia, python, pyjulia, PyCall)?

This is with this PR, so it seems here is the right place to discuss it? Otherwise I have julia 1.1.0, python 3.6.6, and PyCall v1.91.1.

tkf · 2019-04-04T07:47:16Z

Which is the part you don't understand? I'm just saying that we can "hide" the current misbehavior of py"..." (not aware of Python's local scope) by implementing the variable interpolation as you did for $(...).

This is with this PR

Ah, sorry. I thought it happened without this PR.

tkf · 2019-04-04T07:51:26Z

I can reproduce it.

tkf · 2019-04-04T08:10:47Z

JuliaPy/PyCall.jl#675 fixed it for me

src/julia/magic.py

marius311 · 2019-04-04T22:14:25Z

Which is the part you don't understand? I'm just saying that we can "hide" the current misbehavior of py"..." (not aware of Python's local scope) by implementing the variable interpolation as you did for $(...).

Ah sorry I had thought PyCall also "misbehaved" like this but I realize now it works correctly. I fixed it here as well, see the updated doc which now returns "local".

Trying to fix test failures now...

coveralls · 2019-04-04T23:48:26Z

Pull Request Test Coverage Report for Build 658

4 of 7 (57.14%) changed or added relevant lines in 1 file are covered.
33 unchanged lines in 6 files lost coverage.
Overall coverage decreased (-3.09%) to 74.101%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/julia/magic.py	4	7	57.14%

Files with Coverage Reduction	New Missed Lines	%
src/julia/ipy/monkeypatch_completer.py	1	58.0%
src/julia/pytestplugin.py	1	65.79%
src/julia/pseudo_python_cli.py	2	79.78%
src/julia/find_libpython.py	2	71.88%
src/julia/magic.py	4	81.03%
src/julia/core.py	23	74.74%

Totals
Change from base Build 652:	-3.09%
Covered Lines:	907
Relevant Lines:	1224

💛 - Coveralls

coveralls · 2019-04-04T23:48:26Z

Pull Request Test Coverage Report for Build 677

10 of 10 (100.0%) changed or added relevant lines in 1 file are covered.
136 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.03%) to 77.216%

Files with Coverage Reduction	New Missed Lines	%
src/julia/pytestplugin.py	6	67.5%
src/julia/tools.py	20	65.15%
src/julia/with_rebuilt.py	24	30.95%
src/julia/core.py	86	79.11%

Totals
Change from base Build 652:	0.03%
Covered Lines:	993
Relevant Lines:	1286

💛 - Coveralls

tkf · 2019-04-05T00:09:48Z

You can use tox -- --julia-compiled-modules=no etc. to run tests locally (even if you don't have dynamically linked Python) now if you merge master to your branch. See:
https://pyjulia.readthedocs.io/en/latest/testing.html
https://pyjulia.readthedocs.io/en/latest/pytest.html#options

marius311 · 2019-04-05T05:24:30Z

I think its ready from my end.

test/test_magic.py

src/julia/pyjulia_helper.jl

docs/source/usage.md

marius311 · 2019-04-08T04:53:50Z

After thinking about it I think you're right about

a valid Julia code snippet has to be evaluated correctly in %%julia magic

This means interpolation from Python via $ should only happen outside of strings and quotes, since inside of them (and only inside), $foo already has a valid Julia meaning. Similarly, in pure Julia, :(py"...") would not call PyCall rather it would just create that expression object, so inside of %julia magic that shouldn't interpolate either. So the rule now is basically:

Interpolation from Python is only done outside of strings and quote blocks

This leads to things like

In [6]: foo="Python"
        %julia foo="Julia"
        %julia "this is $foo", "this is $($foo)"
Out[6]: ('this is Julia', 'this is Python')

which I think is entirely natural. I added the logic to track the "quote depth" which you need to do this to the PR, it was really simple. I also turned off interpolation inside of macros entirely, I think you're right that there you just have to err on the safe side. And I removed $$ entirely, that seems unnecessary now; it could potentially be given PyCall-like behavior later if we want to. Ready for any more comments you may have.

tkf · 2019-04-08T06:09:47Z

This PR looks great now. Thanks a lot for implementing it!

So the rule now is basically:

Interpolation from Python is only done outside of strings and quote blocks

This sounds good to me. Thanks for noticing the string case (which I forgot). The example looks good.

Before merging, a bit of nitpicking of the wording...

Doesn't the word "interpolation" suggest that the following code prints the same number five times?

for _ in 1:5
    println(py"numpy.random.rand()")
end

Also, if we interpret "interpolation" as "pre-evaluation object injection" (as in @benchmark and PyCall.py"..."), the semantics $x of is a bit different from usual :($x):

In [2]: x = 1

In [3]: %%julia
        println($x)
        py"""
        x = 2
        """
        println($x)

which print 1 and 2 in this PR but with "pre-evaluation object injection" semantics it would be 1 twice.

This is fine by me (although I'm somewhat nervous a little bit) as we can do whatever we want do to define the semantics of depth-1 $ and it's more powerful than "pre-evaluation object injection."

Maybe we can just say

You can "interpolate" Python code into Julia code

instead of Python objects? Or is there a better phrase for it?

marius311 · 2019-04-08T07:34:06Z

Fair point, how about:

You can call Python code from inside of %julia blocks via $var for accessing single variables or py"..." for more complex expressions:

marius311 · 2019-04-08T07:35:33Z

FWIW I find both Julia and PyCall behavior totally weird compared to what we have here. In Julia it looks like its always the final value of the interpolated thing (if it was going to be a fixed thing wouldn't the initial value make more sense?), e.g:

julia> x=1
1

julia> "$x $(x=2) $(x=3) $x $(x=4)"
"4 2 3 4 4"

julia> x
4

and PyCall is maybe trying to be that but I think there's got to be a bug?

julia> x=1
1

julia> py"""
       print($x)
       $(x=2)
       print($x)
       """
2
2

julia> x
2

julia> x=1
1

julia> py"""
       print($x)
       $(x=2)
       print($x)
       """
1
2

tkf · 2019-04-08T08:28:10Z

Fair point, how about:

You can call Python code from inside of %julia blocks via $var for accessing single variables or py"..." for more complex expressions:

Ah, this sounds more explicit and easy to understand. It'd be great if you can tweak the docs to avoid using the word "interpolation" like this.

BTW the examples you brought up is interesting. I didn't know that's what Julia do. FYI this is what Julia evaluates:

julia> Meta.@lower "$x $(x=2) $(x=3) $x $(x=4)"
:($(Expr(:thunk, CodeInfo(
    @ none within `top-level scope'
1 ─      x = 2
│        x = 3
│        x = 4
│   %4 = Base.string(x, " ", 2, " ", 3, " ", x, " ", 4)
└──      return %4
))))

But I have no idea why this happens. Maybe report a bug to Julia?

OTOH, PyCall seems to not track the order (maybe because Dict forgets the order?)

julia> Meta.@lower py"""
       print($x)
       $(x=2)
       print($x)
       """
:($(Expr(:thunk, CodeInfo(
    @ /home/takafumi/.julia/packages/PyCall/h14lX/src/pyeval.jl:230 within `top-level scope'
1 ─ %1  = PyCall.pynamespace(Main)
│         #27#m = %1
│   @ /home/takafumi/.julia/packages/PyCall/h14lX/src/pyeval.jl:231 within `top-level scope'
│         x = 2
│   %4  = PyCall.PyObject(2)
│         Base.setindex!(#27#m, %4, "__julia_localvar_0_2")
│   %6  = PyCall.PyObject(x)
│         Base.setindex!(#27#m, %6, "__julia_localvar_0_3")
│   %8  = PyCall.PyObject(x)
│         Base.setindex!(#27#m, %8, "__julia_localvar_0_1")
│   @ /home/takafumi/.julia/packages/PyCall/h14lX/src/pyeval.jl:232 within `top-level scope'
│   %10 = Base.getproperty(Base, :string)
│   %11 = (%10)("print(__julia_localvar_0_1)\n__julia_localvar_0_2\nprint(__julia_localvar_0_3)\n")
│   %12 = PyCall.pyeval_(%11, #27#m, #27#m, 257, "/home/takafumi/.julia/packages/PyCall/h14lX/src/pyeval.jl")
│   %13 = (PyAny)(%12)
│         #28#ret = %13
│   @ /home/takafumi/.julia/packages/PyCall/h14lX/src/pyeval.jl:234 within `top-level scope'
└──       return #28#ret
))))

stevengj · 2019-04-08T12:28:47Z

In Julia it looks like its always the final value of the interpolated thing (if it was going to be a fixed thing wouldn't the initial value make more sense?), ... and PyCall is maybe trying to be that but I think there's got to be a bug?

I wasn't really thinking about this case when I implemented pyeval … py"..." could probably be modified to call PyObject(arg) on each argument before evaluating the next argument, or whatever semantics we want.

tkf · 2019-04-08T20:38:39Z

@marius311 Thanks for your awesome work!

tkf · 2019-04-08T20:45:32Z

@stevengj Would it be fixed by just using locals = Pair{Union{String,Int},Any}[] instead of locals = Dict{Union{String,Int},Any}() in interpolate_pycode?

marius311 · 2019-04-08T20:48:05Z

Thanks for all the comments, this was way better because of them!

marius311 · 2019-04-10T07:07:41Z

Super minor but any idea what happened to the syntax highlighting on cells 6 & 7 in https://pyjulia.readthedocs.io/en/latest/usage.html#ipython-magic ? It looks right when I build the docs locally. Note those are marked language Julia bc the $ screwed up the highlighting if it was marked python (at least, that's locally what happened)

tkf · 2019-04-10T17:16:20Z

Hmm... Good point. I have no idea. Maybe the package versions are different? Though looking at some of the packages listed in https://readthedocs.org/projects/pyjulia/builds/8884519/ they seem to use fairly new ones. We can in principle freeze the version of all the packages by putting them in docs/requirements.txt. But I guess it's better to switch to IPython Sphinx Directive https://ipython.readthedocs.io/en/stable/sphinxext.html than fixing this. We can use it as doctest too.

tkf mentioned this pull request Apr 1, 2019

Add new magic option that loads python variable into julia #205

Closed

tkf mentioned this pull request Apr 3, 2019

Improve exception handling in %julia magic #266

Open

add variable interpolation

8407cc4

marius311 force-pushed the interpolation branch from 170db0c to 8407cc4 Compare April 3, 2019 09:57

add py"..." interpolation and more docs

ef1947b

marius311 changed the title ~~add variable interpolation~~ WIP: add variable interpolation Apr 4, 2019

make %julia scope be Main

ac92d82

tkf mentioned this pull request Apr 4, 2019

Fix pyraise JuliaPy/PyCall.jl#675

Merged

tkf reviewed Apr 4, 2019

View reviewed changes

src/julia/magic.py Outdated Show resolved Hide resolved

don't require MacroTools in default package environment

9780c69

fix tests

d9f9b0a

marius311 force-pushed the interpolation branch from 210b88e to d9f9b0a Compare April 5, 2019 01:40

fix 32bit test failure

637c08a

marius311 changed the title ~~WIP: add variable interpolation~~ add variable interpolation Apr 5, 2019

tkf requested changes Apr 5, 2019

View reviewed changes

marius311 added 2 commits April 5, 2019 16:33

address review comments

6d9e4a2

improve interpolation rules

94dca79

tkf approved these changes Apr 8, 2019

View reviewed changes

tweak docs

ba186f4

tkf merged commit f129fa4 into JuliaPy:master Apr 8, 2019

tkf mentioned this pull request Apr 8, 2019

Do not swallow exceptions in %julia magic #265

Closed

marius311 mentioned this pull request May 12, 2020

allow nested variable interpolation #381

Draft

MilesCranmer mentioned this pull request Nov 2, 2022

Fix CI test suite and and remove Python 2.7 support #508

Merged

add variable interpolation #262

add variable interpolation #262

Uh oh!

Conversation

marius311 commented Apr 1, 2019

Uh oh!

stevengj commented Apr 1, 2019

Uh oh!

marius311 commented Apr 1, 2019

Uh oh!

marius311 commented Apr 1, 2019

Uh oh!

tkf commented Apr 1, 2019

Uh oh!

marius311 commented Apr 1, 2019

Uh oh!

tkf commented Apr 1, 2019

Uh oh!

tkf commented Apr 1, 2019

Uh oh!

marius311 commented Apr 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marius311 commented Apr 3, 2019

Uh oh!

tkf commented Apr 3, 2019

Uh oh!

marius311 commented Apr 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkf commented Apr 3, 2019

Uh oh!

marius311 commented Apr 4, 2019

Uh oh!

marius311 commented Apr 4, 2019

Uh oh!

marius311 commented Apr 4, 2019

Uh oh!

tkf commented Apr 4, 2019

Uh oh!

tkf commented Apr 4, 2019

Uh oh!

marius311 commented Apr 4, 2019

Uh oh!

tkf commented Apr 4, 2019

Uh oh!

tkf commented Apr 4, 2019

Uh oh!

tkf commented Apr 4, 2019

Uh oh!

Uh oh!

marius311 commented Apr 4, 2019

Uh oh!

coveralls commented Apr 4, 2019

Pull Request Test Coverage Report for Build 658

💛 - Coveralls

Uh oh!

coveralls commented Apr 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 677

💛 - Coveralls

Uh oh!

tkf commented Apr 5, 2019

Uh oh!

marius311 commented Apr 5, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marius311 commented Apr 8, 2019

Uh oh!

tkf commented Apr 8, 2019

Uh oh!

marius311 commented Apr 8, 2019

Uh oh!

marius311 commented Apr 8, 2019

Uh oh!

marius311 commented Apr 3, 2019 •

edited

Loading

marius311 commented Apr 3, 2019 •

edited

Loading

coveralls commented Apr 4, 2019 •

edited

Loading

tkf commented Apr 8, 2019 •

edited

Loading

stevengj commented Apr 8, 2019 •

edited

Loading