Doesn't work on 1.7.0-beta2 or 1.8

[PallHaraldsson]

This works:

(@v1.8) pkg> add PythonCall#master  # with or without master, while I get an error for 1.7, likely bad installation of Julia.

[Do you have any idea what might be happening? And since works in 1.6, an issue for Julia not your package?]

But doesn't work in 1.8 nor beta2 (and likely not in beta3 either):

julia> @time using PythonCall
ERROR: InitError: type DataType has no field mutable
Stacktrace:
  [1] getproperty
    @ ./Base.jl:37 [inlined]
  [2] pointer_from_obj(o::Vector{String})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/utils.jl:236
  [3] PyJuliaValue_SetValue(__o::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:70
  [4] PyJuliaValue_New(t::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:89
  [5] PyJuliaVectorValue_New
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:244 [inlined]
  [6] PyJuliaValue_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:245 [inlined]
  [7] PyObject_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/object.jl:82 [inlined]
  [8] exec_impl(f::PythonCall.var"#245#263", code::PythonCall.PyCode, globals::PythonCall.PyDict{String, PythonCall.PyObject}, locals::Nothing, extrakeys::Tuple{PythonCall.PyInternedString, PythonCall.PyInternedString}, extravals::Tuple{Vector{String}, Bool})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:38
  [9] macro expansion
    @ ~/.julia/packages/PythonCall/r2eLx/src

┌ Error: Pkg.precompile error
│   exception =
│    SystemError: mktemp: Permission denied
│    Stacktrace:
│      [1] systemerror(p::Symbol, errno::Int32; extrainfo::Nothing)
│        @ Base ./error.jl:174
│      [2] #systemerror#68
│        @ ./error.jl:173 [inlined]
│      [3] systemerror
│        @ ./error.jl:173 [inlined]
│      [4] mktemp(parent::String; cleanup::Bool)


[ Info: Precompiling PythonCall [6099a3de-0909-46bc-b1f4-468b9a2dfc0d]
ERROR: InitError: type DataType has no field mutable
Stacktrace:
  [1] getproperty
    @ ./Base.jl:37 [inlined]
  [2] pointer_from_obj(o::Vector{String})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/utils.jl:236
  [3] PyJuliaValue_SetValue(__o::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:70
  [4] PyJuliaValue_New(t::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:89
  [5] PyJuliaVectorValue_New
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:244 [inlined]
  [6] PyJuliaValue_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:245 [inlined]
  [7] PyObject_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/object.jl:82 [inlined]
  [8] exec_impl(f::PythonCall.var"#245#263", code::PythonCall.PyCode, globals::PythonCall.PyDict{String, PythonCall.PyObject}, locals::Nothing, extrakeys::Tuple{PythonCall.PyInternedString, PythonCall.PyInternedString}, extravals::Tuple{Vector{String}, Bool})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:38
  [9] macro expansion
    @ ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:156 [inlined]
 [10] (::PythonCall.var"#243#261")()
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/init.jl:203
 [11] with_gil(f::PythonCall.var"#243#261", c::Bool)
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/gil.jl:10
 [12] with_gil(f::Function)
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/gil.jl:9
 [13] init()
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/init.jl:195
 [14] (::PythonCall.var"#275#276")()
    @ PythonCall ~/.julia/packages/Requires/035xH/src/init.jl:11
 [15] __init__()
    @ PythonCall ~/.julia/packages/Requires/035xH/src/init.jl:18
 [16] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:768
 [17] _require_from_serialized(path::String)
    @ Base ./loading.jl:821
 [18] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1130
 [19] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1013
 [20] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:997
 [21] top-level scope
    @ timing.jl:210
during initialization of module PythonCall

[cjdoris]

Hi thanks for the report.

I'm actually rewriting this package from scratch, so won't be fixing bugs in the current release. The new version is on branch 'rewrite'. Not documented yet but very similar API to old version.

I believe this particular bug is not present in the new version. Please try it out and let me know.

[PallHaraldsson]

Yes, works in 1.8, and beta2. I didn't test further, for now:

julia> @time using PythonCall # 1.6
  3.005584 seconds (2.36 M allocations: 139.871 MiB, 1.84% gc time)

julia> @time using PythonCall # 1.7
  3.966541 seconds (2.46 M allocations: 153.414 MiB, 1.56% gc time, 0.17% compilation time)

julia> @time using PythonCall # 1.8
  3.786829 seconds (2.34 M allocations: 146.184 MiB, 0.99% gc time, 0.32% compilation time)

I'm not sure why 1.6 is fastest, possibly related to allocations, while seems not the whole story.


$ ~/julia-1.8-DEV-d1145d4569/bin/julia -O1
julia> @time using PythonCall
  2.681354 seconds (2.34 M allocations: 146.187 MiB, 1.46% gc time, 0.19% compilation time)

$ ~/julia-1.8-DEV-d1145d4569/bin/julia -O0 --compile=min
julia> @time using PythonCall
  0.692783 seconds (465.87 k allocations: 35.170 MiB, 1.85% gc time)

You can close the issue on my account, or keep it open a little longer so people may see it after 1.7 release in case you've not made the rewrite official.

The last timing is comparable to PyCall.jl, with or without NumPyArrays.jl from @mkitti.

[cjdoris]

Thanks, those are interesting timings. I'll look into making startup quicker once the package is more complete. It will probably always be slower than PyCall to start simply because this is a bigger package.

[mkitti]

After writing NumPyArrays.jl, it occurred to me that PyCall.jl might actually be too big. Why should a Py[thon]Call support NumPy arrays at all? NumPy is not the only n-dimensional array package for Python. xarray comes to mind.

My suggestion is to consider breaking up the package into smaller modular packages. Maybe PythonCall is the the package that depends on multiple smaller packages, with a core package called PythonCallCore or PythonCallBase.

An example of this is https://github.com/JuliaDiff/ChainRulesCore.jl and https://github.com/JuliaDiff/ChainRules.jl

[cjdoris]

In fact PythonCall does not directly support numpy. Instead it supports the buffer protocol and the array interface, which a broad class of array types support (bytes, ndarray, pandas, xarray, ...)