python.exe not found

[gyansinha]

IOError: could not spawn 'H:\.julia\environments\v1.9\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT)

It looks like the python executable isn't there at all after "add PythonCall" - any ideas why this may be happening.

[cjdoris]

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

[cjdoris]

My guess though is that your antivirus deleted python.exe after it was installed.

[gyansinha]

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

add PythonCall
using PythonCall
add quantlib

] conda resolve

This is the error I get:

CondaPkg Creating environment
         │ H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe
         │ -r H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root
         │ create
         │ -y
         │ -p \\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env
         │ --override-channels
         │ --no-channel-priority
         │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
         │ quantlib[version='*']
         └ -c conda-forge

critical libmamba remove: Access is denied.: "\PROD-FILE\XXXX$\User\gsinha.julia\dev\Clover.CondaPkg\env\DLLs_ctypes.pyd.mamba_trash"
ERROR: failed process: Process('H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe' -r 'H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p '\\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env' --override-channels --no-channel-priority "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" "quantlib[version='*']" -c conda-forge, ProcessExited(1)) [1]

Stacktrace:
[1] pipeline_error
@ .\process.jl:565 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base .\process.jl:480
[3] run(::Cmd)
@ Base .\process.jl:477
[4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
@ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:398
[5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
@ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:299
[6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
@ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:547
[7] resolve(; force::Bool)
@ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:82
[8] resolve()
@ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:81
[9] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
@ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:409
[10] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
@ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:390
[11] do_cmd
@ C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:380 [inlined]
[12] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
@ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:557
[13] #invokelatest#2
@ .\essentials.jl:816 [inlined]
[14] invokelatest
@ .\essentials.jl:813 [inlined]
[15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\LineEdit.jl:2647
[16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\REPL.jl:1300
[17] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL .\task.jl:514

I am having the anti-virus looked into - thanks for your tip on that.

[gyansinha]

After this step, I do the followiing after exiting Julia entirely:

julia> using PythonCall
ERROR: InitError: Python executable "h:\.julia\dev\Clover\.CondaPkg\env\python.exe" is not executable.
Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] init_context()
@ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:88
[3] init()
@ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21
[4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base .\loading.jl:1115
[5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
@ Base .\loading.jl:1061
[6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
@ Base .\loading.jl:1506
[7] _require(pkg::Base.PkgId, env::String)
@ Base .\loading.jl:1783
[8] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base .\loading.jl:1660
[9] macro expansion
@ .\loading.jl:1648 [inlined]
[10] macro expansion
@ .\lock.jl:267 [inlined]
[11] require(into::Module, mod::Symbol)
@ Base .\loading.jl:1611
during initialization of module C

caused by: IOError: could not spawn 'h:\.julia\dev\Clover\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT)
Stacktrace:
[1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}})
@ Base .\process.jl:128
[2] #760
@ .\process.jl:139 [inlined]
[3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}})
@ Base .\process.jl:223
[4] _spawn
@ .\process.jl:138 [inlined]
[5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) (repeats 2 times)
@ Base .\process.jl:166
[6] run(::Base.CmdRedirect; wait::Bool)
@ Base .\process.jl:479
[7] run
@ .\process.jl:477 [inlined]
[8] init_context()
@ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:86
[9] init()
@ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21
[10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base .\loading.jl:1115
[11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
@ Base .\loading.jl:1061
[12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
@ Base .\loading.jl:1506
[13] _require(pkg::Base.PkgId, env::String)
@ Base .\loading.jl:1783
[14] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base .\loading.jl:1660
[15] macro expansion
@ .\loading.jl:1648 [inlined]
[16] macro expansion
@ .\lock.jl:267 [inlined]
[17] require(into::Module, mod::Symbol)
@ Base .\loading.jl:1611

[cjdoris]

Can you remove that .CondaPkg folder entirely and try again?

[schlichtanders]

I am running into a similar behaviour by first installing some packages with CondaPkg, before even installing PythonCall.

It seems to me that this order is not supported currently. A workaround is to first install PythonCall into the environment before the first time import CondaPkg is executed in that environment. Then the __init__ of CondaPkg will automatically pickup all dependencies from PythonCall.

---

Unfortunately that workaround does not work for me, because my package depends on RCall, which I like to build on top of CondaPkg. However unlike PythonCall, RCall requires a ready R installation before it can be installed, hence I need to import CondaPkg before installing my package which depends on R. Hence all the Conda dependencies from my package are neither picked up nor resolved by CondaPkg.

So for me the only workaround left is to

• delete .CondaPkg after RCall is installed, using rm(dirname(CondaPkg.envdir()), recursive=true)
• and run import CondaPkg; CondaPkg.resolve() again after I added my package.

[cjdoris]

@schlichtanders That order should definitely work.

Can you give the sequence of steps to reproduce your error? And also give the full error you see.

[schlichtanders]

It was similar to

julia -e 'import Pkg; \
    Pkg.Registry.add("General"); \
    Pkg.add("CondaPkg"); \
    import CondaPkg; \
    CondaPkg.add("r-base"); \
    ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
    Pkg.add("RCall"); \
    Pkg.add("PythonCall"); \
    Pkg.precompile(); \
'

[schlichtanders]

Here a complete reproducible example

1. Start a docker container with docker run -it --rm julia:1.9-bookworm bash
2. run the above installation instructions

julia -e 'import Pkg; \
    Pkg.Registry.add("General"); \
    Pkg.add("CondaPkg"); \
    import CondaPkg; \
    CondaPkg.add("r-base"); \
    ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
    Pkg.add("RCall"); \
    Pkg.add("PythonCall"); \
    Pkg.precompile(); \
'

3. run julia -e "import PythonCall"

then the output is

ERROR: InitError: Python executable "/root/.julia/environments/v1.9/.CondaPkg/env/bin/python" is not executable.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:88
  [3] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
  [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [7] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [8] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
  [9] macro expansion
    @ ./loading.jl:1648 [inlined]
 [10] macro expansion
    @ ./lock.jl:267 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
during initialization of module C

caused by: IOError: could not spawn `/root/.julia/environments/v1.9/.CondaPkg/env/bin/python --version`: no such file or directory (ENOENT)
Stacktrace:
  [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:128
  [2] #760
    @ ./process.jl:139 [inlined]
  [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:223
  [4] _spawn
    @ ./process.jl:138 [inlined]
  [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, IO}}) (repeats 2 times)
    @ Base ./process.jl:166
  [6] run(::Base.CmdRedirect; wait::Bool)
    @ Base ./process.jl:479
  [7] run
    @ ./process.jl:477 [inlined]
  [8] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:86
  [9] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [15] macro expansion
    @ ./loading.jl:1648 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611

---

Note that this is only a dummy example. Instead of PythonCall the package which is actually needed is MyPackage.

• MyPackage depends on RCall AND CondaPkg, defining a couple of R dependencies via CondaPkg.toml.
• Hence MyPackage is a bit like PythonCall in this example, in that it depends on CondaPkg.
• And a bit like RCall in that adding/importing MyPackage would build RCall itself, but RCall's R_HOME should be set by CondaPkg.envdir(). Hence MyPackage needs to be added after CondaPkg.resolve()/CondaPkg.envdir().

The ideal solution for me would be that RCall is resolving its dependency dynamically, but as of now it is not foreseeable if this will ever happen.

It would be great if CondaPkg can resolve newly added other julia dependencies which also include CondaPkg dependencies.

[cjdoris]

Ok I see, so the issue is that you (implicitly) resolve the Conda environment in order to build RCall and then load some other packages, which also have Conda dependencies, and these aren't installed because CondaPkg doesn't auto resolve this time.

The reason for this is that CondaPkg doesn't resolve if you've already resolved, unless you explicitly add any packages.

A nice solution that I think you were getting at is to also resolve if the installed Julia packages change.

Is it possible to cheaply detect if the Julia environment has changed? Maybe Pkg tracks the time that the last resolve happened? Or we could see if Pkg.project() changed but that might be quite a slow check.

[cjdoris]

Ok after a discussion on Slack the best approach seems to be to check if the mtime/hash of Manifest.toml changed since the last resolve.

[schlichtanders]

Sounds great! Thank you for inspecting/discussing this further