Error when using PythonCall attached to a previous Conda environment.

[martinmestre]

Hello,
I would be very gratefull to understand what is the problem. I followed the instructions:

julia > ENV["JULIA_CONDAPKG_BACKEND"] = "System"
julia > ENV["JULIA_CONDAPKG_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"

Then I install PythonCall.

(RiosDelCosmos) pkg> st
      Status `~/casa/work/2022/RiosDelCosmos/Project.toml`
  [6099a3de] PythonCall v0.9.6

When I try to use it I get an error (I do have astropy installed in the cosmos environment):

julia> using PythonCall
julia> ast = pyimport("astropy")
ERROR: Python: ModuleNotFoundError: No module named 'astropy'
Python stacktrace: none
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/DqZCE/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/DqZCE/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/DqZCE/src/concrete/import.jl:11
 [4] top-level scope
   @ REPL[5]:1

I also tried with
julia > ENV["JULIA_CONDAPKG_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python"
obtaining the same result.
Thank you very much in advance.

[cjdoris]

See here: https://cjdoris.github.io/PythonCall.jl/stable/pythoncall/#If-you-don't-want-to-use-Conda

I think you just need

ENV["JULIA_PYTHONCALL_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"

You must also ensure your Conda environment is activated before starting Julia, since CondaPkg is not used if you specify the python executable.

[martinmestre]

Thanks @cjdoris. It worked!
Just for my understanding:

The commands:

julia > ENV["JULIA_CONDAPKG_BACKEND"] = "System"
julia > ENV["JULIA_CONDAPKG_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"
julia > ENV["JULIA_PYTHONCALL_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"

must be applied once before installing PythonCall, or each time I run Julia?

I think I didnt understand. I followed the instructions named: "If you already have Conda, Mamba or MicroMamba" so I guess I wnat to use conda (my conda environment) but not CondaPkg.

[martinmestre]

Hello @cjdoris, I do not understand why your command
ENV["JULIA_PYTHONCALL_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"
worked, because according to the tutorial, that env variable must be set when one doesn't want to use conda.
And I want to use conda. In fact I want to use my conda environment.

Alternatively I also tried to use CondaPkg but I have a Python packages that is installed like this and it gave me errors.

[cjdoris]

OK, maybe the wording of those instructions could be made clearer: by "use Conda" I really mean "use CondaPkg to automatically create a new Conda environment with all your dependencies", which is not what you want, since you want to re-use an existing environment.

By setting JULIA_PYTHONCALL_EXE, you tell PythonCall exactly which Python executable you'd like to use (it doesn't matter whether or not it was installed by Conda) and PythonCall will not use CondaPkg and so will not install any Conda packages for you. So it is up to you to ensure all your Python dependencies are already installed.

It should not be necessary to set any other environment variables if you are only using PythonCall, but you can also set JULIA_CONDAPKG_BACKEND=Null to completely prevent CondaPkg from installing anything. You do not need to set JULIA_CONDAPKG_EXE at all (and in your example it was wrong, since it is meant to be a Conda executable, not Python).

These env vars need to be set every session. To do this automatically you can do this in your startup.jl file, or put them into the init file of whichever shell you use.

[martinmestre]

Thanks! Ok, so I think I have understood almost everything. I have been able to use PythonCall with a previous conda env by setting only: JULIA_PYTHONCALL_EXE.
For completeness, only remains to know in which case I should set
JULIA_CONDAPKG_EXE.
Thank you very much again.

[cjdoris]

JULIA_CONDAPKG_EXE is (currently) only used with the System backend (JULIA_CONDAPKG_BACKEND=System), to change which conda (or mamba or micromamba) executable to use. The System backend only changes which conda executable is used, but otherwise has the default CondaPkg behaviour, namely it will create a new Conda environment specific to your Julia project. Hence the only way to use a pre-existing Conda environment right now is to set JULIA_PYTHONCALL_EXE=/path/to/python or JULIA_CONDAPKG_BACKEND=Null.

[cjdoris]

I've reworded the docs a bit, I'd be grateful if you could read and tell me if they are clearer: https://cjdoris.github.io/PythonCall.jl/dev/pythoncall/#pythoncall-config

[martinmestre]

Thanks, of course I will read it.
I think it is clearer now. Just to confirm I understood correctly. If I want to use a conda environment similar to a preexisting one, and be able to add further packages, then I should use CondaPkg with:

ENV["JULIA_CONDAPKG_BACKEND"] = "Current"
ENV["JULIA_CONDAPKG_EXE"] = "/path/to/conda"  # optional

Right?

[cjdoris]

Yes, the Current backend will allow CondaPkg to install things into whatever Conda environment is currently active. It's a new backend (I added it today) and has not been released yet.

[martinmestre]

Thank you a lot! issue closed!