conda install of openPMD-api causes numpy error

[PrometheusPi]

Performed steps
I freshly installed openPMD-api via conda as follows

conda create -n openpmd2 -c conda-forge openpmd-api
conda activate openpmd2
conda install -c conda-forge jupyter

This worked without any problems and due to its dependencies installed numpy alongside openPMD-api and ipython alongside juypter.
However, when loading openPMD-api in ipython, the following error occurred:

In [1]: import openpmd_api as io
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~/.conda/envs/openpmd2/lib/python3.9/site-packages/numpy/core/__init__.py in <module>
     21 try:
---> 22     from . import multiarray
     23 except ImportError as exc:

~/.conda/envs/openpmd2/lib/python3.9/site-packages/numpy/core/multiarray.py in <module>
     11 
---> 12 from . import overrides
     13 from . import _multiarray_umath

~/.conda/envs/openpmd2/lib/python3.9/site-packages/numpy/core/overrides.py in <module>
      6 
----> 7 from numpy.core._multiarray_umath import (
      8     add_docstring, implement_array_function, _get_implementing_args)

ImportError: libcblas.so.3: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-f17e2ff3c4c0> in <module>
----> 1 import openpmd_api as io

~/.conda/envs/openpmd2/lib/python3.9/site-packages/openpmd_api/__init__.py in <module>
      1 from . import openpmd_api_cxx as cxx
      2 from .openpmd_api_cxx import *  # noqa
----> 3 from .DataFrame import particles_to_dataframe
      4 from .DaskDataFrame import particles_to_daskdataframe
      5 from .DaskArray import record_component_to_daskarray

~/.conda/envs/openpmd2/lib/python3.9/site-packages/openpmd_api/DataFrame.py in <module>
      7 """
      8 import math
----> 9 import numpy as np
     10 try:
     11     import pandas as pd

~/.conda/envs/openpmd2/lib/python3.9/site-packages/numpy/__init__.py in <module>
    148     from . import _distributor_init
    149 
--> 150     from . import core
    151     from .core import *
    152     from . import compat

~/.conda/envs/openpmd2/lib/python3.9/site-packages/numpy/core/__init__.py in <module>
     46 """ % (sys.version_info[0], sys.version_info[1], sys.executable,
     47         __version__, exc)
---> 48     raise ImportError(msg)
     49 finally:
     50     for envkey in env_added:

ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/home/.../.conda/envs/openpmd2/bin/python"
  * The NumPy version is: "1.21.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libcblas.so.3: cannot open shared object file: No such file or directory

(The same errors occurs when using python instead of ipython.)

System environment

• operating system: Linux version 3.10.0-1127.19.1.el7.x86_64 (mockbuild@x86-vm-28.build.eng.bos.redhat.com)
• machine: ZIH (TU Dresden) taurus (head node)
• name and version of Python implementation: Python 3.9.7
• conda version: conda 4.4.7
• openPMD-api: openpmd-api 0.14.2 nompi_py39h7b626f2_100 conda-forge
• numpy: numpy 1.21.2 py39hdbf815f_0 conda-forge

Additional context
none

[franzpoeschel]

Copying my note from internal chat here for publicity:

I wanted to run the same thing, but somehow my TUD VPN connection stopped working, so I can't access Taurus right now. Do you know if they changed something about that?

As for the error, I did not see that particular error yet, but import errors in Python very often have to do with wrong paths. (PATH, LD_LIBRARY_PATH, PYTHONPATH) Can you check if there is something weird going on there? Can you also check if which python and which ipython point to the right binaries in the right versions? Can you check if you can import numpy successfully, and if you see the same import errors in a normal python shell?

[PrometheusPi]

Sorry for the late reply @franzpoeschel - I am back in the office and will have a look at the issue today.

[ax3l]

A general hint what you can try: conda's dependency resolver is not that good for complex dependencies as present in the -c conda-forge repository.

You could drop-in replace it with mamba: conda install -c conda-forge mamba and then use mamba install -c conda-forge ... for all further installation steps.

So in your original workflow:

conda create -n openpmd2 -c conda-forge openpmd-api mamba
conda activate openpmd2
mamba install -c conda-forge jupyter
# repeat `mamba install ...` at will :)

Alternatively, if you want to install solely with conda, spin up the whole environment at once like this. That also works robustly:

conda create -n openpmd2 -c conda-forge openpmd-api jupyter  # all at once
conda activate openpmd2
# do not add more software to this environment now

[PrometheusPi]

A perhaps related issue:
Instead of using conda I tried to use python's virtualenv directly. This is the alternative approach for user modules suggested by ZIH.

Te steps were as follows:

    # get a x86 node:
    srun --pty --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2541 --time=08:00:00 bash -l
    # make sure you are in the home directory 
    cd
    # load python used by jupyterHub
    module load Python/3.8.6-GCCcore-10.2.0

    # setup own virtualenv
    mkdir openPMD-api_kernel
    cd openPMD-api_kernel/
    virtualenv --system-site-packages openPMDapi-kernel
    source openPMDapi-kernel/bin/activate
    pip install ipykernel
    pip install --upgrade pip
    python -m ipykernel install --user --name openPMDapi-kernel --display-name="openPMD-api kernel"
    pip install openpmd-api 

On the terminal, I could run

    import openpmd_api as io 

without a problem.

But starting the kernel in jupyterHub caused the kernel to crash. Other (default) kernels worked fine.
The error in the log was not very specific (user and system related information are removed ...):

    [I 2021-11-02 11:25:36.515 SingleUserLabApp kernelmanager:179] Kernel started: ... , name: openpmdapi-kernel
    [I 2021-11-02 11:25:36.517 SingleUserLabApp log:158] 201 POST /jupyter/user/.../api/sessions?... (...@::...:127.0.0.1) 42.24ms
/home/h3/.../openPMD-api_kernel/openPMDapi-kernel/bin/python: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory 

Next, I will try mamba.

@franzpoeschel as far as I know ZIH did not change anything with the TUD VPN.

[ax3l]

@PrometheusPi in order to install your own kernels in Jupyter for HPC, you need to modify the startup script slightly.
The rationale behind this is, that Jupyter needs to:

• load the modules you used for your Python and
• activate the virtual environment where you installed additional software in

so it can start up the right Python interpreter and environment from Jupyter.

NERSC has a great document on this, all you need is a little helper script for the Jupyter kernel config file:
https://docs.nersc.gov/services/jupyter/#conda-environments-as-kernels

They show it on conda, but it's equivalent for virtual envs. Your content of that kernel-helper.sh script would be something like:

#!/bin/bash -l
module load Python/3.8.6-GCCcore-10.2.0
source $HOME/openPMDapi-kernel/bin/activate
exec python -m ipykernel_launcher "$@"

I also added a cross-link on this in the WarpX docs:
https://warpx.readthedocs.io/en/latest/install/hpc/cori.html#post-processing

Maybe you could add a similiar sub-section in PIConGPU's Taurus docs to document process (unless ZIH has a place that you can link to).

[ax3l]

Another side-note: for more robust installs, replace pip install with python3 -m pip install .... That way, you are 100% sure which Python environment you are dealing with (the one from which python) and this also works on systems where PATH is not modified when pip got installed.