hdf5 files

[rezaplasma]

hi every one
I am new in Warp, i have a question about output formats
i have read that Warp can write diagnostics data as openPMD format (e.g. hdf5), but when i run (script).py with python the results are not as ().h5, i would just to know how i can create HDF5 files
does i need add feature in my script or not?
thank you

[ax3l]

Hi @rezaplasma,

Thanks for the question!

We implement multiple diagnostics in WarpX, ultimately moving fully to openPMD. Our openPMD output is either in .h5 (HDF5) or .bp (ADIOS2): https://warpx.readthedocs.io/en/latest/dataanalysis/formats.html

when i run (script).py with python the results are not as ().h5, i would just to know how i can create HDF5 files does i need add feature in my script or not?

Yes, there is an option to add until we change the default. In PICMI Python input, the options you need to add are:

ParticleDiagnostic( # ...
    warpx_format='openpmd',
    warpx_openpmd_backend='h5'
)

and similar for FieldDiagnostic.

Does this work for you? Let us know if it does or if you have further questions :)

[rezaplasma]

hi @ax3l
thanks for your answering
I will try it and let you know
does these
warpx_format='openpmd', and
warpx_openpmd_backend='h5'
are arguments for
ParticleDiagnostic and FieldDiagnostic objects ?

[rezaplasma]

hi @ax3l
i have set these arguments but i have again faced this warning

--- INFO    : Writing openPMD file diags/Python_LaserAcceleration_1d_plt000000
terminate called after throwing an instance of 'openPMD::error::WrongAPIUsage'
  what():  Wrong API usage: openPMD-api built without support for backend 'HDF5'.

does i must add an option warpx_file_prefix
what's the issue?
thank you

[ax3l]

Hi @rezaplasma,

Thank you for the update!

How did you install WarpX? Do you mind sharing the exact details (setup of dependencies, CMake commands and output)? It looks like HDF5 was not provided during compilation, and thus cannot be used.

Best,
Axel

[rezaplasma]

Hi @ax3l

Thanks for replying
This is the error that i got

--- INFO    : Writing openPMD file ./Python_LaserAcceleration_1d_plt000000
terminate called after throwing an instance of 'openPMD::error::WrongAPIUsage'
  what():  Wrong API usage: openPMD-api built without support for backend 'HDF5'.
SIGABRT
/usr/bin/addr2line: '/home/user/warp_x/warpx': No such file
/usr/bin/addr2line: '/home/user/warp_x/warpx': No such file
/usr/bin/addr2line: '/home/user/warp_x/warpx': No such file
/usr/bin/addr2line: '/home/user/warp_x/warpx': No such file
.
.
.

I have installed the warpx by using the pypi package
On 2 local computers but error was same
First i used the Apt tutorials to install warpx dependencies
Then used these commands respectively:

1. python3 -m pipi install -U pip setuptools wheel
2. python3 -m pipi install -U cmake
3. python3 -m pip wheel -v git+... WarpX.git
   and ending
4. python3 -m pip install *whl

[rezaplasma]

@ax3l
It is possible to guide me
Solving this issue is taking more than 2 weeks

[ax3l]

Hi @rezaplasma,

Thanks for the details.

So as mentioned above, your lack some optional WarpX dependencies, the right variant of HDF5 in particular.

Since you followed the Debian/Ubuntu apt logic, you have everything in place for an MPI-based install. In particular, parallel HDF5.

Try this to switch MPI on (default is OFF currently for the Python bindings), which will then find HDF5:

WARPX_MPI=ON python3 -m pip wheel -v git+https://github.com/ECP-WarpX/WarpX.git
python3 -m pip install --force-reinstall *whl

During compilation, you can already confirm if HDF5 was found checking the table at the end of the CMake output. It should read something like this:

...
openPMD build configuration:
  library Version: 0.15.1
  openPMD Standard: 1.1.0
  C++ Compiler: GNU 11.3.0 
    /usr/bin/c++

  Installation: OFF

  Build Type: Release
  Library: static
  CLI Tools: OFF
  Examples: OFF
  Testing: OFF
  Invasive Tests: OFF
  Internal VERIFY: ON
  Build Options:
    MPI: ON
    HDF5: ON     <--------------------
    ADIOS1: OFF
    ADIOS2: ON
    PYTHON: OFF
    CUDA_EXAMPLES: OFF

...
WarpX build configuration:
  Version: 23.06 ()
  C++ Compiler: GNU 11.3.0 
    /usr/bin/c++

  Installation prefix: /usr/local
        bin: bin
        lib: lib
    include: include
      cmake: lib/cmake/WarpX

  Build type: Release
  Build options:
    APP: ON
    ASCENT: OFF
    COMPUTE: OMP
    DIMS: 3
    Embedded Boundary: OFF
    GPU clock timers: OFF
    IPO/LTO: OFF
    LIB: ON
    MPI: ON
    PSATD: OFF
    PRECISION: DOUBLE
    PARTICLE PRECISION: DOUBLE
    OPENPMD: ON
    QED: ON
    QED table generation: OFF
    SENSEI: OFF

-- Configuring done
-- Generating done
-- Build files have been written to: ...

[rezaplasma]

hi @ax3l

thanks for replying

dr ax3l ,
i have tried the command mentioned above, but the HDF5 is not switched on , as following:

 -- Downloading openPMD-api ...
  -- openPMD-api repository: https://github.com/openPMD/openPMD-api.git (0.15.1)
  -- Found MPI: TRUE (found version "3.1") found components: CXX
  -- Using the single-header code from /tmp/pip-req-build-3suzx7_k/build/temp.linux-x86_64-cpython-39/1/_deps/fetchedopenpmd-src/share/openPMD/thirdParty/json/single_include/
  -- nlohmann-json: Using INTERNAL version '3.9.1'
  -- toml11: Using INTERNAL version '3.7.1'
  -- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
  -- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C) (found version "")
  -- Could NOT find ADIOS2 (missing: ADIOS2_DIR)

  openPMD build configuration:
    library Version: 0.15.1
    openPMD Standard: 1.1.0
    C++ Compiler: GNU 11.3.0
      /usr/bin/c++

    Installation: OFF

    Build Type: Release
    Library: static
    CLI Tools: OFF
    Examples: OFF
    Testing: OFF
    Invasive Tests: OFF
    Internal VERIFY: ON
    Build Options:
      MPI: ON
      HDF5: OFF         <------------
      ADIOS1: OFF
      ADIOS2: OFF
      PYTHON: OFF
      CUDA_EXAMPLES: OFF

  -- Found Git: /usr/bin/git (found version "2.34.1")

  WarpX build configuration:
    Version: 23.06 ()
    C++ Compiler: GNU 11.3.0
      /usr/bin/c++

    Installation prefix: /usr/local
          bin: bin
          lib: lib
      include: include
        cmake: lib/cmake/WarpX

    Build type: Release
    Build options:
      APP: OFF
      ASCENT: OFF
      COMPUTE: OMP
      DIMS: 1
      Embedded Boundary: OFF
      GPU clock timers: OFF
      IPO/LTO: OFF
      LIB: ON (static)
      MPI: ON
      PSATD: OFF
      PRECISION: DOUBLE
      PARTICLE PRECISION: DOUBLE
      OPENPMD: ON
      QED: ON
      QED table generation: OFF
      SENSEI: OFF

and at the end of running , i have got the following error:

/home/user/anaconda3/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
  failure.
  removing: _configtest.c _configtest.o
  error: Cannot compile MPI programs. Check your configuration!!!
  error: subprocess-exited-with-error
  
  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/user/anaconda3/bin/python3 /home/user/anaconda3/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmps4nifozw
  cwd: /tmp/pip-wheel-kpqww6cq/mpi4py_cfd2164865fa4992bdab2a5a92559979
  Building wheel for mpi4py (pyproject.toml) ... error
  ERROR: Failed building wheel for mpi4py
Successfully built pywarpx
Failed to build mpi4py
ERROR: Failed to build one or more wheels

[rezaplasma]

@ax3l

I installed warp on 2 types of system (local and clustered operating systems) with pypi package, but the HDF5 option is not switched on for both

I can't really figure out what goes wrong!

[ax3l]

Hi @rezaplasma,

That is unusual indeed, but we can provide parallel HDF5 one way or another. Which operating system is this, Ubuntu 22.04? Let me double check this in a minimal docker container to see if any package might be missing.

If this turns out to be a problem with apt, then I will show you how to quickly compile HDF5 yourself for your system.

For the cluster, what operating system is that and do you have a software module that you can load for HDF5?