Release 0.14.3 backports #1135

ax3l · 2021-11-03T21:35:00Z

https://github.com/openPMD/openPMD-api/milestone/2

Read Bugs, C++17 Mixing and HDF5 Performance

This release makes reads more robust by fixing small API, file-based parsing and test bugs. Building the library in C++14 and using it in C++17 will not result in incompatible ABIs anymore. HDF5 1.10.1+ performance was improved significantly.

Thanks to @ax3l, @franzpoeschel and @jeanbez for contributing to this release!

Which version am I running?

Python

import openpmd_api
print(openpmd_api.__version__)

C++

#include <openPMD/openPMD.hpp>
#include <iostream>

int main() {
    // query compile-time macros:
    std::cout << OPENPMDAPI_VERSION_MAJOR << "."
              << OPENPMDAPI_VERSION_MINOR << "."
              << OPENPMDAPI_VERSION_PATCH << std::endl;
    // there is also: OPENPMDAPI_VERSION_LABEL (e.g. "alpha")

    // or query the runtime API (since 0.12.0):
    std::cout << openPMD::getVersion() << std::endl;
    return 0;
}

CLI

# command line option (since 0.12.0):
openpmd-ls --version

* Python: time/dt round-trip Test writing and reading time and dt on an iteration via properties. * Fix: Iteration read of long double time Support reading of `dt` and `time` attributes if they are of type `long double`. (openPMD standard: all `floatX` supported)

Set `CXX_EXTENSIONS OFF` and `CXX_STANDARD_REQUIRED ON` for created executables. This mitigates issues with NVCC 11.0 and C++17 builds seen as added `-std=gnu++17` flags that lead to ``` nvcc fatal : Value 'gnu++17' is not defined for option 'std' ``` when using `nvcc` as CXX compiler directly.

Mention the `-DPython_EXECUTABLE` twice more in build examples.

* NVCC + C++17 Work-around a build issue with NVCC in C++17 builds. ``` include/openPMD/backend/Attributable.hpp(437): error openPMD#289: no instance of constructor "openPMD::Attribute::Attribute" matches the argument list argument types are: (std::__cxx11::string) detected during instantiation of "__nv_bool openPMD::AttributableInterface::setAttribute(const std::__cxx11::string &, T) [with T=std::__cxx11::string]" ``` from ``` inline bool AttributableInterface::setAttribute( std::string const & key, char const value[] ) { return this->setAttribute(key, std::string(value)); } ``` Seen with: - NVCC 11.0.2 + GCC 8.3.0 - NVCC 11.0.2 + GCC 7.5.0 * NVCC 11.0.2 C++17 work-around: Add Comment

* Use deferred iteration parsing in openpmd-ls * Make lazy/deferred parsing searchable * Add a way to search for usesteps key

Document a HDF5 read work-around that we currently need on OLCF Jupyter (https://jupyter.olcf.ornl.gov), due to a mounting issue of GPFS in the Jupyter serice (OLCFHELP-3685). From the HDF5 1.10.1 Release Notes: ``` Other New Features and Enhancements =================================== Library ------- - Added a mechanism for disabling the SWMR file locking scheme. The file locking calls used in HDF5 1.10.0 (including patch1) will fail when the underlying file system does not support file locking or where locks have been disabled. To disable all file locking operations, an environment variable named HDF5_USE_FILE_LOCKING can be set to the five-character string 'FALSE'. This does not fundamentally change HDF5 library operation (aside from initial file open/create, SWMR is lock-free), but users will have to be more careful about opening files to avoid problematic access patterns (i.e.: multiple writers) that the file locking was designed to prevent. Additionally, the error message that is emitted when file lock operations set errno to ENOSYS (typical when file locking has been disabled) has been updated to describe the problem and potential resolution better. (DER, 2016/10/26, HDFFV-9918) ``` This also exists as a compilation option for HDF5 in CMake, where it defaults to ``TRUE`` by default, which is also what distributions/ package managers ship. Disabling from Bash: ```bash export HDF5_USE_FILE_LOCKING=FALSE ``` Disabling from Python: ```py import os os.environ['HDF5_USE_FILE_LOCKING'] = "FALSE" ```

* Make Series class final * Use private constructor to avoid object slicing

Document OpenMPI MPI-I/O backend control. We have documented this long in openPMD#446.

Include this, handy functions.

We use `<variant>` or `<mpark/variant.hpp>` in our public API interface for datatypes, depending on the C++ standard. This pull request makes sure that the same implementation is used in downstream code, even if the C++ standard is switched. This avoids ABI issues when, e.g., using a C++14 built openPMD-api in a C++17 downstream code.

The `-r` argument was removed from `spack load` and is now implied.

* GH Action: Add MSVC & ClangCL on Win * Fix AppVeyor: Python Executable * Avoid mismatching system Python and Conda Python * Conda: Fix Numpy * CMake: Skip Pipe Test Written in a too special way, we cannot assume SH is always present

…penPMD#1131) * added support to read variable encoding, plus fixed some bugs * fixed style * Update examples/8b_benchmark_read_parallel.cpp remove commented out code Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update examples/8b_benchmark_read_parallel.cpp Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * removed commented line * updated 8b env option Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Include HDF5 optimization options * Fix code style check * Fix validations and include checks * Fix style check * Remove unecessary strict check * Update documentation with HDF5 tuning options * Update contributions * Fix Guards for H5Pset_all_coll_metadata* * MPI Guard: H5Pset_all_coll_metadata* * Remove duplicated variable Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Update known issues with HDF5 and collective metadata operations * Fix rst link and tiny typo * Add targeted bugfix releases. Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Include check for paged allocation * Update ParallelHDF5IOHandler.cpp

* libfabric 1.6+: Document SST Work-Arounds Document work-arounds for libfabric 1.6+ on Cray systems when using data staging / streaming with ADIOS2 SST. Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

* [Draft] Fix: Read Inconsistent Zero Pads Some codes mess up the zero-padding in `fileBased` encoding, e.g., when specifying padding to 5 digits but creating >100'000 output steps. Files like those cannot yet be parsed and fell back to no padding, which fails to open the file: ``` openpmd_00000.h5 openpmd_02000.h5 openpmd_101000.h5 openpmd_01000.h5 openpmd_100000.h5 openpmd_104000.h5 ``` Error: ``` RuntimeError: [HDF5] Failed to open HDF5 file diags/diag1/openpmd_0.h5 ``` * Revert previous changes except for test Parse iteration numbers that are longer than their padding Read inconsistent zero padding * Overflow Padding: Read Test * Warn if the prefix does end in a digit * Fix: Don't let oversize numbers accidentally bump the padding * Update test * Issue warnings on misleading patterns also when writing * Minor Style Update Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

ax3l · 2021-11-04T02:11:09Z

CHANGELOG.rst

@@ -3,6 +3,49 @@
 Changelog
 =========

+0.14.2


Upsi 🙈

Should have read 0.14.3. Classic.

ax3l and others added 18 commits November 3, 2021 13:56

Doc: More Locations -DPython_EXECUTABLE (openPMD#1104)

57892c3

Mention the `-DPython_EXECUTABLE` twice more in build examples.

Lazy parsing: Make findable in docs and use in openpmd-ls (openPMD#1111)

23a5431

* Use deferred iteration parsing in openpmd-ls * Make lazy/deferred parsing searchable * Add a way to search for usesteps key

Avoid object slicing when deriving from Series class (openPMD#1107)

bea0c0c

* Make Series class final * Use private constructor to avoid object slicing

Doc: OMPI_MCA_io Control (openPMD#1114)

eb3e50d

Document OpenMPI MPI-I/O backend control. We have documented this long in openPMD#446.

openPMD.hpp: Include auxiliary StringManip (openPMD#1124)

29b0069

Include this, handy functions.

Spack: No More load -r (openPMD#1125)

906d68f

The `-r` argument was removed from `spack load` and is now implied.

Fix AppVeyor: Python Executable (openPMD#1127)

d3f687e

* GH Action: Add MSVC & ClangCL on Win * Fix AppVeyor: Python Executable * Avoid mismatching system Python and Conda Python * Conda: Fix Numpy * CMake: Skip Pipe Test Written in a too special way, we cannot assume SH is always present

Include known issues section for HDF5 (openPMD#1132)

8d580aa

* Update known issues with HDF5 and collective metadata operations * Fix rst link and tiny typo * Add targeted bugfix releases. Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

Include check for paged allocation (openPMD#1133)

144e8b6

* Include check for paged allocation * Update ParallelHDF5IOHandler.cpp

libfabric 1.6+: Document SST Work-Arounds (openPMD#1134)

5943435

* libfabric 1.6+: Document SST Work-Arounds Document work-arounds for libfabric 1.6+ on Cray systems when using data staging / streaming with ADIOS2 SST. Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

ax3l added bug affects latest release documentation labels Nov 3, 2021

ax3l added this to the 0.14.3 milestone Nov 3, 2021

ax3l force-pushed the release-0.14.3-backports branch from f177233 to 284b50d Compare November 3, 2021 21:37

Release: 0.14.3

ddf6db3

ax3l force-pushed the release-0.14.3-backports branch from 284b50d to ddf6db3 Compare November 3, 2021 21:38

ax3l merged commit 00532a7 into openPMD:release-0.14.3 Nov 4, 2021

ax3l deleted the release-0.14.3-backports branch November 4, 2021 01:13

ax3l commented Nov 4, 2021

View reviewed changes

CHANGELOG.rst

@@ -3,6 +3,49 @@

Changelog

=========

0.14.2

Copy link

Member Author

ax3l Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upsi 🙈

Should have read 0.14.3. Classic.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 0.14.3 backports #1135

Release 0.14.3 backports #1135

ax3l commented Nov 3, 2021 •

edited

Loading

ax3l Nov 4, 2021

Release 0.14.3 backports #1135

Release 0.14.3 backports #1135

Conversation

ax3l commented Nov 3, 2021 • edited Loading

Read Bugs, C++17 Mixing and HDF5 Performance

Which version am I running?

Python

C++

CLI

ax3l Nov 4, 2021

Choose a reason for hiding this comment

ax3l commented Nov 3, 2021 •

edited

Loading