ADIOS2: Flush to disk within a step #1207

franzpoeschel · 2022-02-24T18:26:40Z

This was recently added in ADIOS2 (~~I think~~ only for BP5) and we need it to save memory e.g. in PIConGPU

Also it could be used to enable the IO pattern used by HiPace that so far only works in HDF5.

TODO

include/openPMD/IO/AbstractIOHandler.hpp

franzpoeschel · 2022-03-23T12:42:09Z

@psychocoderHPC asked about more fine-grained control for this to avoid flushing to disk when only some KB are in the buffers. There are several possible solutions that I can imagine:

Add a flushing threshold to our ADIOS2 backend, e.g. adios2.engine.flushing_threshold = "2GiB". If the ADIOS2 backend notices that more than 2GiB were flushed, then it will store data to disk, otherwise use PerformPuts().
Problem -> parallelism
Turn "flush_during_step" into a dataset-specific option
Add JSON configuration to the flush() API, e.g. series.flush(R"(adios2.preferred_flush_target = "disk")");

ax3l · 2022-04-15T17:23:36Z

test/SerialIOTest.cpp

@@ -30,6 +30,10 @@
 #include <tuple>
 #include <vector>

+#if __GNUC__ >= 11
+#include <filesystem>


Can we use an auxiliary function for this?
Filesystem is tricky and sometimes needs also extra link flags like -lstdc++fs, that I don't yet want to get into.

Or did this get better with GCC 11+?

Note that the __GNUC__ macro does not mean GCC specifically and is also set by clang, icpc and others.

Ah, good catch about the __GNUC__ thing, will fix

I can have a look if I can do the same test with fstat or similar as well, might also go with this.
I only need this test to ensure that the right API call is chosen by the ADIOS2 backend, so restricting it to a single compiler set that implements <filesystem> well should be ok as an alternative

franzpoeschel · 2022-05-05T09:38:05Z

docs/source/details/backendconfig.rst

@@ -119,6 +122,12 @@ Explanation of the single keys:
  Please refer to the `official ADIOS2 documentation <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_ for the available engine parameters.
  The openPMD-api does not interpret these values and instead simply forwards them to ADIOS2.
 * ``adios2.engine.usesteps``: Described more closely in the documentation for the :ref:`ADIOS2 backend<backends-adios2>` (usesteps).
+* ``adios2.engine.preferred_flush_target`` Only relevant for BP5 engine, possible values are ``"disk"`` and ``"buffer"`` (default: ``"disk"``).


I've called this preferred_flush_target since it is only implemented by BP5. Otherwise, we will see user requests "I specified flush_target=disk, why is BP4 buffering the data anyway?".
I don't really want to explicitly use the BP5 name, e.g. bp5_flush_target=disk because other engines might in future implement this too.

ax3l · 2022-05-09T22:32:26Z

@franzpoeschel please needs a small rebase

franzpoeschel · 2022-06-02T12:32:29Z

include/openPMD/IO/FlushParametersInternal.hpp

+#pragma once
+
+#include "openPMD/IO/AbstractIOHandler.hpp"
+#include "openPMD/auxiliary/JSON_internal.hpp"


Cannot include this in public headers, hence the necessary distinction between FlushParams and ParsedFlushParams, even if both structs are used only internally.
Unifying both structs requires removing AbstractIOHandler.hpp from public headers.

Good question why we need AbstractIOHandler.hpp in public headers actually.

Because we use it heavily in the private: sections of our public classes, which are still defined in the headers. We would have to move the entire implementation logic out of the public headers to avoid including it, which would be a very heavy restructuring (and most likely end up in a solution similar what ADIOS2 does by having a separate source tree only for the public APIs)

Same impression, would not have this added to our priorities right now.

docs/source/details/backendconfig.rst

Parse string JSON config into nlohmann representation

Also, use fstat in tests rather than <filesystem>

There is no advantage from doing this, readers will only see the data after EndStep anyway, but there's the disadvantage that attributes cannot be overwritten within this step anymore

ax3l · 2022-07-28T01:15:41Z

Can you please post the multi-KB .png images as attachment to the PR description?
Then you can link to that image (click on it, get link) via this .rst syntax: ECP-WarpX/impactx#173

.. figure:: https://user-images.githubusercontent.com/1353258/180332191-f9ce11fc-8c56-4713-a91a-2ad12ab09805.png
   :alt: Chicane beam width and emittance evolution

This allows us to not have to check them into the source code and avoid that our source repo gets too large (we clone openPMD-api very often in CMake superbuilds).

franzpoeschel · 2022-07-28T09:53:42Z

BP5 without intermittent flushes (old behavior, now opt-in):

BP5 with intermittent flushes (new behavior)

BP4 file-based with specified InitialBufferSize as ~6GB:

Catastrophic behavior: BP4 without using steps, no intermittent output possible

Streaming with a slow reader, writer gets congested:

Streaming with fast reader, InitialBufferSize specified:

Variable-based encoding in BP4 with InitialBufferSize specified, buffer is reused:

Variable-based encoding in BP4 without InitialBufferSize specified, exponential probing phase in the beginning (memory usage >= twice the actually needed memory size due to reallocation)

franzpoeschel · 2022-07-28T11:16:14Z

Can you please post the multi-KB .png images as attachment to the PR description? Then you can link to that image (click on it, get link) via this .rst syntax: ECP-WarpX/impactx#173
.. figure:: https://user-images.githubusercontent.com/1353258/180332191-f9ce11fc-8c56-4713-a91a-2ad12ab09805.png
   :alt: Chicane beam width and emittance evolution
This allows us to not have to check them into the source code and avoid that our source repo gets too large (we clone openPMD-api very often in CMake superbuilds).

Good idea, yeah
Last commit does that

ax3l

Awesome, thanks a lot 🎉

* dev: (70 commits) Docs: Recommend Static Build for Superbuilds (openPMD#1325) Python 3.11 (openPMD#1323) pybind11: v2.10.1+ (openPMD#1322) Add Attribute::getOptional<T>() and use to add some more dynamic datatype conversions at read time (openPMD#1278) Mapping between ADIOS steps and openPMD iterations (openPMD#949) Deprecate shareRaw (openPMD#1229) Fix append mode double attributes (openPMD#1302) Constant scalars: Don't flush double (openPMD#1315) Remove caching cmake vars (openPMD#1313) [pre-commit.ci] pre-commit autoupdate (openPMD#1311) storeChunk: Add an overload for shared_ptr<T[]> (openPMD#1296) Fix `operationAsString` Export (openPMD#1309) ADIOS2: more fine-grained control for file endings (openPMD#1218) [pre-commit.ci] pre-commit autoupdate (openPMD#1307) Fix file existence check in parallel tests (openPMD#1303) ADIOS2: Flush to disk within a step (openPMD#1207) [pre-commit.ci] pre-commit autoupdate (openPMD#1304) [pre-commit.ci] pre-commit autoupdate (openPMD#1295) Update catch2 to v2.13.9 (openPMD#1299) [pre-commit.ci] pre-commit autoupdate (openPMD#1292) ... # Conflicts: # .github/workflows/linux.yml

franzpoeschel added backend: ADIOS2 api: new additions to the API labels Feb 24, 2022

franzpoeschel force-pushed the topic-flush-inside-step branch 3 times, most recently from cb3ed7b to bd3aec3 Compare February 24, 2022 19:03

franzpoeschel mentioned this pull request Feb 28, 2022

openPMD plugin: Flush data to disk within a step ComputationalRadiationPhysics/picongpu#4002

Merged

5 tasks

franzpoeschel force-pushed the topic-flush-inside-step branch 2 times, most recently from d4d7be3 to 1cba129 Compare March 2, 2022 15:13

ax3l requested review from ax3l and guj March 8, 2022 18:54

ax3l reviewed Mar 8, 2022

View reviewed changes

include/openPMD/IO/AbstractIOHandler.hpp Outdated Show resolved Hide resolved

franzpoeschel force-pushed the topic-flush-inside-step branch from e83b4ff to 456bbac Compare March 10, 2022 10:50

franzpoeschel mentioned this pull request Mar 10, 2022

Pass-through flushing parameters #1226

Merged

franzpoeschel changed the title ~~[WIP] ADIOS2: Flush to disk within a step~~ ADIOS2: Flush to disk within a step Mar 16, 2022

franzpoeschel requested a review from ax3l March 16, 2022 09:59

ax3l reviewed Apr 15, 2022

View reviewed changes

franzpoeschel mentioned this pull request Apr 28, 2022

Should Iteration::close(flush=true) write files? #986

Open

franzpoeschel force-pushed the topic-flush-inside-step branch 3 times, most recently from 95a84f1 to d5bb4b6 Compare May 4, 2022 13:27

franzpoeschel commented May 5, 2022

View reviewed changes

franzpoeschel force-pushed the topic-flush-inside-step branch from 61f286e to 97ccd40 Compare May 16, 2022 15:41

ax3l self-assigned this May 24, 2022

franzpoeschel commented Jun 2, 2022

View reviewed changes

franzpoeschel commented Jun 3, 2022

View reviewed changes

docs/source/details/backendconfig.rst Outdated Show resolved Hide resolved

franzpoeschel force-pushed the topic-flush-inside-step branch 2 times, most recently from bc68d36 to 9387b8b Compare July 5, 2022 15:11

franzpoeschel mentioned this pull request Jul 6, 2022

Add unique_ptr overloads for storeChunk calls #1294

Merged

9 tasks

franzpoeschel added 6 commits July 19, 2022 10:49

Add backendConfig parameter to flush call

b12c752

Parse string JSON config into nlohmann representation

ADIOS2 implementation: preferred_flush_target parameter

8609cd8

Testing

a22d25b

Documentation

a7ee2ce

Add override parameters, update tests and docs

0808003

Also, use fstat in tests rather than <filesystem>

Flush attributes before PerformDataWrites()

f06c909

franzpoeschel force-pushed the topic-flush-inside-step branch from 1d463da to f06c909 Compare July 19, 2022 08:51

Don't write buffered attributes upon PerformDataWrites

dd7b3cb

There is no advantage from doing this, readers will only see the data after EndStep anyway, but there's the disadvantage that attributes cannot be overwritten within this step anymore

Use images from PR thread

b5513c4

ax3l approved these changes Jul 28, 2022

View reviewed changes

ax3l merged commit b05171d into openPMD:dev Jul 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADIOS2: Flush to disk within a step #1207

ADIOS2: Flush to disk within a step #1207

franzpoeschel commented Feb 24, 2022 •

edited

Loading

franzpoeschel commented Mar 23, 2022 •

edited

Loading

ax3l Apr 15, 2022 •

edited

Loading

ax3l Apr 15, 2022

franzpoeschel Apr 25, 2022

franzpoeschel May 5, 2022

ax3l commented May 9, 2022

franzpoeschel Jun 2, 2022

ax3l Jul 28, 2022

franzpoeschel Jul 28, 2022

ax3l Jul 28, 2022 •

edited

Loading

ax3l commented Jul 28, 2022 •

edited

Loading

franzpoeschel commented Jul 28, 2022 •

edited

Loading

franzpoeschel commented Jul 28, 2022

ax3l left a comment

ADIOS2: Flush to disk within a step #1207

ADIOS2: Flush to disk within a step #1207

Conversation

franzpoeschel commented Feb 24, 2022 • edited Loading

franzpoeschel commented Mar 23, 2022 • edited Loading

ax3l Apr 15, 2022 • edited Loading

Choose a reason for hiding this comment

ax3l Apr 15, 2022

Choose a reason for hiding this comment

franzpoeschel Apr 25, 2022

Choose a reason for hiding this comment

franzpoeschel May 5, 2022

Choose a reason for hiding this comment

ax3l commented May 9, 2022

franzpoeschel Jun 2, 2022

Choose a reason for hiding this comment

ax3l Jul 28, 2022

Choose a reason for hiding this comment

franzpoeschel Jul 28, 2022

Choose a reason for hiding this comment

ax3l Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

ax3l commented Jul 28, 2022 • edited Loading

franzpoeschel commented Jul 28, 2022 • edited Loading

franzpoeschel commented Jul 28, 2022

ax3l left a comment

Choose a reason for hiding this comment

franzpoeschel commented Feb 24, 2022 •

edited

Loading

franzpoeschel commented Mar 23, 2022 •

edited

Loading

ax3l Apr 15, 2022 •

edited

Loading

ax3l Jul 28, 2022 •

edited

Loading

ax3l commented Jul 28, 2022 •

edited

Loading

franzpoeschel commented Jul 28, 2022 •

edited

Loading