improve Blosc compression operator #2592

psychocoderHPC · 2021-01-20T12:44:54Z

fix: #2018

The current implementation of the Blosc ADIOS2 operator is due to limitations of Blosc not supporting adios variables larger than 2GiB.
ADIOS1 was originally using a chunked format to work around this limitation.

Add option threshold to set a threshold under which data will only be copied instead of compressed (equivalent to Blosc in ADIOS1).
Use chunk format to avoid 2GiB limit for variables (datasets).
Add support to decompress Blosc data written with ADIOS before this PR.
- update blosc tests

The header written to decompressed the date is different from the ADIOS1 implementation.
Compared to ADIOS1 this PR implements compression and decompression much cleaner.

@maintainer Where do I should document the new option threshold for the Blosc operator. I have not found anything in the documentation.

Thanks to @franzpoeschel for providing the test code testBlosc.txt
.
CC-ing: @ax3l

pnorbert · 2021-01-20T13:27:19Z

Thank you for fixing this compressor. Unfortunately, you caught us with the docs, there is no section on operators/compressors.
@chuckatkins Can you please add redirect this to the patch release?

pnorbert · 2021-01-20T14:04:39Z

@psychocoderHPC Can you please extend the tests in testing/adios2/engine/bp/operations/TestBPWriteReadBlosc.cpp so that the threshold settings are tested? You need to look at this test anyway as the memory sanitizer fails on it...

williamfgc · 2021-01-20T14:06:39Z

@psychocoderHPC thanks, that was long overdue, only bzip2 was doing that and the hope was to migrate to blosc2 eventually as in this issue. @pnorbert there is almost nothing in the docs on compression operations since each engine supports its own set.

psychocoderHPC · 2021-01-20T15:53:02Z

@psychocoderHPC Can you please extend the tests in testing/adios2/engine/bp/operations/TestBPWriteReadBlosc.cpp so that the threshold settings are tested? You need to look at this test anyway as the memory sanitizer fails on it...

I updated the blosc tests and added test combinations for shuffling and threshold

psychocoderHPC · 2021-01-20T15:58:03Z

source/adios2/operator/compress/CompressBlosc.h

+                               void *dataOut, const size_t sizeOut,
+                               Params &info) const;
+
+    class __attribute__((packed)) DataHeader


@pnorbert I fixed hopefully the ubsan issue in the CI by disallowing the compiler to add padding.
The disadvantage is that there is no standard attribute in C++ :-(
Is this way ok? If not, do you have a suggestion on how I can fix this issue instead?

if it helps, there is an internal helper function to add padding in a portable way.

I am not sure if this helps. The problem is that maybe the ptr for the output is not aligned during the compression. If I use the helper and shift my data a few bytes and later during decompress I get the data within an array with a different base address. If so it would be not possible to find the data because the required padding is different.

If I understand the ubsan error for the old code correctly and what I found in the net the packed attribute is required to guarantee that the struct has the same layout even if you compile blosc on a different system. This attributes forbids the compiler to transform the struct which is IMO correct.

I can also remove the header class und work with two pure integers but I found it better readable to use the header class because it allows the code to shift with sizeof(Header) instead of using a magic number 8byte.

I am not sure which solution guarantees portability.

Maybe it is enough and better if I align the struct to 4 byte.
Is ADIOS guaranteeing that my pointers given to compress operators are 4 byte aligned?

The problem is that maybe the ptr for the output is not aligned during the compression.

The function I shared can verify that returning the required padding in bytes (if 0 no padding is required).

Is ADIOS guaranteeing that my pointers given to compress operators are 4 byte aligned?

That's a good question. It depends on the serialization buffer consuming this interface. The BP buffer is char so it shouldn't be a problem. I'd check with @pnorbert for other existing strategies.

I am not sure which solution guarantees portability.

Yup, that's the challenge.

It looks like Blosc is not covered by CI and thus breaking on MSVC: #2745

I found this on SO, we could add an extra handling for MSVC and assume other compilers support the currently GCC-like syntax for now: https://stackoverflow.com/a/3312896/2719194

psychocoderHPC · 2021-01-20T16:07:33Z

that and the hope was to migrate to blosc2 eventually as in this issue.

I also hoped to switch to blosc2 and it is solving the 2GiB issue but unluckily as I know they now support larger size-types but you need to explicitly pack the data with the framing API and there is no simple function available which is doing it transparently.
Therefore I decided to stay with the well-tested approach from ADIOS1 with blosc1.

The current implementation of the Blosc ADIOS2 operator is due to limitations of Blosc not supporting variables larger than 2GiB. - Add option `threshold` to set a threshold under which data will only be copied instead of compressed (equivalent to Blosc in ADIOS1). - Use chunk format to avoid 2GiB limit for variables (datasets). - Add support to decompress blosc data written with ADIOS before this PR.

- add test combination for new option `threshold` - add test combination for shuffling

chuckatkins · 2021-01-20T18:38:40Z

I've rebased the topic on release_27 as a candidate for the 2.7.1 patch release. Please update your local branch with the remote.

improve Blosc compression operator

Add support for improved blosc support: ornladios/ADIOS2#2592

* setup.py: bump version to 0.13.4.post0 * ADIOS: 2.6.0 -> 2.7.1 Add support for improved blosc support: ornladios/ADIOS2#2592 * BLOSC: 1.20.1 -> 1.21.0 * ADIOS2: Linux PThread Issue w/ Blosc ``` /usr/local/lib/libblosc.a(blosc.c.o): In function `blosc_init.part.9': blosc.c:(.text+0x43e): undefined reference to `pthread_atfork' ``` * ADIOS 2.7.1: No Build/Install Test * Windows: ADIOS 2.7.0 Better an older ADIOS than no BLOSC support at all. * Blosc: Avoid Pickup of Static pthread lib * Nope, keep LDFLAGS * Win: ADIOS 2.7.1 w/ MSVC Blosc Patch

psychocoderHPC mentioned this pull request Jan 20, 2021

C-Blosc: Support for Large Vars #2018

Closed

psychocoderHPC force-pushed the topic-bloscImprovements branch from ae5dc35 to 7327660 Compare January 20, 2021 15:51

psychocoderHPC commented Jan 20, 2021

View reviewed changes

psychocoderHPC added 2 commits January 20, 2021 13:36

update blosc tests

c83fcb7

- add test combination for new option `threshold` - add test combination for shuffling

chuckatkins force-pushed the topic-bloscImprovements branch from 7327660 to c83fcb7 Compare January 20, 2021 18:36

chuckatkins changed the base branch from master to release_27 January 20, 2021 18:37

chuckatkins added this to the ADIOS v2.7.1 Patch Release milestone Jan 20, 2021

chuckatkins approved these changes Jan 22, 2021

View reviewed changes

chuckatkins merged commit a7d9507 into ornladios:release_27 Jan 22, 2021

chuckatkins pushed a commit that referenced this pull request Jan 22, 2021

Merge pull request #2592 from psychocoderHPC/topic-bloscImprovements

ea846af

improve Blosc compression operator

psychocoderHPC mentioned this pull request Jan 28, 2021

openPMD error ADIOS2 ComputationalRadiationPhysics/picongpu#3506

Closed

psychocoderHPC deleted the topic-bloscImprovements branch March 3, 2021 07:29

ax3l added a commit to ax3l/openPMD-api that referenced this pull request Jun 4, 2021

ADIOS: 2.6.0 -> 2.7.1

319e75d

Add support for improved blosc support: ornladios/ADIOS2#2592

ax3l mentioned this pull request Jun 4, 2021

Wheels: 0.13.4.post0 openPMD/openPMD-api#1002

Merged

ax3l added a commit to ax3l/openPMD-api that referenced this pull request Jun 4, 2021

ADIOS: 2.6.0 -> 2.7.1

8adb4c3

Add support for improved blosc support: ornladios/ADIOS2#2592

This was referenced Jun 4, 2021

ADIOS 2.7.1 Build Issues (Conda-Forge) #2617

Closed

Compress Blosc: Fix MSVC Build #2746

Merged

psychocoderHPC mentioned this pull request Dec 1, 2022

BLOSC2 framing API #3397

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve Blosc compression operator #2592

improve Blosc compression operator #2592

psychocoderHPC commented Jan 20, 2021 •

edited

Loading

pnorbert commented Jan 20, 2021

pnorbert commented Jan 20, 2021

williamfgc commented Jan 20, 2021

psychocoderHPC commented Jan 20, 2021

psychocoderHPC Jan 20, 2021

williamfgc Jan 20, 2021

psychocoderHPC Jan 20, 2021 •

edited

Loading

psychocoderHPC Jan 20, 2021 •

edited

Loading

williamfgc Jan 20, 2021

ax3l Jun 4, 2021 •

edited

Loading

psychocoderHPC commented Jan 20, 2021

chuckatkins commented Jan 20, 2021

improve Blosc compression operator #2592

improve Blosc compression operator #2592

Conversation

psychocoderHPC commented Jan 20, 2021 • edited Loading

pnorbert commented Jan 20, 2021

pnorbert commented Jan 20, 2021

williamfgc commented Jan 20, 2021

psychocoderHPC commented Jan 20, 2021

psychocoderHPC Jan 20, 2021

Choose a reason for hiding this comment

williamfgc Jan 20, 2021

Choose a reason for hiding this comment

psychocoderHPC Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

psychocoderHPC Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

williamfgc Jan 20, 2021

Choose a reason for hiding this comment

ax3l Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

psychocoderHPC commented Jan 20, 2021

chuckatkins commented Jan 20, 2021

psychocoderHPC commented Jan 20, 2021 •

edited

Loading

psychocoderHPC Jan 20, 2021 •

edited

Loading

psychocoderHPC Jan 20, 2021 •

edited

Loading

ax3l Jun 4, 2021 •

edited

Loading