Repository navigation
Conversation
brtnfld
requested review from
fortnern,
gheber,
glennsong09,
hyoklee,
jhendersonHDF,
lrknox,
mattjala and
vchoi-hdfgroup
as code owners
September 26, 2026 08:12
This was referenced Sep 26, 2026
Open
brtnfld
force-pushed
the
filter-config/04-dcpl-api
branch
from
September 26, 2026 09:06
a318288 to
991852f
Compare
brtnfld
force-pushed
the
filter-config/05-pline-v3
branch
from
September 26, 2026 09:06
d769f42 to
4a763ce
Compare
brtnfld
force-pushed
the
filter-config/05-pline-v3
branch
from
September 26, 2026 11:19
4a763ce to
ebbde92
Compare
brtnfld
force-pushed
the
filter-config/04-dcpl-api
branch
from
September 26, 2026 11:19
991852f to
85d8b8a
Compare
brtnfld
force-pushed
the
filter-config/05-pline-v3
branch
from
September 27, 2026 06:34
ebbde92 to
70b7f8a
Compare
brtnfld
added a commit
to brtnfld/RFC-HDFG-2026-001
that referenced
this pull request
Sep 27, 2026
A reader rejects a known block type that sets a flag bit the type does not assign, accepts blocks in any order, and rejects an H5O_PLINE_EXT_CONFIG payload containing a NUL byte. These match the implementation in HDFGroup/hdf5#6701.
Add version 3 of the filter pipeline object header message. Each filter entry carries the version 2 fields followed by a list of extension blocks: a 2-byte block count, then for each block a 2-byte type, a flags byte, a reserved byte, a 4-byte payload length and the payload. Bit 0 of the flags byte marks a block critical. Blocks are written in ascending type order and no type appears twice in an entry. Block type 0x0001 (H5O_PLINE_EXT_CONFIG, not critical) holds the filter's canonical configuration string without a NUL terminator; an entry without a string has no blocks. Type 0x0002 is reserved for a critical block that a later change defines. The decoder skips a non-critical block of a type it does not know and fails on a critical one, naming the type and the filter ID. It ignores the reserved byte and flag bits 1-7 of an unknown type, and rejects a configuration string block that sets a flag bit it does not define. It also rejects a repeated block type, a configuration string longer than H5Z_CONFIG_STRING_MAX or holding a NUL byte, and a block count, block header or block length that runs past the end of the message. A zero-length configuration string block is read as no string. H5O_PLINE_VERSION_LATEST becomes 3 and the H5F_LIBVER_V300 row of H5O_pline_ver_bounds admits version 3. H5O_pline_set_version() selects version 3 when some entry carries a string and the file's high bound admits it. When the high bound is below H5F_LIBVER_V300, it fails with H5E_BADRANGE rather than writing the pipeline without the string, so H5Dcreate() and H5Dcreate_anon() fail; H5Ocopy() into such a file already fails through the existing version check in H5O__pline_pre_copy_file(). Pipelines without strings keep the existing version selection, so files that store no strings are unchanged when the low bound is below H5F_LIBVER_V300, while a low bound of H5F_LIBVER_V300 or H5F_LIBVER_LATEST writes version 3 messages with empty block lists. The pipeline message copy callback carries the string, so H5Ocopy() preserves it; blocks of unknown types are not kept in memory and are not copied. H5Pget_filter_params_by_idx() returns the stored string for a dataset opened from a file without loading the filter. The H5Pappend_filter(), H5Pget_filter_params_by_idx() and get_config documentation describe the persistence and the library version bound check, and the file format specification gains the version 3 filter description and extension block layouts. gen_filters creates test/testfiles/test_filters_v3.h5, a checked-in scale-offset dataset with a stored string, without object timestamps so that regenerating it gives the same bytes. tfilter2 gains tests for the on-disk round-trip without the plugin, H5Ocopy, H5Dcreate, H5Dcreate_anon and H5Ocopy failing with H5E_BADRANGE below H5F_LIBVER_V300, a string-free version 3 message, the golden file, and a pipeline mixing class2 and class3 filters. Two tests replace a dataset's block list with hand-built lists of the same size (on a version 1 object header, so the patched bytes reach the pipeline decoder instead of an object header checksum check): unknown non-critical blocks before or after the configuration string are skipped, the reserved byte and unknown-type flag bits are ignored, unknown critical blocks (including type 0x0002) fail the open while a sibling dataset still opens, and oversized, overrunning, over-counted, duplicated, NUL-holding and unknown-flag configuration blocks are rejected. The reopen checks in the set_local, canonicalization, H5Pmodify_filter_by_idx and property-list tests now expect the stored string. t_filters_parallel gains string-configuration tests that check the stored string after a parallel create and reopen.
brtnfld
force-pushed
the
filter-config/05-pline-v3
branch
from
October 5, 2026 08:19
70b7f8a to
a6b230c
Compare
brtnfld
force-pushed
the
filter-config/04-dcpl-api
branch
from
October 5, 2026 08:19
85d8b8a to
2f91677
Compare
| local_err = 1; | ||
|
|
||
| /* Collective reduce; a deadlocked rank would never reach this point */ | ||
| mpi_code = MPI_Allreduce(&local_err, &global_err, 1, MPI_INT, MPI_SUM, comm); |
Collaborator
There was a problem hiding this comment.
It's nitpicking a bit, but this test is really only testing MPI considering that H5Pappend_filter() performs no particular rank synchronization or collective error checking (similar to pretty much any other API function) and the usual, naive way of calling it on all ranks could still deadlock if a particular rank fails and others don't.
|
|
||
| local_level = cd_values[0]; | ||
| rank0_level = local_level; | ||
| mpi_code = MPI_Bcast(&rank0_level, 1, MPI_UNSIGNED, 0, comm); |
Collaborator
There was a problem hiding this comment.
This test also seems like it's just testing MPI and not really anything about HDF5
3 tasks done
brtnfld
changed the title
mixv3_set_config takes const H5Z_config_t *. Hex-floats are no longer accepted, so the plugin-free persistence test uses a decimal value and the format spec no longer describes a hex-float rewrite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Part of the string-based filter configuration work, RFC-HDFG-2026-001 (#6153), split into stacked PRs that merge into the integration branch
feature/filter-config-3.0. That branch merges into develop as a unit once the stack is complete, which moves develop to 3.0; nothing here targets the 2.3 release. Review only the top commit; the base branch is the previous PR in the stack.Adds version 3 of the filter pipeline object header message. Each filter entry carries the version 2 fields followed by an extension-block list: a 2-byte block count, then per block a 2-byte type, a flags byte (bit 0 = critical), a reserved byte and a 4-byte length, followed by the payload. This PR defines one block type,
H5O_PLINE_EXT_CONFIG(0x0001, non-critical), holding the canonical configuration string without a NUL; 0x0002 is reserved for the critical blob-reference block of RFC-HDFG-2026-003, which targets the same 3.0 format. Readers skip unknown non-critical blocks and fail on unknown critical ones, accept blocks in any order, ignore the reserved byte and unknown types' flag bits, and reject duplicate types, config strings overH5Z_CONFIG_STRING_MAXor containing a NUL, and any length that runs past the message.H5O_PLINE_VERSION_LATESTbecomes 3 and theH5F_LIBVER_V300row admits it. The format spec gains the layout, andtest_filters_v3.h5is regenerated with deterministic timestamps.H5Ocopy()preserves the string, andH5Pget_filter_params_by_idx()returns it for a reopened dataset without loading the filter.test/testfiles/test_filters_v3.h5(fromgen_filters) is a checked-in golden file.High bound below V300. With strings present and the high bound below V300,
H5Dcreate(),H5Dcreate_anon()andH5Ocopy()fail withH5E_BADRANGE; the remedy is to raise the bound or append the filter withH5Z_PARAMS_CDVALUES. This matches every other libver-gated feature (datatypes including complex, object header v2, layouts, chunks over 4 GB, dataspace, attribute and fill messages), and it never silently drops the string that plugin-free readers rely on. The RFC is being updated to match; it previously described a silent downgrade to version 2.Tested locally: 1,423/1,423 (tfilter2, dsets, objcopy, objcopy_ref, ohdr, tmisc, links, testhdf5, enc_dec_plist, all h5dump/h5repack/plugin tests).
t_filters_parallel(new par-01..par-04 string-config tests) passes run directly with 2 ranks, but it is slow on a laptop and hit ctest's 1200 s timeout while other builds were running. Please watch the parallel CI job's time.Issue ticket number (GitHub or JIRA)
#6153
Checklist before requesting a review
Stack
Merges into
feature/filter-config-3.0, in order:Prerequisite, merging into
feature/filter-config-3.0on its own rather than as part of the stack: #6727 (H5F_LIBVER_V300bound and the 3.0.0 version bump).Related standalone fixes against develop: #6693, #6694, #6695, #6696.