Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 221 additions & 1 deletion docs/doxygen/dox/H5.format.4.0.dox
Original file line number Diff line number Diff line change
Expand Up @@ -9222,7 +9222,7 @@ u
</table>

<table>
<caption align="top"><strong>Fields: Filter Description</strong></caption>
<caption align="top">\anchor FMT4FilterDescV2Fields <strong>Fields: Filter Description</strong></caption>
<tr>
<th width="30%">Field Name</th>
<th>Description</th>
Expand Down Expand Up @@ -9335,6 +9335,226 @@ u
</tr>
</table>

<table>
<caption align="top"><strong>Layout: Filter Pipeline Message - Version 3</strong></caption>
<tr align="center">
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th>byte</th>
</tr>
<tr align="center">
<td>Version</td>
<td>Number of Filters</td>
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
</tr>
<tr align="center">
<td colspan="4"><br />Filter Description List <em>(variable size)</em><br /><br /></td>
</tr>
</table>

<table>
<caption align="top"><strong>Fields: Filter Pipeline Message - Version 3</strong></caption>
<tr>
<th width="30%">Field Name</th>
<th>Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>The version number for this message. This table describes version 3. The library writes version 3
when at least one filter in the pipeline carries an extension block (see the Filter Description and
Extension Block tables below; the library writes a block only for a stored configuration string)
and the file's latest format version bound (set with <code>H5Pset_libver_bounds()</code>) is
<code>H5F_LIBVER_V300</code> or later; if that bound is lower, creating the dataset or copying the
object fails rather than writing the pipeline without its blocks. A pipeline with no extension blocks
is written as version 1 or 2 under the usual rules, byte-identical to earlier releases, unless the file's earliest format version bound is
<code>H5F_LIBVER_V300</code> or later, in which case it is written as version 3 with every
<em>Number of Extension Blocks</em> zero.</td>
</tr>
<tr valign=top>
<td>Number of Filters</td>
<td>The total number of filters described in this message. The maximum possible number of filters in a
message is 32.</td>
</tr>
<tr valign=top>
<td>Filter Description List</td>
<td>A description of each filter. A filter description appears in the next table.</td>
</tr>
</table>

<table>
<caption align="top"><strong>Layout: Filter Description - Version 3</strong></caption>
<tr align="center">
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th>byte</th>
</tr>
<tr align="center">
<td colspan="2">Filter Identification</td>
<td colspan="2">Name Length <em>(optional)</em></td>
</tr>
<tr align="center">
<td colspan="2">Flags</td>
<td colspan="2">Number Client Data Values</td>
</tr>
<tr align="center">
<td colspan="4"><br />Name <em>(variable size, optional)</em><br /><br /></td>
</tr>
<tr align="center">
<td colspan="4"><br />Client Data <em>(variable size, optional)</em><br /><br /></td>
</tr>
<tr align="center">
<td colspan="2">Number of Extension Blocks</td>
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
</tr>
<tr align="center">
<td colspan="4"><br />Extension Block List <em>(variable size, optional)</em><br /><br /></td>
</tr>
</table>

<table>
<caption align="top"><strong>Fields: Filter Description - Version 3</strong></caption>
<tr>
<th width="30%">Field Name</th>
<th>Description</th>
</tr>
<tr valign=top>
<td>Filter Identification, Name Length, Flags, Number of Client Data Values, Name, Client Data</td>
<td>Identical in encoding and semantics to the
\ref FMT4FilterDescV2Fields "version 2 Filter Description fields".</td>
</tr>
<tr valign=top>
<td>Number of Extension Blocks</td>
<td>A two-byte unsigned integer giving the number of extension blocks in the <em>Extension Block
List</em> that follows. This field is present for every filter description in a version 3 message,
unlike <em>Name Length</em>, which may be omitted for internal filters. A value of zero means the
filter has no extension blocks, and the description is a version 2 description followed by these two
bytes.</td>
</tr>
<tr valign=top>
<td>Extension Block List</td>
<td>A sequence of <em>Number of Extension Blocks</em> extension blocks, laid out one after another with
no padding. Each block is described in the next table. Blocks are written in ascending order of
<em>Block Type</em>. A block type that is not defined as repeatable appears at most once in a
filter description, and no type defined so far is repeatable; a reader that finds a block type
more than once in one filter description rejects the message. A reader also rejects the message
when the blocks do not fit within it.</td>
</tr>
</table>

<table>
<caption align="top"><strong>Layout: Filter Pipeline Extension Block</strong></caption>
<tr align="center">
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th>byte</th>
</tr>
<tr align="center">
<td colspan="2">Block Type</td>
<td>Block Flags</td>
<td>Reserved (zero)</td>
</tr>
<tr align="center">
<td colspan="4">Block Length</td>
</tr>
<tr align="center">
<td colspan="4"><br />Block Data <em>(variable size, optional)</em><br /><br /></td>
</tr>
</table>

<table>
<caption align="top"><strong>Fields: Filter Pipeline Extension Block</strong></caption>
<tr>
<th width="30%">Field Name</th>
<th>Description</th>
</tr>
<tr valign=top>
<td>Block Type</td>
<td>A two-byte unsigned integer identifying the kind of data in <em>Block Data</em>. Type codes are
assigned by The HDF Group; there is no private or experimental range. The types defined so far are:
<table>
<tr>
<th width="15%">Type</th>
<th width="15%">Critical</th>
<th align="left">Description</th>
</tr>
<tr>
<td align="center"><code>0x0000</code></td>
<td align="center">N/A</td>
<td>Reserved</td>
</tr>
<tr>
<td align="center"><code>0x0001</code></td>
<td align="center">No</td>
<td>Configuration string (see below)</td>
</tr>
<tr>
<td align="center"><code>0x0002</code></td>
<td align="center">Yes</td>
<td>Reserved for a future configuration blob. Until that block is defined, a reader treats it as an
unrecognized critical block.</td>
</tr>
<tr>
<td align="center"><code>0x0003-0xFFFF</code></td>
<td align="center">N/A</td>
<td>Unassigned</td>
</tr>
</table></td>
</tr>
<tr valign=top>
<td>Block Flags</td>
<td>A one-byte bit field. The bit values defined so far are:
<table>
<tr>
<th width="20%">Bit</th>
<th align="left">Description</th>
</tr>
<tr>
<td align="center"><code>0</code></td>
<td>Critical. A reader that does not recognize <em>Block Type</em> skips a block with this bit clear,
using <em>Block Length</em>, and continues; the filter remains usable. A reader that does not
recognize <em>Block Type</em> in a block with this bit set fails to decode the message, since the
filter cannot be applied correctly without the block. A given block type is always written with
the same value of this bit.</td>
</tr>
<tr>
<td align="center"><code>1-7</code></td>
<td>Defined by each block type, and written as zero unless the type assigns them. None of the types
defined so far assigns them. A reader ignores these bits in a block whose type it does not
recognize, and rejects a block of a type it recognizes that sets a bit the type does not
assign.</td>
</tr>
</table></td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>Written as zero and ignored by readers.</td>
</tr>
<tr valign=top>
<td>Block Length</td>
<td>A four-byte unsigned integer giving the size, in bytes, of <em>Block Data</em>. A reader rejects a
block whose header or data runs past the end of the message.</td>
</tr>
<tr valign=top>
<td>Block Data</td>
<td>The block's payload, exactly <em>Block Length</em> bytes, interpreted according to <em>Block
Type</em>.<br />
For a configuration string block (type <code>0x0001</code>), the payload is the caller's filter
parameter string in canonical form: the contents of a
<a href="https://toml.io/en/v1.0.0">TOML v1.0.0</a> inline table (comma-separated
<code>key = value</code> pairs) with any optional outer braces and surrounding whitespace stripped,
so that a reader implementing only standard TOML v1.0.0 parsing &mdash; not just the HDF5 library
&mdash; can recover the exact values by enclosing the bytes in braces. The string is UTF-8 and is <strong>not</strong> null-terminated; its length is given entirely
by <em>Block Length</em>, which the library limits to 4096 (<code>H5Z_CONFIG_STRING_MAX</code>) on
write and on read. A configuration string block is written only for a filter that has a stored
string, and at most once in a filter description. This
block allows a filter's configuration to be recovered by a program that has not loaded the filter
plugin: see <code>H5Pget_filter_params_by_idx()</code> in the C API reference.</td>
</tr>
</table>

\subsubsection subsubsec_fmt4_dataobject_hdr_msg_attribute IV.A.3.m. The Attribute Message
<table>
<tr>
Expand Down
10 changes: 8 additions & 2 deletions release_docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ We would like to thank the many HDF5 community members who contributed to this r

### Added the H5F_LIBVER_V300 library version bound

The `H5F_libver_t` enumeration gains `H5F_LIBVER_V300` for the 3.0 file format, and `H5F_LIBVER_LATEST` now maps to it. Every object header message version admitted by `H5F_LIBVER_V300` is currently the same as for `H5F_LIBVER_V200`; later format changes in the 3.0 release will be gated on it. The constant is also available in the Fortran (`H5F_LIBVER_V300_F`) and Java (`HDF5Constants.H5F_LIBVER_V300`) bindings, and `h5repack --low`/`--high` accept the value 6.
The `H5F_libver_t` enumeration gains `H5F_LIBVER_V300` for the 3.0 file format, and `H5F_LIBVER_LATEST` now maps to it. Every object header message version admitted by `H5F_LIBVER_V300` is the same as for `H5F_LIBVER_V200` except the filter pipeline message, which admits version 3 (see below); later format changes in the 3.0 release will also be gated on it. The constant is also available in the Fortran (`H5F_LIBVER_V300_F`) and Java (`HDF5Constants.H5F_LIBVER_V300`) bindings, and `h5repack --low`/`--high` accept the value 6.

### Added the H5Z_class3_t filter class and H5Zget_filter_class_info()

Expand All @@ -122,12 +122,18 @@ We would like to thank the many HDF5 community members who contributed to this r

The new `H5Pappend_filter()` function appends a filter to a pipeline from an `H5Z_params_t` descriptor, declared in `H5Zpublic.h`. With `H5Z_PARAMS_RAW(n, cd_values)` it behaves like `H5Pset_filter()`; with `H5Z_PARAMS_STR("level = 6")` the library loads the filter if necessary and calls its `set_config` callback to translate the `key = value` string into `cd_values`. The string is canonicalized (outer braces and surrounding whitespace stripped) and stored on the pipeline entry together with the resulting `cd_values`, and the filter's canonical name is recorded in the entry. `H5Pmodify_filter_by_idx()` replaces the configuration of the filter at a given pipeline index, in either form, without changing the filter order. `H5Pget_filter_params_by_idx()` returns the parameter string of the filter at a given index: the stored string if there is one, otherwise the filter's `get_config` reconstruction from `cd_values`, otherwise a `cd_values=v0:v1:...` listing. `H5Pmodify_filter()` clears a stored string, since the new `cd_values` no longer match it.

Stored strings are carried by `H5Pcopy()` and by `H5Pencode()`/`H5Pdecode()`, but are not yet written to files: a dataset created from such a property list stores only its `cd_values`, and the creation property list of a dataset opened from a file reports the `get_config` or `cd_values` form.
Stored strings are carried by `H5Pcopy()` and by `H5Pencode()`/`H5Pdecode()`, and are written to files in version 3 of the filter pipeline message (see below), so the creation property list of a dataset opened from a file returns the stored string.

The built-in filters now implement `set_config`: deflate accepts `level` (0-9, default 6), szip accepts `coding` (`"nn"` or `"entropy"`) and `pixels_per_block`, and scale-offset requires `scale_type` (`"int"`, `"float_dscale"` or `"float_escale"`) and `scale_factor`. Shuffle, Fletcher32 and N-bit accept only an empty string. Deflate, szip and scale-offset also implement `get_config`.

An `H5Pencode()` buffer for a dataset creation property list in which some filter carries a parameter string uses an extended encoding of the filter pipeline property that earlier releases reject when decoding. Property lists without parameter strings encode exactly as before, and buffers produced by earlier releases decode as before.

### Added version 3 of the filter pipeline message

Version 3 of the filter pipeline object header message ends each filter description with a list of extension blocks, each with a 2-byte type, a flags byte whose low bit marks the block critical, a reserved byte, a 4-byte length and the payload. A reader skips a non-critical block of a type it does not recognize and fails to decode the message on an unrecognized critical block, so later additions to a filter description can be new block types rather than new message versions. Block type 0x0001 carries the filter's configuration string without a NUL terminator, limited to `H5Z_CONFIG_STRING_MAX` bytes; a filter with no string has no blocks. The library writes version 3 for a dataset whose pipeline carries at least one configuration string when the file's high library version bound is `H5F_LIBVER_V300` or later. If the high bound is lower, `H5Dcreate()`, `H5Dcreate_anon()` and `H5Ocopy()` fail with a minor error code of `H5E_BADRANGE` rather than write the pipeline without its strings. `H5Ocopy()` carries the strings to the copy, and `H5Pget_filter_params_by_idx()` returns them for a dataset opened from a file without loading the filter.

Pipelines without configuration strings are written as version 1 or 2 as before, so a file that stores no strings is byte-identical to one written by earlier releases when the low bound is below `H5F_LIBVER_V300`. Because `H5F_LIBVER_V300` and `H5F_LIBVER_LATEST` admit version 3, a low bound of either writes every filter pipeline message as version 3, even one without strings, and HDF5 2.x and earlier releases cannot open the datasets in such a file. The version 3 layout is described in the file format specification.

### Added support for internally concurrent multithreaded reads of chunked datasets

Added 3 new functions to support this: H5TSset_internal_threads(),
Expand Down
Loading