h5repack: SIGSEGV in Nbit filter processing when repacking datasets

[Nievesjyl]

Describe the bug
Running h5repack on HDF5 files causes segmentation faults in the Nbit filter processing path during dataset copy. Two related PoC inputs trigger crashes in the same Nbit filter/chunk-read workflow: 1. One input reaches H5Z__nbit_decompress_one_byte() through H5Z__nbit_decompress() and H5Z__filter_nbit(), causing a SIGSEGV during Nbit decompression. 2. Another input crashes directly in H5Z__filter_nbit() with malformed filter parameters, including cd_values=0x0. Both crashes occur while h5repack is copying objects and reading dataset data through H5Dread() and do_copy_objects().

Steps to Reproduce:

PoC inputs:

OOB_Read.zip
OOB_Read_03.zip

Command lines:

./h5repack -l CHUNK=2x2 OOB_Read out.h5
./h5repack -l COMPA OOB_Read_03 out.h5

Expected behavior
h5repack should exit cleanly with an error message, rather than crashing with SIGSEGV during filter processing or decompression.

Platform (please complete the following information)

• HDF5 version:

  • Git branch: develop

  • Commit: 8a2fb79

• OS and version:

  • Ubuntu 22.04.5 LTS

• Compiler and version:

  • clang: Ubuntu clang version 14.0.0-1ubuntu1.1

  • gcc: gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0

• Build system and generator:

  • CMake 4.3.2

  • Generator: Unix Makefiles, if you used the default CMake generator

• Configure options:

  • Not explicitly recorded in this run.

  • If same as the previous report:

    -DHDF5_BUILD_TOOLS=ON -DHDF5_BUILD_EXAMPLES=OFF -DHDF5_BUILD_TESTING=OFF -DHDF5_BUILD_HL_LIB=ON -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo

• MPI library and version:

  • Parallel HDF5 not enabled; no MPI library used.

Additional context
GDB output excerpt for the first PoC:

Program received signal SIGSEGV, Segmentation fault.

#0 0x00000000009e978e in H5Z__nbit_decompress_one_byte (
data=,
data_offset=194736,
k=1,
begin_i=3,
end_i=0,
buffer=0x289de490 ":\252\263ʫ>",
j=0x7ffeccdd4cf0,
buf_len=0x7ffeccdd4d10,
p=0x7ffeccdd4c90,
datatype_len=)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Znbit.c:1044
#1 H5Z__nbit_decompress_one_atomic (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Znbit.c:1100
#2 0x00000000009e6eea in H5Z__nbit_decompress (
data=0xbf3cc0 "",
buffer=0x0,
parms=,
d_nelmts=)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Znbit.c:1291
#3 H5Z__filter_nbit (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Znbit.c:956
#4 0x00000000009e34f8 in H5Z_pipeline (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Z.c:1451
#5 0x0000000000a43d19 in H5D__chunk_lock (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Dchunk.c:4855
#6 0x0000000000a34cd2 in H5D__chunk_read (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Dchunk.c:3177
#12 H5Dread (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5D.c:1049
#13 0x0000000000435229 in do_copy_objects (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack_copy.c:1053
#14 0x0000000000431bb7 in copy_objects (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack_copy.c:326
#15 0x00000000004553fa in h5repack (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack.c:53

GDB output excerpt for the second PoC:

Program received signal SIGSEGV, Segmentation fault.

#0 H5Z__filter_nbit (
flags=flags@entry=2305,
cd_nelmts=0,
cd_nelmts@entry=140728831307008,
cd_values=cd_values@entry=0x0,
nbytes=nbytes@entry=31,
buf_size=0x7ffdfbffee60,
buf=0x7ffdfbffeef0)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Znbit.c:935
#1 0x00000000009e34f8 in H5Z_pipeline (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Z.c:1451
#2 0x0000000000a43d19 in H5D__chunk_lock (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Dchunk.c:4855
#3 0x0000000000a34cd2 in H5D__chunk_read (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Dchunk.c:3177
#4 0x0000000000523f70 in H5D__read (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5Dio.c:398
#9 H5Dread (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/src/H5D.c:1049
#10 0x0000000000435229 in do_copy_objects (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack_copy.c:1053
#11 0x0000000000431bb7 in copy_objects (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack_copy.c:326
#12 0x00000000004553fa in h5repack (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack.c:53
#13 0x000000000045ab07 in main (...)
at /root/SpecDrive/benchmarks/hdf5/hdf5/tools/src/h5repack/h5repack_main.c:10

[gheber]

OOB_Read is corrupt:

./tools/h5policy --continue-after-rejection ~/h5/OOB_Read
{
  "decision": "reject_corrupt",
  "profile": "untrusted_strict",
  "file": "/home/bob/h5/OOB_Read",
  "mapping_mode": "strict",
  "phase": "0-4",
  "analysis": {
    "complete": true,
    "walk_started": true,
    "walk_completed": true,
    "stop_reason": "",
    "continue_after_rejection": true,
    "findings_truncated": false
  },
  "boundary": {
    "libhdf5": false,
    "plugins": false,
    "decompression": false,
    "external_file_opens": false,
    "repair": false,
    "writes": false,
    "application_deserialization": false
  },
  "features": {
    "external_links": 0,
    "external_storage": 0,
    "vds": 0,
    "dynamic_filters": [],
    "decode_filters": [1, 5, 6, 4],
    "unknown_messages": 0,
    "max_rank_seen": 2,
    "max_logical_dataset_bytes_seen": 336
  },
  "metrics": {
    "metadata_bytes_seen": 66752,
    "object_count": 25,
    "attribute_count": 0,
    "object_header_chunk_count": 0,
    "chunk_index_refs": 88,
    "visited_object_headers": 25,
    "visited_continuations": 0,
    "visited_btree_refs": 28,
    "walk_operations": 8458
  },
  "findings": [
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 49600,
      "object": "/Deflate_float_data_be",
      "message": "dataset declares the deflate/gzip filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 69632,
      "object": "/Nbit_float_data_be",
      "message": "dataset declares the n-bit filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_FILTER_NELMTS_MISMATCH",
      "class": "corrupt",
      "has_location": true,
      "offset": 66264,
      "object": "/Nbit_float_data_le",
      "message": "n-bit/scale-offset filter element count does not match the chunk element count",
      "evidence": {
        "field": "filter.decode_element_count",
        "actual": 67108876,
        "expected": 12,
        "comparison": "equal"
      }
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 19168,
      "object": "/Scale_offset_char_data_be",
      "message": "dataset declares the scale-offset filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_FILTER_NELMTS_MISMATCH",
      "class": "corrupt",
      "has_location": true,
      "offset": 1672,
      "object": "/Scale_offset_float_data_le",
      "message": "n-bit/scale-offset filter element count does not match the chunk element count",
      "evidence": {
        "field": "filter.decode_element_count",
        "actual": 12,
        "expected": 1048588,
        "comparison": "equal"
      }
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_OFFSET_OUT_OF_FILE",
      "class": "corrupt",
      "has_location": true,
      "offset": 38272,
      "object": "/Scale_offset_long_long_data_be",
      "message": "chunk data outside file"
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 57424,
      "object": "/Szip_float_data_be",
      "message": "dataset declares the szip filter; a consumer must decode/decompress untrusted data to read it"
    }
  ]
}

[gheber]

OOB_Read_03 is corrupt:

./tools/h5policy --continue-after-rejection ~/h5/OOB_Read_03 
{
  "decision": "reject_corrupt",
  "profile": "untrusted_strict",
  "file": "/home/bob/h5/OOB_Read_03",
  "mapping_mode": "strict",
  "phase": "0-4",
  "analysis": {
    "complete": true,
    "walk_started": true,
    "walk_completed": true,
    "stop_reason": "",
    "continue_after_rejection": true,
    "findings_truncated": false
  },
  "boundary": {
    "libhdf5": false,
    "plugins": false,
    "decompression": false,
    "external_file_opens": false,
    "repair": false,
    "writes": false,
    "application_deserialization": false
  },
  "features": {
    "external_links": 0,
    "external_storage": 0,
    "vds": 0,
    "dynamic_filters": [],
    "decode_filters": [1, 5, 6, 4],
    "unknown_messages": 0,
    "max_rank_seen": 2,
    "max_logical_dataset_bytes_seen": 336
  },
  "metrics": {
    "metadata_bytes_seen": 66752,
    "object_count": 25,
    "attribute_count": 0,
    "object_header_chunk_count": 0,
    "chunk_index_refs": 88,
    "visited_object_headers": 25,
    "visited_continuations": 0,
    "visited_btree_refs": 28,
    "walk_operations": 8448
  },
  "findings": [
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 49600,
      "object": "/Deflate_float_data_be",
      "message": "dataset declares the deflate/gzip filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 69632,
      "object": "/Nbit_float_data_be",
      "message": "dataset declares the n-bit filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_NBIT_PARAMS_OVERRUN",
      "class": "corrupt",
      "has_location": true,
      "offset": 66416,
      "object": "/Nbit_float_data_le",
      "message": "n-bit filter parameter tree reads past the client-data array"
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 19168,
      "object": "/Scale_offset_char_data_be",
      "message": "dataset declares the scale-offset filter; a consumer must decode/decompress untrusted data to read it"
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_CHUNK_SIZE_EXCEEDS_32BIT",
      "class": "corrupt",
      "has_location": true,
      "offset": 34880,
      "object": "/Scale_offset_long_long_data_le",
      "message": "chunk byte size exceeds the legacy 32-bit chunk-size field",
      "evidence": {
        "field": "layout.chunk_size_overflow_product",
        "actual": 7545410000,
        "expected": 4294967295,
        "comparison": "less_than_or_equal"
      }
    },
    {
      "severity": "error",
      "code": "H5_CORRUPT_FILTER_NELMTS_MISMATCH",
      "class": "corrupt",
      "has_location": true,
      "offset": 34616,
      "object": "/Scale_offset_long_long_data_le",
      "message": "n-bit/scale-offset filter element count does not match the chunk element count",
      "evidence": {
        "field": "filter.decode_element_count",
        "actual": 12,
        "expected": 7545410000,
        "comparison": "equal"
      }
    },
    {
      "severity": "warning",
      "code": "H5_ADVISORY_DECODE_FILTER",
      "class": "warning",
      "has_location": true,
      "offset": 57424,
      "object": "/Szip_float_data_be",
      "message": "dataset declares the szip filter; a consumer must decode/decompress untrusted data to read it"
    }
  ]
}