Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Draft] Non-kerchunk backend for HDF5/netcdf4 files. #87

Open
wants to merge 94 commits into
base: main
Choose a base branch
from
Open

[Draft] Non-kerchunk backend for HDF5/netcdf4 files. #87

wants to merge 94 commits into from

Commits on Apr 19, 2024

  1. Generate chunk manifest backed variable from HDF5 dataset.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    6b7abe2 View commit details
    Browse the repository at this point in the history

  2. Transfer dataset attrs to variable.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    bca0aab View commit details
    Browse the repository at this point in the history

  3. Get virtual variables dict from HDF5 file.

    @sharkinsspatial
    sharkinsspatial committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    384ff6b View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Update virtual_vars_from_hdf to use fsspec and drop_variables arg.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4c5f9bd View commit details
    Browse the repository at this point in the history

  2. mypy fix to use ChunkKey and empty dimensions list.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1dd3370 View commit details
    Browse the repository at this point in the history

  3. Extract attributes from hdf5 root group.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d92c75c View commit details
    Browse the repository at this point in the history

  4. Use hdf reader for netcdf4 files.

    @sharkinsspatial
    sharkinsspatial committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    0ed8362 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    f4485fa View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed May 8, 2024
    Configuration menu
    Copy the full SHA
    3cc1254 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Fix ruff complaints.

    @sharkinsspatial
    sharkinsspatial committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0123df7 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. First steps for handling HDF5 filters.

    @sharkinsspatial
    sharkinsspatial committed May 10, 2024
    Configuration menu
    Copy the full SHA
    332bcaa View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Initial step for hdf5plugin supported codecs.

    @sharkinsspatial
    sharkinsspatial committed May 13, 2024
    Configuration menu
    Copy the full SHA
    c51e615 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Small commit to check compression support in CI environment.

    @sharkinsspatial
    sharkinsspatial committed May 16, 2024
    Configuration menu
    Copy the full SHA
    0083f77 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed May 18, 2024
    Configuration menu
    Copy the full SHA
    3c00071 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 19, 2024
    Configuration menu
    Copy the full SHA
    207c4b5 View commit details
    Browse the repository at this point in the history

  2. Fix mypy complaints for hdf_filters.

    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    c573800 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'hdf5_reader' of https://github.com/TomNicholas/Virtuali… 

    …Zarr into hdf5_reader
    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    ef0d7a8 View commit details
    Browse the repository at this point in the history

  4. Local pre-commit fix for hdf_filters.

    @sharkinsspatial
    sharkinsspatial committed May 19, 2024
    Configuration menu
    Copy the full SHA
    588e06b View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Use fsspec reader_options introduced in #37.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    725333e View commit details
    Browse the repository at this point in the history

  2. Fix incorrect zarr_v3 if block position from merge commit ef0d7a8.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    72df108 View commit details
    Browse the repository at this point in the history

  3. Fix early return from hdf _extract_attrs.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    d1e85cb View commit details
    Browse the repository at this point in the history

  4. Test that _extract_attrs correctly handles multiple attributes.

    @sharkinsspatial
    sharkinsspatial committed May 21, 2024
    Configuration menu
    Copy the full SHA
    1e2b343 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. Initial attempt at scale and offset via numcodecs.

    @sharkinsspatial
    sharkinsspatial committed May 22, 2024
    Configuration menu
    Copy the full SHA
    7f1c189 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Tests for cfcodec_from_dataset.

    @sharkinsspatial
    sharkinsspatial committed May 23, 2024
    Configuration menu
    Copy the full SHA
    908e332 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Temporarily relax integration tests to assert_allclose.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    0df332d View commit details
    Browse the repository at this point in the history

  2. Add blosc_lz4 fixture parameterization to confirm libnetcdf environment.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ca6b236 View commit details
    Browse the repository at this point in the history

  3. Check for compatability with netcdf4 engine.

    @sharkinsspatial
    sharkinsspatial committed May 24, 2024
    Configuration menu
    Copy the full SHA
    b7426c5 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Use separate fixtures for h5netcdf and netcdf4 compression styles.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    dac21dd View commit details
    Browse the repository at this point in the history

  2. Print libhdf5 and libnetcdf4 versions to confirm compiled environment.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e968772 View commit details
    Browse the repository at this point in the history

  3. Skip netcdf4 style compression tests when libhdf5 < 1.14.

    @sharkinsspatial
    sharkinsspatial committed May 27, 2024
    Configuration menu
    Copy the full SHA
    9a98e57 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Include imagecodecs.numcodecs to support HDF5 lzf filters.

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    7590b87 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    e9fbc8a View commit details
    Browse the repository at this point in the history

  3. Remove test that verifies call to read_kerchunk_references_from_file.

    @sharkinsspatial
    sharkinsspatial committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    14bd709 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Add additional codec support structures for imagecodecs and numcodecs.

    @sharkinsspatial
    sharkinsspatial committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    acdf0d7 View commit details
    Browse the repository at this point in the history

  2. Add codec config test for Zstd.

    @sharkinsspatial
    sharkinsspatial committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4ba323a View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Include initial cf decoding tests.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    e14e53b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b808ded View commit details
    Browse the repository at this point in the history

  3. Revert typo for scale_factor retrieval.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b052f8c View commit details
    Browse the repository at this point in the history

  4. Update reader to use new numpy manifest representation.

    @sharkinsspatial
    sharkinsspatial committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    01a3980 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. Temporarily skip test until blosc netcdf4 issue is solved.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    c37d9e5 View commit details
    Browse the repository at this point in the history

  2. Fix Pydantic 2 migration warnings.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    17b30d4 View commit details
    Browse the repository at this point in the history

  3. Include hdf5plugin and imagecodecs-numcodecs in mamba test environment.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    f6b596a View commit details
    Browse the repository at this point in the history

  4. Mamba attempt with imagecodecs rather than imagecodecs-numcodecs.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    eb6e24d View commit details
    Browse the repository at this point in the history

  5. Mamba attempt with latest imagecodecs release.

    @sharkinsspatial
    sharkinsspatial committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    c85bd16 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Use correct iter_chunks callback function signtature.

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ca435da View commit details
    Browse the repository at this point in the history

  2. Include pip based imagecodecs-numcodecs until conda-forge availability.

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    3017951 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ccf0b73 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Handle non-coordinate dims which are serialized to hdf as empty dataset.

    @sharkinsspatial
    sharkinsspatial committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    32ba135 View commit details
    Browse the repository at this point in the history

  2. Use reader_options for filetype check and update failing kerchunk call.

    @sharkinsspatial
    sharkinsspatial committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    64f446c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial authored Jun 27, 2024
    Configuration menu
    Copy the full SHA
    1c590bb View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2024

  1. Fix chunkmanifest shaping for chunked datasets.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    9797346 View commit details
    Browse the repository at this point in the history

  2. Handle scale_factor attribute serialization for compressed files.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    c833e19 View commit details
    Browse the repository at this point in the history

  3. Include chunked roundtrip fixture.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    701bcfa View commit details
    Browse the repository at this point in the history

  4. Standardize xarray integration tests for hdf filters.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    08c988e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'hdf5_reader' of https://github.com/TomNicholas/Virtuali… 

    …Zarr into hdf5_reader
    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    e6076bd View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d684a84 View commit details
    Browse the repository at this point in the history

  7. Update reader selection logic for new filetype determination.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    4cb4bac View commit details
    Browse the repository at this point in the history

  8. Use decode_times for integration test.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d352104 View commit details
    Browse the repository at this point in the history

  9. Standardize fixture names for hdf5 vs netcdf4 file types.

    @sharkinsspatial
    sharkinsspatial committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    3d89ea4 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Handle array add_offset property for compressed data.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    c9dd0d9 View commit details
    Browse the repository at this point in the history

  2. Include h5py shuffle filter.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    db5b421 View commit details
    Browse the repository at this point in the history

  3. Make ScaleAndOffset codec last in filters list.

    @sharkinsspatial
    sharkinsspatial committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    9a1da32 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Apply ScaleAndOffset codec to _FillValue since it's value is now down… 

    …stream.
    @sharkinsspatial
    sharkinsspatial committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9b2b0f8 View commit details
    Browse the repository at this point in the history

  2. Coerce scale and add_offset values to native float for JSON serializa… 

    …tion.
    @sharkinsspatial
    sharkinsspatial committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    9ef1362 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    30005bd View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    14f7a99 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Temporarily xfail integration tests for main

    @sharkinsspatial
    sharkinsspatial committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    f4f9c8f View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    d257cb9 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    e795c2c View commit details
    Browse the repository at this point in the history

  2. Remove pydantic dependency as per pull/210.

    @sharkinsspatial
    sharkinsspatial committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    a9e59f2 View commit details
    Browse the repository at this point in the history

  3. Update test for new kerchunk reader module location.

    @sharkinsspatial
    sharkinsspatial committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    2b33bc2 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Fix branch typing errors.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    a57ae9e View commit details
    Browse the repository at this point in the history

  2. Re-include automatic file type determination.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    e21fc69 View commit details
    Browse the repository at this point in the history

  3. Handle various hdf flavors of _FillValue storage.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    df69a12 View commit details
    Browse the repository at this point in the history

  4. Include loadable variables in drop variables list.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    169337c View commit details
    Browse the repository at this point in the history

  5. Mock readers.hdf.virtual_vars_from_hdf to verify option passing.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    bdcbfbf View commit details
    Browse the repository at this point in the history

  6. Convert numpy _FillValue to native Python for serialization support.

    @sharkinsspatial
    sharkinsspatial committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    77f1689 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Support groups with HDF5 reader.

    @sharkinsspatial
    sharkinsspatial committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    42c653a View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Handle empty variables with a shape.

    @sharkinsspatial
    sharkinsspatial committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9c86e0d View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    001a4a7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hdf5_reader

    @sharkinsspatial
    sharkinsspatial committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    79f9921 View commit details
    Browse the repository at this point in the history

  3. Import top-level version of xarray classes.

    @sharkinsspatial
    sharkinsspatial committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    1589776 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Add option to explicitly specify use of an experimental hdf backend.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    772c580 View commit details
    Browse the repository at this point in the history

  2. Include imagecodecs and hdf5plugin in all CI environments.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    3ab90c6 View commit details
    Browse the repository at this point in the history

  3. Add test_hdf_integration tests to be skipped for non-kerchunk env.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    150d06d View commit details
    Browse the repository at this point in the history

  4. Include imagecodecs in dependencies.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8ccba34 View commit details
    Browse the repository at this point in the history

  5. Diagnose imagecodecs-numcodecs installation failures in CI.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    81874e0 View commit details
    Browse the repository at this point in the history

  6. Ignore mypy complaints for VirtualBackend.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f87abe2 View commit details
    Browse the repository at this point in the history

  7. Remove checksum assert which varies across different zstd versions.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    70e7e29 View commit details
    Browse the repository at this point in the history

  8. Temporarily xfail integration tests with coordinate inconsistency.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    43bc0e4 View commit details
    Browse the repository at this point in the history

  9. Remove backend arg for non-hdf network file tests.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    82a6321 View commit details
    Browse the repository at this point in the history

  10. Fix mypy comment moved by ruff formatting.

    @sharkinsspatial
    sharkinsspatial committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b34f260 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Make HDR reader dependencies optional.

    @sharkinsspatial
    sharkinsspatial committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    f9ead06 View commit details
    Browse the repository at this point in the history

  2. Handle optional imagecodecs and hdf5plugin dependency imports for tests.

    @sharkinsspatial
    sharkinsspatial committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    5608292 View commit details
    Browse the repository at this point in the history