File Chunk Store

[ajelenak]

Hello!

I want to propose adding a new Zarr store type when all array chunks are located in a single binary file. A propotype implementation, named file chunk store, is described in this Medium post. In this approach, Zarr metadata (.zgroup, .zarray, .zattrs, or .zmetadata) are stored in one of the current Zarr store types while the array chunks are in a binary file. The file chunk store translates array chunk keys into file seek and read operations and therefore only provides read access to the chunk data.

The file chunk store requires a mapping between array chunk keys and their file locations. The prototype implementation put this information for every Zarr array in JSON files named .zchunkstore. An example is below:

   {
    "BEAM0001/tx_pulseflag/0": {
        "offset": 94854560,
        "size": 120
    },
    "BEAM0001/tx_pulseflag/1": {
        "offset": 94854680,
        "size": 120
    },
    "BEAM0001/tx_pulseflag/2": {
        "offset": 94854800,
        "size": 120
    },
    "BEAM0001/tx_pulseflag/3": {
        "offset": 94854920,
        "size": 120
    },
    "BEAM0001/tx_pulseflag/4": {
        "offset": 96634038,
        "size": 120
    },
    "BEAM0001/tx_pulseflag/5": {
        "offset": 96634158,
        "size": 123
    },
    "source": {
        "array_name": "/BEAM0001/tx_pulseflag",
        "uri": "https://e4ftl01.cr.usgs.gov/GEDI/GEDI01_B.001/2019.05.26/GEDI01_B_2019146164739_O02560_T04067_02_003_01.h5"
    }
}

Array chunk file location is described with the starting byte (offset) in the file and the number of bytes to read (size). Also included is the file information (source) to enable verification of chunk data provenance. The file chunk store prototype uses file-like Python objects, delegating to users the responsibility to arrange access to correct files.

We can discuss specific implementation details If there is enough interest in this new store type.

Thanks!

[alimanfoo]

Hi @ajelenak, sorry for slow response here. Would this be exactly the implementation that @rsignell-usgs described in the medium blog post? I.e., this would enable reading of HDF5 files (as well as other file types storing chunks)?

[ajelenak]

Hi, thanks for looking into this.

Yes, the approach is the same as in the Medium post. Actual implementation details are open for discussion. Producing mappings between Zarr chunk keys and their file locations is not included since it differs based on the source file format.

[alimanfoo]

Is this something you'd like to be able to layer over different types of storage? I.e., would you like to be able to use this over cloud object stores, as well as possibly local files?

[ajelenak]

Yes. The file chunk store takes a file-like object so is flexible with the file's actual storage system. I used it on local files with objects from the open() function.

[rsignell-usgs]

@alimanfoo, I'm wondering what folks think about this proposal. I'd like to get the conversation going again now that xarray has merged pydata/xarray#3804

[rabernat]

Together with pydata/xarray#3804, the idea proposed here could unlock an amazing capability: accessing a big HDF5 file using Zarr very efficiently. I think it's important to find a way to move forward with it.

[manzt]

+1 here! We have been generating a complimentary offsets.json file for retrieving tiles from OME-TIFF images on the web. Having something like this could make reading open image formats as Zarr much easier.

[jakirkham]

I'm curious if people here have tried existing single file stores like ZipStore, DBMStore, LMDBStore or SQLiteStore? Should add these probably contain optimizations in the underlying formats that we may not have considered or would miss out on (at least initially) when implementing something new.

[pbranson]

I have used ZipStore quite extensively to assist in packaging (reducing inodes) and archival storage of DirectoryStores on HPC. I am interested to try a ZipStore on a http storage system (like S3) using fsspec, but have yet to try it, would be interested to know if others have tried that.

However, I think the salient point here is that there is already a significant latent investment in netCDF/HDF that is stored in cloud storage and this unlocks significant performance gains and to a degree software stack simplification if these can be accessed via Zarr store without needing to invest considerable overhead to convert the data format.

[manzt]

I'm curious if people here have tried existing single file stores like ZipStore, DBMStore, LMDBStore or SQLiteStore?

I've experimented with ZipStore, but reading a ZipStore remotely via HTTP is not very performant and not well supported. Traversing the central directory at the end of the zip file to find the chunk byte offsets takes a long time (and many requests) for large stores, making the DirectoryStore most ideal for archival storage.

As a side note, I wrote a small python package to serve the underlying store for any zarr-python zarr.Array or zarr.Group over HTTP (simple-zarr-server). It works by mapping HTTP requests to the underlying store.__getitem__ and store.__setitem___, making any store accessible by a python client with fsspec.HTTPFileSystem. Not ideal for archival storage, again, but at least a way access non-DirectoryStore remotely via fsspec.

However, I think the salient point here is that there is already a significant latent investment in netCDF/HDF that is stored in cloud storage and this unlocks significant performance gains and to a degree software stack simplification if these can be accessed via Zarr store without needing to invest considerable overhead to convert the data format.

Agreed. Something like a "File Chunk Store" might offer a more standardized way to read other tiled/chunked formats without requiring conversion (despite not being as performant as the built-in stores).

[joshmoore]

#556 (comment) We have been generating a complimentary offsets.json file for retrieving tiles from OME-TIFF images on the web. Having something like this could make reading open image formats as Zarr much easier.

@manzt, you haven't tried writing a Zarr store implementation in front of OME-TIFF yet have you? That might allow unifying versions of the offset file.

[manzt]

@joshmoore I haven't found the time yet but was planning to investigate ... I experimented with a store implementation for OME-TIFF but it just used tifffile. We use the offsets file outside of zarr at the moment with geotiff.js, so it doesn't have a mapping of zarr-specfiic keys.

The thing with a "File Chunk Store" is that it would be much easier to enable cross-language support for many different types of files. Rather than requiring an extra runtime dependency (in each zarr implementation) to parse the file format and find chunk offsets, this can be done prior in whatever language the offsets file was created.

[manzt]

@joshmoore

TL;DR: An offsets file enables the explicit retrofitting of non-Zarr, array-like data as Zarr and simplifies accessing remotes single file stores via byte-range requests.

Ok, so I experimented with this today, unifying a version of the "offsets" file for both a zarr.ZipStore and a multiscale OME-TIFF image. It seems a "File Chunk Store" could be implemented just as some type extension for Zarr, where an offsets file is generated to complement some archival/read-only store.

Details about the actual file format are necessary for updating (writing) chunks, which is why read-only, but having the offsets in a separate file makes accessing the bytes remotely straight-forward (e.g. using HTTP range requests).

ZipStore: https://observablehq.com/@manzt/zarr-js-file-chunk-store
OME-TIFF: https://observablehq.com/@manzt/ome-tiff-file-chunk-store

For the zarr.ZipStore example, it was just a matter of traversing the underlying ZipFile directory and gathering the byte offsets and sizes for all the data:

ZipStore "file chunk store" metadata

def get_offsets(zstore):
    offsets = {}
    # Traverse the file directory and compute the offsets and chunks
    for i in zstore.zf.infolist():
        # File contents in zip file start 30 + n bytes after the file
        # header offset, where n is the number of bytes of the filename.
        name_bytes = len(i.filename.encode("utf-8"))
        offsets[i.filename] = dict(
            offset=i.header_offset + 30 + name_bytes,
            size=i.compress_size,
        )
    return offsets

{
  ".zarray": {
    "offset": 37,
    "size": 361
  },
  "0.0.0": {
    "offset": 433,
    "size": 89446
  },
  "0.1.0": {
    "offset": 89914,
    "size": 100028
  },
  "1.0.0": {
    "offset": 189977,
    "size": 88337
  },
  "1.1.0": {
    "offset": 278349,
    "size": 87346
  }
}

For the OME-TIFF, it was a bit more involved to generate the "offsets" file. The key was mapping compressed byte-ranges (chunks) in the OME-TIFF to the same schema, and filling in .zarray/.zattrs/.zgroup metadata where necessary. Hence, the offsets files were the same structure in both cases, except in the OME-TIFF the metadata keys contain the actual JSON metadata rather than offset and size.

OME-TIFF "file chunk store" metadata

{
  ".zgroup": {
    "zarr_format": 2
  },
  "0/.zarray": {
    "chunks": [
      1,
      1024,
      1024
    ],
    "compressor": {
      "id": "zlib",
      "level": 8
    },
    "dtype": "|u1",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
      5,
      34560,
      24960
    ],
    "zarr_format": 2
  },
  "0/0.0.0": {
    "offset": 1055,
    "size": 1039
  },
  "1/0.0.0": {
    "offset": 1452747820,
    "size": 9542
  ...
}