Question: What is the difference between this and pyfilesystem?

[rdblue]

The description mentions pyfilesystem, but doesn't explain how this is different. What are the differences and why is a new filesystem API needed? It sounds like the other one might already have a community so working within that API (if possible) could make sense.

[martindurant]

I completely agree, @rdblue , that if the opinions I get from trying this are broadly along the lines of "hey, pyfilesystems is great, conform to that!", then indeed, the various file-systems I am involved with should be rewritten as plugins for pyfilesystems. That is certainly a possibility.

However, it feels to me that the file-systems I refer to (hdfs3, s3fs, gcsfs, arrow-hdfs, azure adlfs) have a different flavour that is more familiar and easier for pyhonic users. Maybe that more precisely reflects our users, i.e., selected for the people that happen to already use these libraries - maybe not a fair comparison.

On the other hand, the use case and usage patterns is somewhat different, oriented to fully-conformant and performant file-like object implementations with buffering, and remote-oriented operations. For example the existing s3 file implementation actually writes to a local temporary file rather than streaming to/from the target.

Furthermore, we are interested implementations of generic key-value mappers, FUSE mounting and other extra functionality that maybe could be bolted onto pyfilesystems, but it is not entirely clear to me how easy such things would be. Opinions welcome!

[willmcgugan]

Hello. PyFilesystem author here. Happy to brainstorm on this, no matter what the conclusion.

[martindurant]

@willmcgugan , do you have any comments on what I have here or on the layout of the linked file-systems, aside from whether they could fit under pyfilesystems?

The general design behind my stuff has been around interoperability with dask, which requires reading random bytes chunks from files in parallel, and finding of sets of files on remote servers where directory listing may be expensive. Other operations like the mutable-mapping and FUSE are nice extras (which pyfilesystems may like too).

[aparamon]

Hello Gents!

HDF5 is a data model, library, and file format for numerical scientific data. HDF5 format may be viewed as a file-system-in-a-file, with groups similar to FS directories and datasets to files. From that point of view, HDF5 is a good candidate for implementation of abstract file system interface.

However, HDF5 datasets are different from FS files because they are structured, and file-like interface is only sensible for a very specific and rarely used OPAQUE datasets. Still, having a common FS interface for group hierarchy seems attractive.

What do you think about it? Does it present a sensible goal to try implement HDF5 driver for PyFilesystem or filesystem_spec?

[martindurant]

@aparamon , there indeed many file-system like target that could be implemented, whether they be themselves within file container, or some other abstraction service. pyfilesystems2 does have some like that for, for example, zip-files.

To what extent HDF5 matches a file-system structure is debatable: one of the main points of HDF5 is not just the hierarchical structure, but the metadata attached to each leaf node, telling you about the structure and purpose or that blob and it's relation to other items in the same collection (e.g., the format encoding, appropriate coordinates, the scaling and units and other attributes).

Actually, this project is partly inspired by an attempt to do the inverse of what you are talking about: to use file-systems as a backing store for zarr, which is an alternative storage format for hdf/netCDF-like array data and also allows for hierarchies and attributes in the same way. Zarr can use any arbitrary key/value mapping store, including simple in-memory dictionaries, but the mapping module here ensures that any file-system meeting the spec can be viewed as such as store, and so can be used by zarr to access netCDF-like data-sets. This comes with the added benefit of being able to transparently access remote data on cloud services like S3 and GCS, and to be able to read the chunks in parallel, both things that HDF5 has problems with.