Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fsspec support #114

Closed
d4l3k opened this issue Dec 4, 2021 · 1 comment
Closed

fsspec support #114

d4l3k opened this issue Dec 4, 2021 · 1 comment

Comments

@d4l3k
Copy link
Member

d4l3k commented Dec 4, 2021

🚀 Feature

Add a new loader similar to the iopath loader that uses fsspec.

https://github.com/pytorch/data/blob/main/torchdata/datapipes/iter/load/iopath.py

https://filesystem-spec.readthedocs.io/en/latest/

Motivation

It would be nice to have fsspec in addition to iopath for loading data from general data sources. A lot of projects already use it and support it which makes it a good to add to torchdata as well for uniform support.

PyTorch Lighting, Tensorboard and TorchX have support for fsspec already. It's quite easy to add support for a new storage provider and has many commons ones available already. Internally there's a Manifold provider which is used with many PyTorch/STL projects.

Alternatives

For common storage providers such as s3 there's generally already support for that in most projects though for custom / less used storage providers a user would have to implement support for each different system. iopath does provide a similar abstraction but it seems like fsspec generally has more OSS adoption so would be nice to have a unified interface across pytorch projects

Additional context

@ejguan
Copy link
Contributor

ejguan commented Dec 6, 2021

Wow. iopath is used a lot by research group, IIRC. If more STL project is using fsspec, it makes sense to provide such modular datapipe using fsspec.
I don't know Manifold is supported by fsspec internally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants