fsspec support

[d4l3k]

🚀 Feature

Add a new loader similar to the iopath loader that uses fsspec.

https://github.com/pytorch/data/blob/main/torchdata/datapipes/iter/load/iopath.py

https://filesystem-spec.readthedocs.io/en/latest/

Motivation

It would be nice to have fsspec in addition to iopath for loading data from general data sources. A lot of projects already use it and support it which makes it a good to add to torchdata as well for uniform support.

PyTorch Lighting, Tensorboard and TorchX have support for fsspec already. It's quite easy to add support for a new storage provider and has many commons ones available already. Internally there's a Manifold provider which is used with many PyTorch/STL projects.

Alternatives

For common storage providers such as s3 there's generally already support for that in most projects though for custom / less used storage providers a user would have to implement support for each different system. iopath does provide a similar abstraction but it seems like fsspec generally has more OSS adoption so would be nice to have a unified interface across pytorch projects

Additional context

[ejguan]

Wow. iopath is used a lot by research group, IIRC. If more STL project is using fsspec, it makes sense to provide such modular datapipe using fsspec.
I don't know Manifold is supported by fsspec internally.