Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torchdata/datapipes/iter/load/fsspec: added fsspec datapipes #116

Closed
wants to merge 1 commit into from
Closed

torchdata/datapipes/iter/load/fsspec: added fsspec datapipes #116

wants to merge 1 commit into from

Conversation

d4l3k
Copy link
Member

@d4l3k d4l3k commented Dec 7, 2021

This adds fsspec datapipes that are equivalent to the existing iopath ones. The tests are largely equivalent and test the file:// and memory:// filesystems to ensure compatibility.

Closes #114

Test plan:

pytest tests/test_fsspec.py

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 7, 2021
@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@d4l3k d4l3k force-pushed the fsspec branch 2 times, most recently from eaa8dd6 to 0f318b9 Compare December 7, 2021 23:25
@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@d4l3k
Copy link
Member Author

d4l3k commented Dec 7, 2021

@ejguan can you review this? Thanks!

ejguan
ejguan reviewed Dec 8, 2021
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please install fsspec for the CI script to turn on the tests?

data/.github/workflows/ci.yml

Line 47 in 6309113

run: pip3 install expecttest iopath==0.1.9 numpy pytest rarfile

You may need to install aiohttp as well.

torchdata/datapipes/iter/load/fsspec.py Outdated Show resolved Hide resolved
@d4l3k d4l3k force-pushed the fsspec branch 4 times, most recently from 5a9d1f2 to 4a9fca2 Compare December 8, 2021 22:52
@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ejguan
ejguan approved these changes Dec 9, 2021
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

ejguan
ejguan approved these changes Dec 9, 2021
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again!

@facebook-github-bot
Copy link
Contributor

@d4l3k has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NivekT NivekT NivekT left review comments

@ejguan ejguan ejguan approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

fsspec support
4 participants
@d4l3k @facebook-github-bot @NivekT @ejguan