Skip to content

Caching and Parallelism Improvements to Data Streaming #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Sep 3, 2021
Merged

Caching and Parallelism Improvements to Data Streaming #7

merged 4 commits into from
Sep 3, 2021

Conversation

PJEstrada
Copy link
Contributor

Main features:

  • Added a caching mechanism to save 1GB of data to avoid duplicate requests
  • Added a threaded approach to fetch the next file.
  • Proactively fetch next N files to reduce network waiting times.
  • Stop using session in request to allow for usage of multiple workers in dataloaders of pytorch.

@PJEstrada PJEstrada self-assigned this Sep 3, 2021
@PJEstrada PJEstrada merged commit 1a4a37f into main Sep 3, 2021
@PJEstrada PJEstrada deleted the caching-and-parallel-fetch branch September 3, 2021 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anthony-chaudhary anthony-chaudhary Awaiting requested review from anthony-chaudhary

Assignees

@PJEstrada PJEstrada

Labels
ready_to_merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@PJEstrada