Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make zarr download begin as soon as initial page of zarr files is obtained #1447

Open
yarikoptic opened this issue May 30, 2024 · 1 comment
Assignees
Labels

Comments

@yarikoptic
Copy link
Member

This is continuation to work done in

where I thought that interleave() would be a generator which starts yielding without delaying for a full list of zarr files to be retrieved first. @jwodder mentioned that my assumption was wrong and that it would need additional refactoring to make it behave as desired.

While addressing this it might be worth checking if interleave could natively provide support for desired use case where it takes care about running that thread with "producer" iterator. It is somewhat similar to how I have done for ProducerConsumer in datalad where Producer is ran into a separate thread as well IIRC.

@yarikoptic
Copy link
Member Author

ping on this one @jwodder , as today I have tried download of a little zarr and it was odd to stare at non-changing download output for quite a while. Example was

$ dandi download dandi://dandi/000719/sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.nwb.zarr

which eventually started to download

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants