Parallel recursive breadth-first w.workspace.list(..., recursive=True, threads=os.cpu_count()) to iterate over 10K notebooks faster
#284
Conversation
nfx
changed the title
w.workspace.list()w.workspace.list()
nfx
changed the title
w.workspace.list()w.workspace.list() to iterate over 10K notebooks faster
nfx
changed the title
w.workspace.list() to iterate over 10K notebooks fasterw.workspace.list(..., recursive=True, threads=os.cpu_count()) to iterate over 10K notebooks faster
There was a problem hiding this comment.
I think this is a good addition to the SDK. Let's drop the reporter for now, or if you want, add it as a parameter to this API. Aside from this, I think there is one bug in the _list method.
Can we add an integration test where we actually create some recursive directory structure, try to list everything, and check that it worked correctly?
| def _list(self, path): | ||
| listing = self._listing(path, notebooks_modified_after=self._notebooks_modified_after) | ||
| for object_info in sorted(listing, key=lambda _: _.path): | ||
| if object_info.object_type != ObjectType.DIRECTORY: |
There was a problem hiding this comment.
I think this should be ==? The code in this block refers to what happens in a directory.
| with self._cond: | ||
| return self._in_progress > 0 | ||
|
|
||
| def _reporter(self): |
There was a problem hiding this comment.
The reporter is a bit weird. Given that this is a generator and users may not list all of the items "right away", this would just continue to print logs until the generator is consumed. Alternatively, this could be provided as a parameter to the parallel list operation, allowing users to provide a callback that is periodically called with statistics like this if they desire, then they can control when/whether they want to log or do something else.
for when we need to list 10K notebooks a bit faster.