-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
issues with dask arrays returned by nd2: depend on ND2File being open, can't be pickled #19
Comments
Actually, I just looked a bit into un/pickling and it seems that adding these to methods to the
I can now pickle the dask arrays and when I unpickle them I can run I will submit a PR. |
yeah, this is a tricky one... I actually wrote a wrapper over at aicsimageio for just this purpose: could use it here as well?
awesome thank you! |
Ugh. I would have to do a bit more reading to fully understand how that proxy works, so not sure about whether this would work here. However, as aicsimageio also incorporates nd2, maybe I should just aicsimageio instead of nd2 (I'd prefer not to have that dependency though). |
the gist of that proxy is essentially these two lines: def compute(self, **kwargs: Any) -> np.ndarray:
with self.__wrapped__._ctx_:
return self.__wrapped__.compute(**kwargs)
def __array__(self, dtype: str = None, **kwargs: Any) -> np.ndarray:
with self.__wrapped__._ctx_:
return self.__wrapped__.__array__(dtype, **kwargs) that is, it's a dask array that, whenever you try to call It looks a tad bit risky at first, but I haven't run into any issues with it yet. In any case, I suspect the issue of trying to use a dask array after closing the file is far more common than whatever hidden issues there are with this proxy. I'm inclined to try it |
I agree. So if I understand you correctly, the idea would be that |
yeah, exactly. |
Hi @tlambert03, didn't get around to it yesterday but installed the latest version including #27 from the main branch just now. In addition, I now also get dying kernels (probably segfault, but this is something I run in a notebook) when I run some more complex computation on the dask array that worked previously when avoiding returning from a context manager. I will try and provide a reproducible example, but probably won't get around to it today. |
ugh 😔 i'm sorry. this is a really frustrating one. I can't get it to segfault on any of my computers (tried mac, win, ubuntu20)... maybe it's a race condition that depends on dataset size? It's certainly an issue though, since it looks like aicsimageio is also having CI issues with the new release. Will keep digging... but let me know if you hit on anything (and send files if necessary) |
Will do. The newly observed segfault happens when I perform `map_blocks` on
the dask array, presumably things are happening concurrently for different
blocks.
I hope to create a minimal example by the end of the week.
…On Tue, 16 Nov 2021 at 16:15, Talley Lambert ***@***.***> wrote:
ugh 😔 i'm sorry. this is a really frustrating one.
I can't get it to segfault on any of my computers (tried mac, win,
ubuntu20)... maybe it's a race condition that depends on dataset size?
It's certainly an issue though, since it looks like aicsimageio is also
having CI issues with the new release. Will keep digging... but let me know
if you hit on anything (and send files if necessary)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#19 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQ3R7D4PXFVJIZS2AO67TLUMJYSNANCNFSM5HXM6MZA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
BTW, no need to be sorry and thanks for all the great support so far !
On Tue, 16 Nov 2021 at 16:22, Volker Hilsenstein <
***@***.***> wrote:
… Will do. The newly observed segfault happens when I perform `map_blocks`
on the dask array, presumably things are happening concurrently for
different blocks.
I hope to create a minimal example by the end of the week.
On Tue, 16 Nov 2021 at 16:15, Talley Lambert ***@***.***>
wrote:
> ugh 😔 i'm sorry. this is a really frustrating one.
>
> I can't get it to segfault on any of my computers (tried mac, win,
> ubuntu20)... maybe it's a race condition that depends on dataset size?
>
> It's certainly an issue though, since it looks like aicsimageio is also
> having CI issues with the new release. Will keep digging... but let me know
> if you hit on anything (and send files if necessary)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#19 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAQ3R7D4PXFVJIZS2AO67TLUMJYSNANCNFSM5HXM6MZA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
|
Ok, I started debugging this. The problem appears to be that the pickling doesn't work correctly if there are additional operations in the compute graph. I don't have a stand-alone example yet but maybe the following will already be giving you enough information: I have the following function that loads an nd2 and performs some re-sorting (using def load_tiles_nd2(file_path: PathLike) -> da.array:
file_path = Path(file_path)
if file_path.suffix.lower() != ".nd2":
raise (ValueError, "Can only read nd2 files.")
f = ND2File(file_path)
tile_array = f.to_dask()
dim_order_tuple = f.sizes.keys()
dim_order = "".join(dim_order_tuple).lower()
# the followig is basically np.einsum (f"{dim_order}->pcyx")
ordered_tile_array = to_tczyx(tile_array, in_order=dim_order, out_order="pcyx")
return ordered_tile_array So when I use this as in
there is no problem. However, when I do
I get a segmentation fault. When I look at the dask graph of the ResourceBackedArray that hasn't been pickled looks like that: In contrast, the dask task graph for the ResourceBackedArray that has been pickled and regenerated by unpickling looks like this: So the layers |
Arghh. Now I can't reproduce the SegFault with a simple |
So the segfault seems to depend on the slicing.
|
Thanks for digging! |
Thanks, interesting, I didn't even think about graph optmiziation steps. |
I think this has been solved over the iterations. currently: In [1]: import nd2
In [2]: f = nd2.ND2File('tests/data/dims_p1z5t3c2y32x32.nd2')
In [3]: d = f.to_dask()
In [4]: f.close()
In [5]: import cloudpickle
In [6]: p = cloudpickle.dumps(d) # no problem |
Hi Talley,
a couple of observations regarding the dask arrays returned by
nd2
. None of this is terribly surprising (so you are probbably aware), and I am not expecting this can really be fixed. But maybe some lines in the readme file could save users some headaches.keep ND2File object in open() state when working with dask array
For my use case I created a little helper function (pseudo-code):
I have similiar functions for other file types, e.g. something like
provide_tiles_as_daskarray_from_folder_of_imgs(filepath)
.On first sight, this works. I used this function from a Jupyter Notebook and it seemed as if it returns a working dask array.
Until you do anything that forces a
compute()
, in which case the python kernel crashes without a traceback.The issue is that
f
is closed when returning out of the context manager. The same thing happens if you don't use a context manager and callf.close()
before returning. Again, not terribly surprising. When running in the VS Code debugger I actually saw a trace back and saw that there is a segmentation fault when the nd2 library is trying to read data.My workaround is to not close
f
which is a bit dirty as there will be open file handles hanging around possibly causing memory leaks.cloudpickle
The dask arrays returned by
nd2
can't be serialized using cloudpickle due to the ND2File object not being pickleable:Again, not terribly surprising. However, the implication is (I haven't tried it, but am fairly certain this is the case) that the dask arrays returned by nd2 cannot be used in a
dask.distributed
context where the pickled array needs to be sent to the various worker processes.A workaround for both would be to create a temporary .zarr file (pseudo-code):
with obvious drawback of duplicating the required storarge space.
I don't know whether there is much that can be done about this, but it may be useful to add a note of caution in the Readme.
The text was updated successfully, but these errors were encountered: