Using the preprocess external option with large images

[mjseignon]

I have encountered this error below using the preprocess external option on large images?
Note that I have used this option before with no issues.
1- I convert the original image from .qptiff to .ome.tiff
2- I pass the post conversion image, .ome.tiff, to the preprocess eternal option of steinbock
The only difference is that I'm running an image that is ~85G post conversion vs ~2.5G post conversion with my initial successful testing.

Any ideas as to why the external option is failing here?

`
Command executed:

steinbock preprocess external images --img external --panel /steinbock/assets/codex_panel_1.csv --infoout img_info.csv

Command exit status:
1

Command output:
(empty)

Command error:
File "/app/steinbock/steinbock/io.py", line 191, in write_image
tifffile.imwrite(
File "/opt/steinbock-venv/lib/python3.8/site-packages/tifffile/tifffile.py", line 1202, in imwrite
result = tif.write(
File "/opt/steinbock-venv/lib/python3.8/site-packages/tifffile/tifffile.py", line 2871, in write
addtag(tags, tagbytecounts, bytecountformat, count, databytecounts)
File "/opt/steinbock-venv/lib/python3.8/site-packages/tifffile/tifffile.py", line 3702, in _addtag
ifdentry.append(pack(valueformat, pack(dataformat, value)))
File "/opt/steinbock-venv/lib/python3.8/site-packages/tifffile/tifffile.py", line 3728, in _pack
return struct.pack(fmt, *val)
struct.error: 'I' format requires 0 <= number <= 4294967295
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
with os.fdopen(os.open(pr_filename,
OSError: [Errno 30] Read-only file system: '/var/crash/_opt_steinbock-venv_bin_steinbock.62988.crash'

`

[jwindhager]

@mjseignon Unfortunately, I don´t think there is an easy way around this at the moment, other than working on smaller image tiles. You can try using the steinbock utils tile --mmap command (may or may not work, depending on your OME-TIFF file structure), but you will likely observe some edge effects when merging the masks using steinbock utils stitch. Sorry to not have better news at this point.

---

Technical background to maintainer @Milad4849, tagging @nilseling:

The steinbock toolkit currently writes images with imagej=True, which apparently implicitly disables bigtiff: https://github.com/cgohlke/tifffile/blob/e680307a1cbafd77151b37c30c5f4c68c24328f4/tifffile/tifffile.py#L1272-L1278. Changing from imagej=True to bigtiff=True would likely address this issue, but this needs to be carefully evaluated (although I wouldn't expect any issues). A change in file format would definitely require a new MINOR version bump.

My 2C: If you do decide to change (probably a good idea), maybe immediately include some OME metadata (e.g. channel names), since tifffile natively supports writing OME-TIFF metadata, and drop the export ome command. In the long run, OME-Zarr may be worth a consideration, but probably not at this point (other tools aren't there yet).

In any case, I think this issue would probably warrant high priority.

[mjseignon]

Thank you Jonas. I'll keep an eye out for any updates on this technical note as well.

[wflynny]

@jwindhager, I agree with your conclusion---saving the images as OME would be preferred. I coincidentally ran into this issue independently and the tifffile maintainer responded here on how to add some of that metadata:
https://forum.image.sc/t/multichannel-tiff-with-custom-channel-names-with-python-tifffile/70573/5

[mjseignon]

Hi @jwindhager,
I'm coming back around to this issue again. Is this bug something that is being worked on currently? I am also open to push in changes if I am pointed to the right location with instructions. Let me know.
Martine

[jwindhager]

Hi @mjseignon, I fear I don't have an answer for you. I have changed position and therefore handed over the development on this project to @Milad4849, the current maintainer of steinbock. I'm sure he will have a more helpful answer and that he would appreciate the contribution! Personally, I can currently only afford to contribute in an advisory capacity (during my freetime).

[mjseignon]

Thank you @jwindhager . @Milad4849 please let me know if I can be of any help.