Saving TIFF volumes: out of memory

[tischi]

@manerotoni
I cannot reproduce this:

FileType: Luxendo HDF5
BitDepth: 16
nX: 2048
nY: 2048
nZ: 101
nC: 2
nT: 144
voxelUnit: micrometer
voxelSizeX: 0.104
voxelSizeY: 0.104
voxelSizeZ: 1.4142135623730951
GB: 244.007829504

# Bin...
Binning: [5, 5, 1, 1, 1]

# Save
Saving started; I/O threads: 1
Files saved: 1/144; time ( spent, left, task ) [ s ]: 16, 2373, 16.6; memory: 2540MB of 9102MB (27%)
Files saved: 1/144; time ( spent, left, task ) [ s ]: 16, 2373, 16.6; memory: 2540MB of 9102MB (27%)
Files saved: 2/144; time ( spent, left, task ) [ s ]: 28, 2011, 14.2; memory: 2540MB of 9102MB (27%)
Files saved: 2/144; time ( spent, left, task ) [ s ]: 28, 2011, 14.2; memory: 2540MB of 9102MB (27%)
Files saved: 3/144; time ( spent, left, task ) [ s ]: 46, 2178, 15.5; memory: 2540MB of 9102MB (27%)
Files saved: 3/144; time ( spent, left, task ) [ s ]: 46, 2178, 15.5; memory: 2540MB of 9102MB (27%)
Files saved: 4/144; time ( spent, left, task ) [ s ]: 64, 2257, 16.1; memory: 2540MB of 9102MB (27%)
Files saved: 4/144; time ( spent, left, task ) [ s ]: 64, 2257, 16.1; memory: 2540MB of 9102MB (27%)
Files saved: 5/144; time ( spent, left, task ) [ s ]: 82, 2290, 16.5; memory: 2540MB of 9102MB (27%)
Files saved: 5/144; time ( spent, left, task ) [ s ]: 82, 2290, 16.5; memory: 2540MB of 9102MB (27%)
Files saved: 6/144; time ( spent, left, task ) [ s ]: 101, 2323, 16.8; memory: 2540MB of 9102MB (27%)
Files saved: 6/144; time ( spent, left, task ) [ s ]: 101, 2323, 16.8; memory: 2540MB of 9102MB (27%)
Files saved: 7/144; time ( spent, left, task ) [ s ]: 120, 2355, 17.2; memory: 2541MB of 9102MB (27%)
Files saved: 7/144; time ( spent, left, task ) [ s ]: 120, 2355, 17.2; memory: 2541MB of 9102MB (27%)
Files saved: 8/144; time ( spent, left, task ) [ s ]: 138, 2356, 17.3; memory: 1730MB of 9102MB (19%)
Files saved: 8/144; time ( spent, left, task ) [ s ]: 138, 2356, 17.3; memory: 1730MB of 9102MB (19%)
Files saved: 9/144; time ( spent, left, task ) [ s ]: 156, 2353, 17.4; memory: 2541MB of 9102MB (27%)
Files saved: 9/144; time ( spent, left, task ) [ s ]: 156, 2353, 17.4; memory: 2541MB of 9102MB (27%)

Could you please record a Javascript macro of what you are doing? Then I could test it better.
Thanks.

[manerotoni]

I created a new fresh copy of imageJ with fresh installation of BP2. It kind of freezes and memory is at 25 GB. . Sorry I can't copy paste the log and Recorder. I just post some screenshots.

Here the log

here the command series

[manerotoni]

Hi @tischi,
when I save the full data set without cropping it seems to work and there is no memory overload. Here is the script when I run a volume tif saving without prior cropping

Directory: V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\Position 3_Settings 1\
Regular expression: t(?<T>\d+)_(?<C>.+).tif
File name pattern: t(?<T>\d+)_(?<C>.+).tif
Sub-folders: []
Fetching files in V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\Position 3_Settings 1\
Found 1200 files in folder: V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\Position 3_Settings 1\
Folder: V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\Position 3_Settings 1\
FileType: TIFF Stacks
BitDepth: 16
TIFF Compression: None
TIFF Strips: 1
nX: 1024
nY: 1024
nZ: 50
nC: 2
nT: 600
voxelUnit: µm
voxelSizeX: 0.173333333333333
voxelSizeY: 0.173333333333333
voxelSizeZ: 1.0
GB: 125.82912
 

# Save
Saving started; I/O threads: 1
Frames saved: 2/2; time ( spent, left, task ) [ s ]: 30, 0, 15.4; memory: 371MB of 24434MB (1%)

// Save...
savingSettings = SavingSettings.getDefaults();
savingSettings.volumesFilePathStump = "V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\tifs\volumes\Position 3_Settings 1";
savingSettings.projectionsFilePathStump = "V:\11700\Eirini\2021_01_29_Viventis light-sheet_trim_MII\tifs\projections\Position 3_Settings 1";
savingSettings.numIOThreads = 1;
savingSettings.numProcessingThreads = 12;
savingSettings.fileType = SaveFileType.TIFFVolumes;
savingSettings.saveProjections = true;
savingSettings.saveVolumes = true;
savingSettings.compression = "None";
savingSettings.tStart = 0;
savingSettings.tEnd = 1;
BigDataProcessor2.saveImageAndWaitUntilDone( image, savingSettings );

[manerotoni]

The imaris saving of the full file also works.

When I do cropping first it just does not proceed with the saving. This is only true for imaris and tif volume saving. Saving as xml/hdf5 works well.

Also what is really strange is that I see the memory overload only when saving locally the cropped version. When I save the full version locally it is all fine.

[manerotoni]

Here is a script using the viventis data. Without cropping it works. If I do cropping first the saving is just stuck after the creation of the ims file, no h5 files are created. Memory goes up to 30 GB. The memory leak is only when saving locally. I am working on a Windows computer.

// Save...
savingSettings = SavingSettings.getDefaults();
savingSettings.volumesFilePathStump = "D:\TestBDP2\imaris_fullfile\volumes\Position 1_Settings 1";
savingSettings.projectionsFilePathStump = "D:\TestBDP2\imaris_fullfile\projections\Position 1_Settings 1";
savingSettings.numIOThreads = 1;
savingSettings.numProcessingThreads = 12;
savingSettings.fileType = SaveFileType.ImarisVolumes;
savingSettings.saveProjections = true;
savingSettings.saveVolumes = true;
savingSettings.compression = "None";
savingSettings.tStart = 0;
savingSettings.tEnd = 1;
BigDataProcessor2.saveImageAndWaitUntilDone( image, savingSettings );

// Crop...
image = BigDataProcessor2.crop( image, [155,282,22,0,0,865,807,27,1,1] );
image.setName( "Position 1_Settings 1-crop" );

// Save...
savingSettings = SavingSettings.getDefaults();
savingSettings.volumesFilePathStump = "D:\TestBDP2\imaris_cropped\volumes\Position 1_Settings 1-crop";
savingSettings.projectionsFilePathStump = "D:\TestBDP2\imaris_cropped\projections\Position 1_Settings 1-crop";
savingSettings.numIOThreads = 1;
savingSettings.numProcessingThreads = 12;
savingSettings.fileType = SaveFileType.ImarisVolumes;
savingSettings.saveProjections = true;
savingSettings.saveVolumes = true;
savingSettings.compression = "None";
savingSettings.tStart = 0;
savingSettings.tEnd = 1;
BigDataProcessor2.saveImageAndWaitUntilDone( image, savingSettings );

[manerotoni]

Here the example with the tif volume that fails (again doing cropping first, without cropping I manage to save). Same phenotype, no data are created and memory usage is maximal.

// Crop...
image = BigDataProcessor2.crop( image, [155,282,22,0,0,865,819,27,1,1] );
image.setName( "Position 1_Settings 1-crop" );

// Save...
savingSettings = SavingSettings.getDefaults();
savingSettings.volumesFilePathStump = "D:\TestBDP2\tifs\volumes\Position 1_Settings 1-crop";
savingSettings.projectionsFilePathStump = "D:\TestBDP2\tifs\projections\Position 1_Settings 1-crop";
savingSettings.numIOThreads = 1;
savingSettings.numProcessingThreads = 12;
savingSettings.fileType = SaveFileType.TIFFVolumes;
savingSettings.saveProjections = true;
savingSettings.saveVolumes = true;
savingSettings.compression = "None";
savingSettings.tStart = 0;
savingSettings.tEnd = 1;
BigDataProcessor2.saveImageAndWaitUntilDone( image, savingSettings );

The console shows also this error but I am not sure it is related. It seems to be connected with the painting process.

Setting SciJava uiService.
Setting SciJava context.
Setting SciJava uiService.
Setting SciJava context.
TransformedBoxOverlaySource.repaint
TransformedBoxOverlaySource.repaint
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.lang.String.substring(String.java:1967)
	at java.awt.TextArea.replaceText(TextArea.java:385)
	at java.awt.TextArea.replaceRange(TextArea.java:372)
	at de.embl.cba.bdp2.record.ScriptRecorder.removeMacroCallFromRecorder(ScriptRecorder.java:369)
	at de.embl.cba.bdp2.record.ScriptRecorder.lambda$record$3(ScriptRecorder.java:138)
	at java.lang.Thread.run(Thread.java:748)

[tischi]

Thanks a lot!

If I do cropping first the saving is just stuck after the creation of the ims file, no h5 files are created. Memory goes up to 30 GB. The memory leak is only when saving locally. I am working on a Windows computer.

What do you mean by: The memory leak is only when saving locally

What happens (i) with cropping and (ii) saving "remotely"? Does it save the data in that case?

And just to make sure: locally = on the same computer; remotely = on a network share?

[manerotoni]

I double checked again with the memory increase. My previous statement was not correct. I see full memory usage when saving on the same computer and on a network share. This makes more sense. May be yesterday the connection to the share was slow and the problem did not appeared so fast.

Overall for me on a Windows computer with Fiji version installed on 16.02 and updated sites I can't save a cropped version of the file for imaris saving and tiff volume. I am sure that this used to work last week as I was demonstrating it to a student.

[tischi]

OK, I will check later. I am in meetings this morning...

1. I had the feeling you may have cropped several times in a row, is that true?
2. I only have this Viventis data:

Would that be sufficient to try to reproduce your issue or do I need more/other data?

[manerotoni]

You can use the data you have. I see the problem also with those data. I only cropped once and then saved the data.

[tischi]

It works for me... (on a Mac though).
Just to be sure I uploaded a new version again.
If you still have trouble we could have an interactive zoom session later today (just write me a mail).

[manerotoni]

I tried, but it is not working yet on my Windows. Have you tried on a windows computer too.
I will try it on the mac and on another windows machine to see how it goes.

Antonio

[tischi]

I am in zoom (sent you a mail).