Memory error while using fieldmaps

[beyzakkoyunlu]

Hi,
Previously I’ve used fMRIprep with no errors, however when I add my fieldmap data (pepolar) I’m getting memory error. In the below you can find data and computer specifics

-I’m using a computer with 12gb RAM, 8 core, Ubuntu 18.04
-Data: MB factor 4, 1.5x1.5x1.5, number of measurements between 125-305

The code I’ve used fmriprep-docker /home/beyza/Desktop/BIDS/Nifti/ /home/beyza/Desktop/preprocessing --participant-label 01 --fs-license-file $FREESURFER_HOME/license.txt --ignore slicetiming --low-mem
When I check the error log, the last lines are always same
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/volumeutils.py”, line 965, in apply_read_scaling
arr = arr * slope
MemoryError

While, I’m searching, I’ve seen a few things about docker memory allocation but, it seems like this problem mostly seen in Windows and MacOS. When I get system specifics from docker, it correctly shows the memory. So I’m wondering, if there is a problem with my installation or something like that. Or my hardware is not enough with this process? First I had 8gb memory and fmriprep gave a warning which says some of the processes might need larger memory. Then I upgraded my hardware so I was hoping it would work fine.

Thanks in advance!

[effigies]

The tracebacks where this error is produced will provide better context as to which specific tasks are running into this issue.

To explain though, the error that you're seeing occurs specifically because the data is saved with scale factors, a slope and intercept that are applied to all voxels. Most likely this is because the data is saved as int16 to be smaller on-disk. If that's the case, then when interpreting the data, the whole array must be loaded into memory, scaled by the slope and shifted by the intercept, which means a copy of the data must be made. This also casts the data to a 64-bit float, which means it's a 4x increase in memory usage.

In a month or so, we should be able to reduce that to 2x, as there are some pending improvements to an upstream library. (@oesteban We can get this right now, if we're willing to depend on a nibabel release candidate.)

In the meantime, I can make two suggestions:

1. The straightforward thing is to use a system with more memory. The 8GB warning is meant to catch systems (and often poorly configured Docker setups) where you're pretty much guaranteed an out-of-memory condition. For higher resolution data, 8GB is not going to be enough. I'm not really sure what you're looking at, but 16-24GB is not uncommon. For large datasets, moving to a high-performance computing center will pay off.

2. You can try converting your data to float32 before running. If I'm right that we continue saving data with the same type as the input, this will prevent us from rescaling the data back to int16 in intermediate files, and allow the --low-mem option to leave the data on disk.

[oesteban]

SDCFlows 2.0 should ease this problem (#2392)