fmriprep crashed, seems to be related to ICA aroma

[hstojic]

As instructed in the console, I'm posting the errors here. This happened only when I used the flag to use ICA aroma, on other datasets where I didn't use it fmriprep worked fine.

This is the traceback from the console ouptut:

181114-06:00:10,207 nipype.workflow INFO:
         [Node] "_fix_surfs7" found cached.
181114-06:00:10,212 nipype.workflow INFO:
         [Node] Finished "fmriprep_wf.single_subject_s031_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.fix_surfs".
181114-06:00:25,225 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_08_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,226 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_07_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,226 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_06_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,226 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_05_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,227 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_04_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,227 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_03_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,227 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_02_wf.ica_aroma_wf.rm_nonsteady
181114-06:00:25,228 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_02_wf.bold_confounds_wf.tcompcor
181114-06:00:25,228 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_01_wf.ica_aroma_wf.rm_nonsteady
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 350, in main
    errno += generate_reports(subject_list, output_dir, work_dir, run_uuid)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/reports.py", line 312, in generate_reports
    for subject_label in subject_list
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/reports.py", line 312, in <listcomp>
    for subject_label in subject_list
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/reports.py", line 302, in run_reports
    return report.generate_report()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/reports.py", line 204, in generate_report
    boilerplate=boilerplate)
  File "/usr/local/miniconda/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/miniconda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/viz/report.tpl", line 161, in top-level template code
    {% for name, spec in error.inputs %}
ValueError: not enough values to unpack (expected 2, got 1)
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
fMRIPrep: Please report errors to https://github.com/poldracklab/fmriprep/issues

And here are 2 of the crash files, 8 of them seem to be related to ica-aroma, each related to one of 8 of the runs I have, so I only posted 1 of them:
crash-20181114-050032-root-tcompcor-f4e1b577-2af0-40cb-902d-498ff4897efd.txt
crash-20181114-051112-root-rm_nonsteady-00609107-89f0-41bb-9405-e1e14348f712.txt

Perhaps of relevance, I have two fieldmaps for this dataset, but phasedidd.json files are correctly done and on other such dataset fmriprep worked fine (but without ICA).

Any thoughts what might be causing issues?

[oesteban]

Could you give it a try first without ICA AROMA?, I'm concerned with the first crash file you posted.

I think the ICA AROMA may have a disconnected path (ref #1335). Probably @jdkent's eyes are more trained for this error.

[hstojic]

Hm, indeed, I ran the same dataset tonight without ICA AROMA, and I still get an error.

181115-00:55:45,200 nipype.workflow INFO:
         [Node] "_update_metadata1" found cached.
181115-00:55:45,203 nipype.workflow INFO:
         [Node] Finished "fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_01_wf.bold_surf_wf.update_metadata".
181115-00:56:51,658 nipype.workflow ERROR:
         could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_02_wf.bold_confounds_wf.tcompcor
Errors occurred while generating reports for participants: s031 (1).

This is the crash report:
crash-20181115-003727-root-tcompcor-f61ebcb5-340e-43c6-86dc-5a9fde1006ef.txt

I checked the files and all files except the confounds file for one run seem to be present, report is also generated.

Perhaps I shouldn't have let fmriprep use cached files?

[jdkent]

The number of volumes that should be skipped is calculated from bold_reference_wf.outputnode.skip_vols, so whenever that node isn't run, the number of volumes to skip will be undefined. That is now the case for acompcor/tcompor as well, so the error for those nodes is not surprising. I'll dig into cases when that node is not run.

@hstojic Would you mind sharing an example call of fmriprep? also what version? A recent fix was applied for version 1.2.2 because of a similar error another user had in version 1.2.1 (ref #1365)

[hstojic]

sure, the command for fmriprep-docker version 1.2.1 was:

fmriprep-docker $DIR_BIDS $DIR_OUT participant -w $DIR_WORK --participant_label $SUBJECT --mem_mb $MAX_MEM --low-mem --nthreads $N_CPU --resource-monitor

variables are understandable, I hope.

I'll try updating to 1.2.2 and running again in the meantime...

UPDATE: after running it with version 1.2.2 I get the same error

181115-21:18:39,546 nipype.workflow INFO: [Node] Finished "fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_01_wf.bold_surf_wf.update_metadata".
181115-21:19:45,964 nipype.workflow ERROR: could not run node: fmriprep_wf.single_subject_s031_wf.func_preproc_task_fnclearning_run_02_wf.bold_confounds_wf.tcompcor
Errors occurred while generating reports for participants: s031 (1).

[effigies]

Do you have sbref images? And did you reuse your working directory? If so, the problem is likely that you need to remove fmriprep_wf/single_subject_*_wf/func_preproc_task_*wf/bold_reference_wf. Hopefully most steps will not need to be rerun, if you're using sbrefs.

[hstojic]

Yes, I do have _sbref images (multiband sequence) and I was using exactly the same command, so the working directory should be the same.

Thanks for the tip, I'll try with that!

[hstojic]

In the meantime I tried ICA AROMA on another subject, now with version 1.2.2. and it seems to work fine, so perhaps it was due to the older version 1.2.1. Although it did take 18h - for 6 runs, about 400 volumes each (multiband seq) - 14 cores and 32GB.

[hstojic]

Simply running the fmriprep again did the trick, missing confounds file has been generated.

@effigies could you clarify a bit why removing fmriprep_wf/single_subject_*_wf/func_preproc_task_*wf/bold_reference_wf folder worked?

Otherwise, issue seems to be resolved and can be closed.

[effigies]

Sure. The reason is that we fixed #1365 by updating EstimateReferenceImage in nipreps/niworkflows#258. But we didn't change any of its inputs. When nipype decides whether to rerun a node, it checks if the inputs have changed; there's (currently) no way to tell that the interface itself has changed. Thus, until you removed the working directory, it wouldn't rerun, and wouldn't produce the correct outputs.