collect_derivatives uses incorrect "from" entity while searching for pre-calculated boldref -> fmap coregistrations (orig vs boldref)

[psadil]

What happened?

fMRIPrep generates rigid xfms between the boldref and the fieldmap that have names like sub-{subject}_ses-{session}_task-{task}_run-{run}_from-boldref_to-{fieldmap_id}_mode-image_xfm.txt.

I am trying to supply precalculated xfms with the --derivatives flag, but it doesn't seem like they are being picked up.

What command did you use?

docker run \
  --init \
  --rm \
  -it \
  -v $PWD:$PWD \
  nipreps/fmriprep:24.1.0 \
  --dummy-scans 15 \
  -w $PWD/work2 \
  --fs-license-file $PWD/license \
  --derivatives boldref2fmap=$PWD/sourcedata/boldref2fmap  \
  --notrack  \
  $PWD/rawdata $PWD/derivatives2/fmriprep participant


WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2024-09-27 22:38:47,401 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
bids-validator@1.14.10
(node:10) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
		./README

	Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                  Available Tasks:                     Available Modalities: 
        24 Files, 771.34MB        cuff                                 MRI                   
        1 - Subject               rest                                                       
        1 - Session               TODO: full task name for cuff                              
                                  TODO: full task name for rest                              


	If you have any questions, please post on https://neurostars.org/tags/bids.
2024-09-27 22:38:57,903 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
2024-09-27 22:38:58,126 [ WARNING] WARNING: Niworkflows will be deprecating reporting in favor of a standalone library "nireports".
2024-09-27 22:38:58,251 [ WARNING] WARNING: Niworkflows will be deprecating visualizations in favor of a standalone library "nireports".
240927-22:38:59,171 nipype.workflow IMPORTANT:
	 Running fMRIPrep version 24.1.0
[...]

What version of fMRIPrep are you running?

24.1.0

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

FreeSurfer

Please copy and paste any relevant log output.

240927-22:38:59,200 nipype.workflow IMPORTANT:
	 Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.
2024-09-27 22:38:59,200 [IMPORTANT] Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.

Additional information / screenshots

I guess the issue is that the searched for pattern uses "from": "orig",?

As in, below, I would have expected "boldref" to work but not "orig"

from pathlib import Path
from fmriprep.utils import bids
import tempfile
import json

entities = {"subject": "0", "task": "rest"}

for from_id in ["boldref", "orig"]:
    with tempfile.TemporaryDirectory() as tmpd_:
        tmpd = Path(tmpd_) / "boldref2fmap"
        funcd = tmpd / f"sub-{entities['subject']}" / "func"
        funcd.mkdir(parents=True)
        stem = f"sub-{entities['subject']}_task-{entities['task']}_from-{from_id}_to-auto00000_mode-image_xfm"
        (funcd / f"{stem}").with_suffix(".txt").write_text("placeholder")
        (funcd / f"{stem}").with_suffix(".json").write_text(
            json.dumps(entities)
        )
        derivs = bids.collect_derivatives(
            derivatives_dir=tmpd,
            entities=entities,
            fieldmap_id=None,
        )
        print(f"looking for from-{from_id}")
        print(derivs)

looking for from-boldref
defaultdict(<class 'list'>, {})
looking for from-orig
defaultdict(<class 'list'>, {'boldref2fmap': '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/tmp58nmly9z/boldref2fmap/sub-0/func/sub-0_task-rest_from-orig_to-auto00000_mode-image_xfm.txt'})

FWIW, I'm trying to supply these xfms as a way to troubleshoot the issue described here: https://neurostars.org/t/poor-fieldmap-functional-registration-in-fmriprep-highly-variable-step/30397, which I mention shamelessly to get more eyes on it ;)

[psadil]

Unless, perhaps that field aims to pick up a different transformation -- not the boldref-auto##### generated during typical use of pepolar with fmriprep?

[effigies]

Yes, I think you're right. And I think the boldref2anat also needs to be updated.

[effigies]

Would you mind opening a PR against the maint/24.1.x branch?

[psadil]

Yes, can do. Just to make sure I understand how this is supposed to work, is the idea that when I run a command like

fmriprep --derivatives boldref2fmap=sourcedata/boldref2fmap [...] "${INPUT}" "${OUTPUT}" participant

where sourcedata/boldref2fmap looks like

sourcedata
└── boldref2fmap
   └── sub-travel2
      └── ses-NS
         └── func
            ├── sub-travel2_ses-NS_task-rest_run-01_from-boldref_to-auto00000_mode-image_xfm.json
            └── sub-travel2_ses-NS_task-rest_run-01_from-boldref_to-auto00000_mode-image_xfm.txt

then that xfm and json will be copied into ${OUTPUT} and used by the relevant nodes?

[effigies]

Not necessarily that it will be copied into the output, but that its generation will be skipped. I do not think we are 100% consistent on whether a precomputed file will be copied to output or not.

[psadil]

I think there might be a couple of other issues

• init_bold_fit_wf expects the precomputed files to be underneath a key "transforms"

  fmriprep/fmriprep/workflows/bold/fit.py

  Lines 249 to 252 in 73189de

  	transforms = precomputed.get('transforms', {})
  	hmc_xforms = transforms.get('hmc')
  	boldref2fmap_xform = transforms.get('boldref2fmap')
  	boldref2anat_xform = transforms.get('boldref2anat')

  but collect_derivatives puts them at the top level (e.g., underneath a field called "boldref2fmap":

  fmriprep/fmriprep/utils/bids.py

  Line 78 in 73189de

  derivs_cache[xfm] = item[0] if len(item) == 1 else item

  )
• The query for precomputed transforms does not include the entities of the relevant image (**q vs **query)

  fmriprep/fmriprep/utils/bids.py

  Lines 71 to 75 in 73189de

  	for xfm, q in spec['transforms'].items():
  	query = {**q, **entities}
  	if xfm == 'boldref2fmap':
  	query['to'] = fieldmap_id
  	item = layout.get(return_type='filename', **q)

  and so all xfms are associated with each input bold

Should I break those into separate issues and pull requests, or just do them together?

[effigies]

Feel free to fix together.