Can't define 2 different B0FieldSource from the same fmap files for fMRIprep

[miltoncamacho]

What happened?

When running fMRIprep on a subject with two fMRI runs I realized that only one of the fMRI runs where sdc. If I am using:

json of epi fmap1 AP
B0FieldIdentifier: ["sub-x_ses_1a_task1","sub-x_ses_1a_task2"]

json of epi fmap2 PA
B0FieldIdentifier: ["sub-x_ses_1a_task1","sub-x_ses_1a_task2"]

Then I try to use the independent B0FieldSource:

json of bold func1:
B0FieldSource: "sub-x_ses_1a_task1"

json of bold func2:
B0FieldSource: "sub-x_ses_1a_task2"

I get an error since the bidirectional dictionary in:

sdcflows/sdcflows/utils/bimap.py

Line 30 in 52209e4

class bidict(dict):

requires unique keys and values and the values (fmap1 and fmap2) for both bids_id in func1 and func2 (

sdcflows/sdcflows/utils/wrangler.py

Line 381 in 52209e4

bids_id=b0_id,

) would be the same.

Was this the intended behaviour? Should I not be able to do this combination? There is probably a good reason for using the bidirectional dictionary that I don't know.

What command did you use?

You can replicate this with:

import shutil

from niworkflows.utils.testing import generate_bids_skeleton

from sdcflows.fieldmaps import clear_registry, get_identifier
from sdcflows.utils.wrangler import find_estimators

def gen_layout(bids_dir, database_dir=None):
    import re

    from bids.layout import BIDSLayout, BIDSLayoutIndexer

    _indexer = BIDSLayoutIndexer(
        validate=False,
        ignore=(
            'code',
            'stimuli',
            'sourcedata',
            'models',
            'derivatives',
            re.compile(r'^\.'),
            re.compile(r'sub-[a-zA-Z0-9]+(/ses-[a-zA-Z0-9]+)?/(beh|eeg|ieeg|meg|micr|perf)'),
        ),
    )

    layout_kwargs = {'indexer': _indexer}

    if database_dir:
        layout_kwargs['database_path'] = database_dir

    layout = BIDSLayout(bids_dir, **layout_kwargs)
    return layout


pepolar_b0ids = {
    '01': [
        {
            'session': '01',
            'anat': [{'suffix': 'T1w', 'metadata': {'EchoTime': 1}}],
            'func': [
                {
                    'task': 'rest',
                    'run': 1,
                    'suffix': 'bold',
                    'metadata': {
                        'RepetitionTime': 0.8,
                        'TotalReadoutTime': 0.5,
                        'PhaseEncodingDirection': 'j',
                        'B0FieldSource': 'b0_pepolar',
                    },
                },
                {
                    'task': 'rest',
                    'run': 2,
                    'suffix': 'bold',
                    'metadata': {
                        'RepetitionTime': 0.8,
                        'TotalReadoutTime': 0.5,
                        'PhaseEncodingDirection': 'j-',
                        'B0FieldSource': 'b0_pepolar_dub',
                    },
                },
            ],
            'fmap': [
                {
                    'run': 1,
                    'suffix': 'epi',
                    'metadata': {
                        'RepetitionTime': 0.8,
                        'TotalReadoutTime': 0.5,
                        'PhaseEncodingDirection': 'j',
                        'B0FieldIdentifier': ['b0_pepolar', 'b0_pepolar_dup'],
                    },
                },
                {
                    'run': 2,
                    'suffix': 'epi',
                    'metadata': {
                        'RepetitionTime': 0.8,
                        'TotalReadoutTime': 0.5,
                        'PhaseEncodingDirection': 'j-',
                        'B0FieldIdentifier': ['b0_pepolar', 'b0_pepolar_dup'],
                    },
                },
            ],
        },
    ]
}

filters = {
    'multi': {'session': '01', 'suffix': ['bold','epi']},
}
clear_registry()
tmpdir = "/tmp/path" #tmp folder where to generated the bids skeleton
bids_dir = str(tmpdir+"/pepolar_b0ids")
shutil.rmtree(bids_dir, ignore_errors=True)
generate_bids_skeleton(bids_dir, pepolar_b0ids)
layout = gen_layout(bids_dir)
est = find_estimators(layout=layout, subject='01', bids_filters=filters["multi"])

should get you:
In [5]: est
Out[5]: [FieldmapEstimation(sources=<2 files>, method=<EstimatorType.PEPOLAR: 2>, bids_id='b0_pepolar')]
but I would expect this to be 
Out[5]: [FieldmapEstimation(sources=<2 files>, method=<EstimatorType.PEPOLAR: 2>, bids_id='b0_pepolar'),FieldmapEstimation(sources=<2 files>, method=<EstimatorType.PEPOLAR: 2>, bids_id='b0_pepolar_dup')]

What version of the software are you running?

25.0.0.dev3+g52209e48a

How are you running this software?

Local installation ("bare-metal")

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

I added some extra debugging statements:
[sdcflows.wrangler - DEBUG]: Dataset includes `B0FieldIdentifier` metadata.Any data missing this metadata will be ignored.
[sdcflows.wrangler - DEBUG]: Searching fieldmaps with B0FieldIdentifier=b0_pepolar
[sdcflows.wrangler - DEBUG]: Found bare_ids [] + listed_ids [<BIDSImageFile filename='/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-1_epi.nii.gz'>, <BIDSImageFile filename='/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-2_epi.nii.gz'>]
this is what the fielmap estimation class gets as bids_id: b0_pepolar
this is what the fielmap estimation class gets as paths: ('/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-1_epi.nii.gz', '/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-2_epi.nii.gz')
[sdcflows.wrangler - DEBUG]: Searching fieldmaps with B0FieldIdentifier=b0_pepolar_dup
[sdcflows.wrangler - DEBUG]: Found bare_ids [] + listed_ids [<BIDSImageFile filename='/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-1_epi.nii.gz'>, <BIDSImageFile filename='/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-2_epi.nii.gz'>]
this is what the fielmap estimation class gets as bids_id: b0_pepolar_dup
this is what the fielmap estimation class gets as paths: ('/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-1_epi.nii.gz', '/Users/milton/Desktop/tmp-sdcflows-test/pepolar_b0ids/sub-01/ses-01/fmap/sub-01_ses-01_run-2_epi.nii.gz')

Additional information / screenshots

No response

[effigies]

No, we should figure out how to handle this. I don't have a great sense without digging of how painful it will be.

That said, I don't think your use case is a good candidate for this feature. Each B0FieldIdentifier will produce a different B0 field estimate; using the same inputs for multiple outputs is just duplicating work. It would be better to set both of your BOLD files to be corrected with the same B0FieldSource.

[miltoncamacho]

I now know that this would result in two estimators that would have the same result under a different name, is just that we have a pipeline which produces the reported structure thinking it would be correct and I was wondering where to apply the fix. Wdyt @bpinsard?

[miltoncamacho]

#129 Maybe this was indeed the intended behaviour #141 but if you think that we can simply make the error a warning I can do that.

[bpinsard]

Yes, I guess sdcflows makes choices for pipelines efficiency that adds constraints which are not in the BIDS specs as itself does not deal with computational considerations.

From what I see, there is an error that gets passed more as a warning, it does not crash pytest :

Error: '('/tmp/pytest-of-basile/pytest-27/test_wrangler_filter_pepolar_b2/pepolar_b0ids/sub-01/ses-03/func/sub-01_ses-03_task-rest_run-1_sbref.nii.gz', '/tmp/pytest-of-basile/pytest-27/test_wrangler_filter_pepolar_b2/pepolar_b0ids/sub-01/ses-03/func/sub-01_ses-03_task-rest_run-2_sbref.nii.gz')' is already  in mapping.

We will need to adjust our BIDS B0 intent filling logic to merge estimator under the same ID.