Update B0fieldIdentifier/Source when having multiple fieldmap runs

[marcelzwiers]

When sessions are interrupted, typically a new fieldmap is acquired. The IntendedFor dynamic value can already limit it's search scope to adjacent runs. The B0FieldIdentifier/Source pairs should get the same possibility (either from within bidscoin or as a post-processing tool), e.g. by adding a run-suffix to the B0Field tags

[mateuszpawlik]

I guess there is no timeline for this feature?

[marcelzwiers]

It will probably be in the next release, but I don't know when that will be. I also haven't thought very much about the best approach to solve this issue

[marcelzwiers]

The new <<session>> value disambiguates this for multi-session data, perhaps I can put it in the same machinery (e.g. <<run>>)

[marcelzwiers]

So just to be clear, for a template bidsmap with {B0FieldIdentifier: tag} and {B0FieldSource: tag}, the desired output for e.g. a repeated (aborted) acquisition paradigm of three functional runs + one fieldmap would be something like:

|-- fmap
|   |-- sub-001_run-1_magnitude1.json        <- {B0FieldIdentifier: tag1}   (SeriesNumber = 01)
|   |-- sub-001_run-1_magnitude2.json        <- {B0FieldIdentifier: tag1}   (SeriesNumber = 01)
|   |-- sub-001_run-1_phasediff.json         <- {B0FieldIdentifier: tag1}   (SeriesNumber = 01)
|   |-- sub-001_run-2_magnitude1.json        <- {B0FieldIdentifier: tag2}   (SeriesNumber = 08)
|   |-- sub-001_run-2_magnitude2.json        <- {B0FieldIdentifier: tag2}   (SeriesNumber = 08)
|   `-- sub-001_run-2_phasediff.json         <- {B0FieldIdentifier: tag2}   (SeriesNumber = 08)
|
`-- func
    |-- sub-001_task-rest_run-1_bold.json    <- {B0FieldSource: tag1}       (SeriesNumber = 02)
    |-- sub-001_task-rest_run-2_bold.json    <- {B0FieldSource: tag1}       (SeriesNumber = 03)
    |-- sub-001_task-rest_run-3_bold.json    <- {B0FieldSource: tag2}       (SeriesNumber = 09)
    |-- sub-001_task-rest_run-4_bold.json    <- {B0FieldSource: tag2}       (SeriesNumber = 10)
    `-- sub-001_task-rest_run-5_bold.json    <- {B0FieldSource: tag2}       (SeriesNumber = 11)

For the fieldmaps this is straightforward to implement (just increment the tag), but unfortunately it is not so easy for the functional scans...

[mateuszpawlik]

I'm no expert but this is how I understand the BIDS specification. When you assign the IntendedFor values in the maps, cannot you inject B0FieldIdentifiers to the functional JSONs?

[marcelzwiers]

That is not possible, because the semantics that IntendedFor can specify (one-to-many) is a subset of what the B0 tags can specify (many-to-many) -- hence the deprecation of IntendedFor

[marcelzwiers]

I implemented this feature in the latest commit. Since you are the main user and presumably have a real-life case at hand, could you perhaps test it out to see if it does what you want? For usage see:

https://bidscoin.readthedocs.io/en/latest/bidsmap.html#fieldmaps-b0fieldidentifier-b0fieldsource

[mateuszpawlik]

I'll try to do some testing today. If I remember correctly, our problem was triggered by multiple fieldmaps in a single session (interrupted session). We used run: <<>> in the BIDS section of the fieldmaps, but we couldn't use it for B0FieldIdentifier/Source values.

[mateuszpawlik]

I tried with some generated data and the newest version on master branch:

SeriesNumber	sequence type
1	fmap magnitude1 and magnitude2 (at our scanner it is one sequence)
2	fmap pahsediff
3	func
4	fmap magnitude1 and magnitude2
5	fmap pahsediff
6	func
7	func

My template bidsmap

Options: 

  bidscoin:
    version: 4.3.1
    bidsignore: [extra_data/]
    subprefix: sub-
    sesprefix: ses-
    unknowntypes: [extra_data]
    ignoretypes: [exclude]
    unzip:
  plugins:
    dcm2niix2bids: 
      command: dcm2niix
      args: -b y -z n -i n
      anon: n
      meta: [.json, .tsv, .tsv.gz]

DICOM:

  subject: <<filepath:/sub-(.*?)/>>
  session: <<filepath:/sub-.*?/ses-(.*?)/>>

  func:
  - provenance:
    attributes:
      SeriesDescription: (?i)^func.*
    bids:
      suffix: bold
      run: <<>>
      task: rest
    meta:
      TaskName: rest
      B0FieldSource:
      - magnitude1_<<session:[0:]>>
      - magnitude2_<<session:[0:]>>
      - phasediff_<<session:[0:]>>

  fmap:
  - provenance:
    attributes: &fmap_dicomattr
      ImageType:
      SeriesDescription:
      EchoNumbers:
    bids: &fmap_bidsattr
      suffix: magnitude1
      run: <<>>
    meta: &fmap_metaattr
      IntendedFor: <<task:[0:]>>
      B0FieldIdentifier: magnitude1_<<session:[0:]>>
  - provenance:
    attributes:
      <<: *fmap_dicomattr
      SeriesDescription: (?i)^fmap.*
      ImageType: .*'M'.*
    bids:
      <<: *fmap_bidsattr
      suffix: magnitude2  # Due to magnitude1 and magnitude2 images in the same series (Siemens). dcm2niix will deal with it correctly.
    meta:
      <<: *fmap_metaattr
      B0FieldIdentifier: magnitude2_<<session:[0:]>>
  - provenance:
    attributes:
      <<: *fmap_dicomattr
      SeriesDescription: (?i)^fmap.*
      ImageType: '[''ORIGINAL'', ''PRIMARY'', ''P'', ''ND'']'
    bids:
      <<: *fmap_bidsattr
      suffix: phasediff
    meta:
      <<: *fmap_metaattr
      B0FieldIdentifier: phasediff_<<session:[0:]>>

The limits of B0FieldSource are not updated in the final json files. I see the debug logs Updating the b0fieldtag ... for fmaps but not for funcs. Could the reason be the set of niifiles in

bidscoin/bidscoin/bidscoiner.py

Line 449 in 2a0dd3a

for niifile in niifiles:

For example, for B0FieldSource: magnitude2_<<session:[0:]>> in the template bidsmap, I get "B0FieldSource": "magnitude2_<<ses1:[0:]>>" in the json file.

A question about the result of <<session>> dynamic value: the part after underscore in, for example, <<ses01_1>> is the run id, right? The docs don't explain that part:

To achieve that you can add a special <<session>> dynamic value to your tags, which will be replaced with the session label during bidscoiner runtime.

It also seems I have a problem with our magnitude fmaps being in one sequence. I don't know how to get different identifiers for them. But that's most probably not related.

Please let me know what I can do.

[marcelzwiers]

Ai, that's a bummer. In the new code, I treat IntendeFor and the B0Field tags more or less in the same way. Is the IntendedFor still working for you? One (rare) omission is that B0FieldIdentifiers outside the fmap folder are missed (I'll add a warning for this). You say you have generated data that can produce this issue, can you share it with me? That helps me tremendously!

[marcelzwiers]

It also seems I have a problem with our magnitude fmaps being in one sequence. I don't know how to get different identifiers for them. But that's most probably not related.

You want to give them different tags, so that suggests you mean that you have magnitude images from two fieldmaps in one folder? I have never seen that (and I'm pretty sure Siemens would never do that), shouldn't they have different SeriesNumbers? If the two magnitude images in the same folder are from the same fieldmap, then you want to use the same tag, no?

[marcelzwiers]

For example, for B0FieldSource: magnitude2_<session:[0:]> in the template bidsmap, I get "B0FieldSource": "magnitude2_<ses1:[0:]>" in the json file.

This should not happen and looks like a bug

A question about the result of <> dynamic value: the part after underscore in, for example, <<ses01_1>> is the run id, right? The docs don't explain that part

Yes, for e.g. ses-01, then <<session>> is replaced with 01 (the session label). But if you add a range specifier, then a separating underscore is appended to the session label, followed by the run-index of the bidsmap. So <<session:[]>> for a run-2 bidsmap will result in 01_2

[marcelzwiers]

For example, for B0FieldSource: magnitude2_session:[0:] in the template bidsmap, I get "B0FieldSource": "magnitude2_ses1:[0:]" in the json file.

Or this is perhaps because the acquisition times in the DICOM header are not increasing with SeriesNumber?

[marcelzwiers]

B0FieldSource:
      - magnitude1_<<session:[0:]>>
      - magnitude2_<<session:[0:]>>
      - phasediff_<<session:[0:]>>

Oh wait, it can be a list of course, I keep forgetting that

p.s. A list is indeed valid according to the specifications but why do you do this instead of just using a single tag (which is how I think you should use the B0Field tags)? A single B0FieldIdentifier tag groups all files of one fieldmap (mag1, mag2 and phasediff).

Name: B0 Field Identifier
Type: Metadata
Description: The presence of this key states that this particular 3D or 4D image MAY be used for fieldmap estimation purposes. Each "B0FieldIdentifier" MUST be a unique string within one participant's tree, shared only by the images meant to be used as inputs for the estimation of a particular instance of the B0 field estimation.

[marcelzwiers]

Bounding terms for lists of B0FieldIdentifiers are much harder to support and I suspect it is going to be very very uncommon that users may want to use lists of tags here at all (I think you are using b0 field tags in a wrong way). I may add support for bounded B0FieldIdentifier lists in the future, but for now, I clarified the docs about this limitation

[mateuszpawlik]

Ai, that's a bummer. In the new code, I treat IntendeFor and the B0Field tags more or less in the same way. Is the IntendedFor still working for you?

Yes. For example:

"B0FieldIdentifier": "magnitude2_<<ses1_1>>",
"IntendedFor": [
    "bids::sub-001/ses-1/func/sub-001_ses-1_task-rest_run-1_bold.nii"
]

One (rare) omission is that B0FieldIdentifiers outside the fmap folder are missed (I'll add a warning for this).

I don't understand. Isn't it the case, that only the fmaps that need the B0FieldIdentifier?

You say you have generated data that can produce this issue, can you share it with me? That helps me tremendously!

Here's the DICOMs: b0_test_dicoms.zip I used my DICOm generator 😉

[mateuszpawlik]

It also seems I have a problem with our magnitude fmaps being in one sequence. I don't know how to get different identifiers for them. But that's most probably not related.

You want to give them different tags, so that suggests you mean that you have magnitude images from two fieldmaps in one folder? I have never seen that (and I'm pretty sure Siemens would never do that), shouldn't they have different SeriesNumbers? If the two magnitude images in the same folder are from the same fieldmap, then you want to use the same tag, no?

Our Siemens scanner generates one sequence (same series number) for both magnitude1 and magnitude2 (and one sequence for phasediff). dcm2niix and BIDScoin handle the magnitude maps correctly if I assign magnitude2 suffix to the sequence. The result are two nifti files. This is how we're dealing with it for a very long time.

I'll reply to the B0 identifier with another comment. As you wrote later, I most probably didn't understand it exactly.

[mateuszpawlik]

For example, for B0FieldSource: magnitude2_session:[0:] in the template bidsmap, I get "B0FieldSource": "magnitude2_ses1:[0:]" in the json file.

Or this is perhaps because the acquisition times in the DICOM header are not increasing with SeriesNumber?

In the resulting json files I see increasing AcquisitionTime values. I use the following to generate them:

ds.AcquisitionDate = datetime.today().strftime('%Y%m%d')
ds.AcquisitionTime = datetime.today().strftime('%H%M%S.%f')

[mateuszpawlik]

p.s. A list is indeed valid according to the specifications but why do you do this instead of just using a single tag (which is how I think you should use the B0Field tags)? A single B0FieldIdentifier tag groups all files of one fieldmap (mag1, mag2 and phasediff).

Name: B0 Field Identifier
Type: Metadata
Description: The presence of this key states that this particular 3D or 4D image MAY be used for fieldmap estimation purposes. Each "B0FieldIdentifier" MUST be a unique string within one participant's tree, shared only by the images meant to be used as inputs for the estimation of a particular instance of the B0 field estimation.

I missed that part. Thanks. I thought all identifiers must be unique. I'll fix that on my side.

I've also found a PR with similar explanation bids-standard/bids-specification#622

[mateuszpawlik]

Bounding terms for lists of B0FieldIdentifiers are much harder to support and I suspect it is going to be very very uncommon that users may want to use lists of tags here at all (I think you are using b0 field tags in a wrong way). I may add support for bounded B0FieldIdentifier lists in the future, but for now, I clarified the docs about this limitation

Do you mean the lists of identifiers in B0FieldIdentifier field. I find it odd and don't know why would you need more than one unique identifier. Maybe for different groupings? Could bids-standard/bids-specification#968 be the use case for that?

[marcelzwiers]

One (rare) omission is that B0FieldIdentifiers outside the fmap folder are missed (I'll add a warning for this).

I don't understand. Isn't it the case, that only the fmaps that need the B0FieldIdentifier?

Well, if for example you acquire the same amount of normal and of inverted dwi scans, you want to put both scans in the dwi folder, right? So the dwi scans are both data and fieldmap at the same time

[marcelzwiers]

Our Siemens scanner generates one sequence (same series number) for both magnitude1 and magnitude2 (and one sequence for phasediff). dcm2niix and BIDScoin handle the magnitude maps correctly if I assign magnitude2 suffix to the sequence. The result are two nifti files. This is how we're dealing with it for a very long time.

Yes, that's what I mean, on our scanners it is the same

[mateuszpawlik]

Another question. Are <<>> characters fine in the identifier string for yaml and BIDS?

[marcelzwiers]

Do you mean the lists of identifiers in B0FieldIdentifier field. I find it odd and don't know why would you need more than one unique identifier. Maybe for different groupings? Could bids-standard/bids-specification#968 be the use case for that?

Yes, indeed, I mean a list of B0FieldIdentifier tags. And the PR link you gave indicates that there are scenarios where one could think about using B0FieldIedntifiers lists, but it is very rare I think. For now I leave such scenarios unsupported (I will deal with it when someone asks for it, altough with the latest commits I already prepared the code a bit for such support)

[marcelzwiers]

Another question. Are <<>> characters fine in the identifier string for yaml and BIDS?

Yes, they are not allowed in the BIDS filenames, but I wouldn't know why they wouldn't be allowed inside a file (such as in the sidecar files)

[mateuszpawlik]

I've just tested 84be28f and it seems to works as expected.

One of my users said that the generated identifier string could be nicer 😉, for example instead of <<ses1_2>> maybe ses-1_run-2 (without the <<>>). But I leave the judgement to you 😄

[marcelzwiers]

I've just tested 84be28f and it seems to works as expected.

Great, I really appreciate you tested this

One of my users said that the generated identifier string could be nicer 😉, for example instead of <<ses1_2>> maybe ses-1_run-2 (without the <<>>). But I leave the judgement to you 😄

Yeah, I don't like it that much either, and in my initial implementation I didn't have the <<>> enclosings. But in the end, I decided to leave the enclosings in there, to avoid name collisions and to make it easier to retrospectively fix the label, in case things went wrong. Also ses-1_run-2 looks weird for session-less datasets (the string then becomes _run-2). I could perhaps use <<ses1_run2>> (but it is unclear from this which scan the 'run2' refers to, hence omitting the 'run' substring)

[marcelzwiers]

Btw, in the latest version you can also switch trackusage on or off by setting the environment variable BIDSCOIN_TRACKUSAGE. I really like the anonymous feedback that I get from trackusage, but I also like to filter out test usage. Pytest are already filtered out, but I was wondering if you are using bidscoin in automatic/CI non-pytest tests? If so, could you then use the new environment variable to filter out test usage? Thanks! I think I can close this issue now and publish a 4.3.1 release. Or are you planning to do more testing (if so, then I can wait a bit more)?

[mateuszpawlik]

Btw, in the latest version you can also switch trackusage on or off by setting the environment variable BIDSCOIN_TRACKUSAGE. I really like the anonymous feedback that I get from trackusage, but I also like to filter out test usage. Pytest are already filtered out, but I was wondering if you are using bidscoin in automatic/CI non-pytest tests? If so, could you then use the new environment variable to filter out test usage? Thanks!

Do you mean to switch it off when testing things? I can do that.

We are in the process of converting historical data and we execute BIDScoin (bidsmapper and bidscoiner) once for each subject session. That means hundreds of executions on real data. Should I suspend it there too?

I think I can close this issue now and publish a 4.3.1 release. Or are you planning to do more testing (if so, then I can wait a bit more)?

I don't know what else I could test, so it's fine for me to close it.

[marcelzwiers]

Do you mean to switch it off when testing things? I can do that.

Thanks!

We are in the process of converting historical data and we execute BIDScoin (bidsmapper and bidscoiner) once for each subject session. That means hundreds of executions on real data. Should I suspend it there too?

No, I consider that to be real usage

I don't know what else I could test, so it's fine for me to close it.

Great, I will then update a few read-the-docs files and publish the new release today or tomorrow :-)

[mateuszpawlik]

@marcelzwiers I've just converted some data without the fmap runs and got "B0FieldIdentifier": "fmap_<<ses1_>>". As a comment only (I didn't want to open an issue for that): maybe the trailing underscore after ses1 could be added with the run.