Assessment of BIDS compliance within Clinica

[ghisvail]

This discussion compiles the outcome of multiple assessment carried on Clinica to evaluate its BIDS compliance.

Clinica produces raw BIDS dataset and its own flavor of derivative datasets named CAPS. An ongoing goal for Clinica has been to improve its BIDS compliance to improve its interoperability with other BIDS compatible tools.

[ghisvail]

BIDS usage

Primitive obsession

Primitive obsession: overuse of primitive types to implement domain-specific functionality instead of composing them into reusable higher-level models and abstractions, see [primitive-obsession].

Most of Clinica's BIDS handling logic relies on bare string manipulation. As a result, fixing a BIDS-compliance issue means scanning the entire codebase to spot buggy raw string composition logic. Use of pathlib.Path may prevent some class of bugs in raw path manipulation, but not in handling BIDS naming scheme.

Another area where Clinica lacks homogeneity and abstraction is in reading and writing modality-agnostic BIDS files. BIDS specifies its own flavour of TSV, which our calls to pandas.read_csv and pandas.DataFrame.to_csv covers partially.

Solution: Provide a proper abstraction for BIDS dataset and filename operations and use it consistently throughout the codebase.

Shotgun parsing

Shotgun parsing: ad-hoc validation during processing, see [langsec-cwes-secdev2016].

Clinica is riddled with instances of shotgun parsing in its codebase. For BIDS management in particular, some instances are centralized within clinica.utils.inputs. There are functions (check_bids_folder, check_caps_folder) which partially validates BIDS or CAPS datasets given their root path. Those validators are then called again by reader functions (clinica_file_reader, clinica_list_of_files_reader) as well as in the engine.Pipeline base class. Those readers can also be called multiple times within a pipeline's build_input_node.

Solution: Leave input validation at the edge of the program and use record types providing stronger guarantees within the rest of the processing.

Inconsistent naming

The BIDS specification defines a glossary of terms, each modelling an important concept or notion. Clinica does not use these terms in a consistent manner, introducing confusion when reading part of the codebase.

For instance:

# in DWI preprocessing pipeline.

from clinica.utils.filemanip import get_subject_id


image_id = get_subject_id(dwi)
# image_id = "sub-<label>_ses-<label>"

Multiple issues here:

• The BIDS specification defines the subject ID as the sub-<label> entity. The name of the function suggests something different than what it actually returns.
• There are no definition of an image ID. The name of the variable to which the returned value is assigned does not correspond to a concept documented in the BIDS specification.
• Finally, the actual return value of the form <sub>-<label>_<ses-label> may correspond to the <source_entities> abbreviation found in selected examples in the BIDS specification, but the latter is not formally defined and only used in the derivatives section.

[omar-rifai]

two interesting upstream issues on BIDS derivative specifications:

• on template and group results
• on longitudinal data - specifically for freesurfer (old issue initiated by @alexandreroutier)

[alexandreroutier]

For templates, they developed https://www.templateflow.org/ which is used in fmriprep / dmriprep if I remember correctly. It covers "well-know" templates and I have the feeling that it can be the solution for custom templates like DARTEL. The only reminding thing to cover would to cover the provenance of the sujets (currently, it is simply a TSV file with participant_id / session_id columns)

For statistics, there is BEP 2 - The BIDS Stats Models Specification but I never had the chance to read it. Hope it helps.

[ghisvail]

BIDS compliance

BIDS datasets

Some converters produce a conversion_info folder with additional data. The BIDS specification discourages storing non-specified information inside raw BIDS datasets.

e.g OASIS:

• Use of the session entity (ses-M00) for a single-session dataset.

CAPS dataset

BIDS derivative specification

Quoted verbatim from version 1.6.0 of the specification.

Storage of derived datasets

A data processing pipeline will typically have a dedicated directory under which it stores all of its outputs. Different components of a pipeline can, however, also be stored under different subfolders. There are few restrictions on the directory names; it is RECOMMENDED to use the format <pipeline>-<variant> in cases where it is anticipated that the same pipeline will output more than one variant (for example, AFNI-blurring and AFNI-noblurring). For the sake of consistency, the subfolder name SHOULD be the GeneratedBy.Name field in data_description.json, optionally followed by a hyphen and a suffix (see Derived dataset and pipeline description).

Every derivatives dataset is considered a BIDS dataset and must include a dataset_description.json file at the root level (see Dataset description. Consequently, files should be organized to comply with BIDS to the full extent possible (that is, unless explicitly contradicted for derivatives). Any subject-specific derivatives should be housed within each subject’s directory; if session-specific derivatives are generated, they should be deposited under a session subdirectory within the corresponding subject directory; and so on.

Metadata conventions

• Unless specified otherwise, individual sidecar JSON files and all metadata fields within are OPTIONAL. However, the appropriate use of these files and pertinent fields is very valuable and thus encouraged. Moreover, for some types of files, there may be one or more required metadata fields, in which case at least one metadata file containing that field must be located somewhere within the file’s hierarchy (per the Inheritance Principle).
• When chaining derivative pipelines, any JSON fields that were specified as mandatory in the input files SHOULD be propagated forward in the output file’s JSON provided they remain valid. Non-required JSON fields MAY be propagated, and are highly useful, but it is the pipeline’s responsibility to ensure that the values are still relevant and appropriate to the type of output data.

File naming conventions

• Filenames that are permissible for a raw BIDS data type have a privileged status. Any modification of raw files must use a modified filename that does not conflict with the raw filename. Further, any files created as part of a derivative dataset must not match a permissible filename of a valid raw dataset. Stated equivalently, if any filename in a derivative dataset has a name permissible for a raw BIDS data, then that file must be an identical copy of that raw file.
• Each Derivatives filename MUST be of the form: <source_entities>[_keyword-<value>]_<suffix>.<ext> (where <value> could either be an <index> or a <label> depending on the keyword; see Definitions)
• When the derivatives chain involves outputs derived from a single raw input, source_entities MUST be the entire source filename, with the omission of the source suffix and extension. One exception to this rule is filename entities that are no longer relevant. Depending on the nature of the derivative file, the suffix can either be the same as the source file if that suffix is still appropriate, or a new appropriate value selected from the controlled list.
• There is no prohibition against identical filenames in different derived datasets, although users should be aware of the potential ambiguity this can create and use the sidecar JSON files to detail the specifics of individual files.
• When necessary to distinguish two files that do not otherwise have a distinguishing entity, the _desc-<label> keyword-value SHOULD be used. This includes the cases of needing to distinguish both differing inputs and differing outputs (for example, _desc-T1w and _desc-T2w to distinguish brain mask files derived from T1w and T2w images; or _desc-sm4 and _desc-sm8 to distinguish between outputs generated with two different levels of smoothing).
• When naming files that are not yet standardized, it is RECOMMENDED to use names consistent with BIDS conventions where those conventions apply. For example, if a summary statistic is derived from a given task, the file name SHOULD contain _task-<label>.

[ghisvail]

Good news, the derivatives section of BIDS version 1.7.0 include more guidelines and proposals:

• Data types and metadata
• Imaging data types

[ghisvail]

T1 linear

Current specification

subjects
└─ <participant_id>/
    └─ <session_id>/
      └─ t1_linear/
         ├─ <source_file>_space-MNI152NLin2009cSym_res-1x1x1_affine.mat
         ├─ <source_file>_space-MNI152NLin2009cSym_res-1x1x1_T1w.nii.gz
         └─ <source_file>_space-MNI152NLin2009cSym_desc-Crop_res-1x1x1_T1w.nii.gz

Motives for non-compliance

• Superfluous space and res entities.
• desc entity not used consistently

Proposal

t1linear
└─ <participant_id>/
    └─ <session_id>/
      └─ xfm/
         ├─ sub-<label>_ses-<label>_from-MNI152NLin2009cSym_to-T1w_mode-image_xfm.mat
      └─ anat/
         ├─ sub-<label>_ses-<label>_space-MNI152NLin2009cSym_desc-warped_T1w.nii.gz
         └─ sub-<label>_ses-<label>_space-MNI152NLin2009cSym_desc-cropped_T1w.nii.gz

[ghisvail]

I wonder whether we could even keep the anat datatype here, since they are still anat/T1w images at the end of the day.

Answering my own question here: yes, the spec suggests to keep the datatype / suffix combo for output derived from a single raw input. See examples.

To summarize, in this simple case, we just have to copy the spec of the corresponding modality and specify our custom entities to distinguish the raw input filename for the output ones.

So the proposal above would become:

t1linear
└─ <participant_id>/
    └─ <session_id>/
      └─ xfm/
         ├─ sub-<label>_ses-<label>_from-MNI152NLin2009cSym_to-T1w_mode-image_xfm.mat
      └─ anat/
         ├─ sub-<label>_ses-<label>_space-MNI152NLin2009cSym_desc-warped_T1w.nii.gz
         └─ sub-<label>_ses-<label>_space-MNI152NLin2009cSym_desc-cropped_T1w.nii.gz

Modulo the order to the space and desc entities.

[ghisvail]

Also covered by this section.

[omar-rifai]

Updated the proposal to reflect these new elements

[ghisvail]

I have asked for confirmation whether BEP014 can be considered stable at it is (at least for the naming convention part).

bids-standard/bids-specification#1042

[ghisvail]

I guess we can call BEP014 as mature enough to store existing software files.

One less thing to worry about 🥳

[ghisvail]

T1 volume

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ t1/
         └─ spm/
            └─ dartel/
               └─ <group_id>/
                  ├─ <source_file>_target-<group_label>_transformation-forward_deformation.nii.gz
                  └─ <source_file>_segm-<segm>_space-Ixi549Space_modulated-{on|off}[_fwhm-<X>mm]_probability.nii.gz
                  └─ atlas_statistics/
                     └─ <source_file>_space-<space>_map-graymatter_statistics.tsv
            └─ segmentation/
               ├─ normalized_space/
               │  ├─ <source_file>_target-Ixi549Space_transformation-{inverse|forward}_deformation.nii.gz
               │  ├─ <source_file>_segm-<segm>_space-Ixi549Space_modulated-{on|off}_probability.nii.gz
               │  └─ <source_file>_space-Ixi549Space_T1w.nii.gz
               ├─ native_space/
               │  └─ <source_file>_segm-<segm>_probability.nii.gz
               └─ dartel_input/
                  └─ <source_file>_segm-<segm>_dartelinput.nii.gz

groups/
└─ <group_id>/
   ├─ <group_id>_subjects_visits_list.tsv
   └─ t1/
      ├─ <group_id>_iteration-<index>_template.nii.gz
      └─ <group_id>_template.nii.gz

Motives for non-compliance

• Superfluous nesting.
• Unclear mapping between group concept and other source entities.
• Superfluous target and space entities, do we really need those?
• Do we really need the modulated entity? Sounds like metadata.

Proposal

t1volume
└─ <participant_id>/
   └─ <session_id>/
      └─ t1volume
          ├─ <source_entities>_target-<group_label>_transformation-forward_deformation.nii.gz
          ├─ <source_entities>_segm-<segm>_space-Ixi549Space_modulated-{on|off}[_fwhm-<X>mm]_probability.nii.gz
          ├─ <source_entities>_space-<space>_map-graymatter_statistics.tsv
          ├─ <source_entities>_target-Ixi549Space_transformation-{inverse|forward}_deformation.nii.gz
          ├─ <source_entities>_segm-<segm>_space-Ixi549Space_modulated-{on|off}_probability.nii.gz
          ├─ <source_entities>_space-Ixi549Space_T1w.nii.gz
          ├─ <source_entities>_segm-<segm>_probability.nii.gz
          └─ <source_entities>_segm-<segm>_dartelinput.nii.gz

[omar-rifai]

so xfm as a suffix? In which case would that fit in for this example?

[ghisvail]

I am guessing deformation -> xfmin our case.

[ghisvail]

We'd have something like:

xfm/
    sub-<label>_ses-<label>_from-Ixi549Space_to-T1w_mode-image_xfm.mat  # forward
    sub-<label>_ses-<label>_from-T1w_to-Ixi549Space_mode-image_xfm.mat  # inverse

And then the space entity is justified to make an entity link.

[alexandreroutier]

• targetshould be replaced by to-<space_label> and `from-<space_label> defined in BEP014 to indicate a transformation from a "space" to another
• transformation-{inverse|forward} becomes superfluous if we use toand from
• modulated is used in Clinica to indicate whether "modulation has been used in SPM to compensate for the effect of spatial normalization." (c.f documentation). This would likely need to go in a sidecar json file (no BEP)
• Replace probability suffix by probseg as defined in BEP011
• segm entity should be replaced by label as defined in BEP011 (??) - (I'd propose we push segminstead as it seems less generic
• fwhm has no equivalent in BIDS and _desc-sm4 and _desc-sm8 were proposed (@alexandreroutier ) to distinguish between outputs generated with two different levels of smoothing

+1 for storing modulation into JSON file. Default parameter for modulation was (almost?) always used within the lab, I don't think external users are using it. It would be the opportunity to also store some CLI parameters into JSON files. The one I have in mind is the Tissue Probability Map.

Regarding desc-sm<N>, this could do the job in volume/surface pipelines as well as "derivatives of derivates" pipelines like stats-surface, stats-volume or ml-prepare-data 🤔

[ghisvail]

Here is an update to the proposal, ignoring the group thing for now:

t1volume
└─ sub-<label>/
   └─ ses_<label>/
      └─ anat
          └─ [<source_entities>]_space-Ixi549Space_T1w.nii.gz
          └─ [<source_entities>]_space-Ixi549Space_T1w.json
          └─ [<source_entities>]_space-Ixi549Space_label-<region>_probseg.nii.gz
          └─ [<source_entities>]_space-Ixi549Space_label-<region>_probseg.json
          └─ [<source_entities>]_space-orig_label-<region>_probseg.nii.gz
          └─ [<source_entities>]_space-orig_label-<region>_probseg.json
       └─ xfm
          └─ [<source_entities>]_from-Ixi549Space_to-orig_mode-image_xfm.nii.gz
          └─ [<source_entities>]_from-Ixi549Space_to-orig_mode-image_xfm.json
          └─ [<source_entities>]_from-orig_to-Ixi549Space_mode-image_xfm.nii.gz
          └─ [<source_entities>]_from-orig_to-Ixi549Space_mode-image_xfm.json

Unclassified yet:

└─ sub-<label>/
   └─ ses_<label>/
      └─ anat
          └─ [<source_entities>]_space-Ixi549Space_label-<region>_modulated-{on|off}_fwhm-<X>mm_probseg.nii.gz
          └─ <source_entities>_space-<space>_label-<region>_statistics.tsv
          └─ [<source_entities>]_label-<region>_dartelinput.nii.gz

^ do we really need those?

• Statistics could be stored in respective sidecar JSON for each label I guess?
• Are there more than one instance of probseg generated per label at each pipeline run? If so, modulation and FWHM metadata should be stored inside the sidecar JSON instead.
• What is dartelinput useful for? Find a more suitable suffix. Perhaps an xfm actually?

[ghisvail]

T1 FreeSurfer (and longitudinal)

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ t1/
         └─ <long_id>/
            └─ freesurfer_longitudinal/
               └─ <participant_id>_<session_id>.long.<participant_id>_<long_id>/
                  ├─ label/
                  ├─ mri/
                  ├─ scripts/
                  ├─ stats/
                  └─ surf/
               └─ regional_measures/
                  ├─ <participant_id>_<session_id>_<long_id>_parcellation-wm_volume.tsv
                  ├─ <participant_id>_<session_id>_<long_id>_segmentationVolumes.tsv
                  └─ <participant_id>_<session_id>_<long_id>_hemi-{left|right}_parcellation-<parcellation>_thickness.tsv

Motives for non-compliance

• Superfluous nesting
• The long prefix for the longitudinal ID is probably not explicit enough.
• Where should the FS specific output go for maximum BIDS compliance?

Proposal

T1FreeSurferLongi
└─ <participant_id>/
   └─ <longitudinal_id>/
      ├─ <source_entities>_<longitudinal_id>_parcellation-wm_volume.tsv
      ├─ <source_entities>_<longitudinal_id>_segmentationVolumes.tsv
      ├─ <source_entities>_<longitudinal_id>_hemi-{L | R}_parcellation-<parcellation>_thickness.tsv
      └─ <source_entities>_<longitudinal_id>_freesurfer/
         ├─ label/
         ├─ mri/
         ├─ scripts/
         ├─ stats/
         └─ surf/

[omar-rifai]

• Parcellation information (discrete surface segmentation) seems to need to transition to giftii files as per spec.
• FS specific output to be moved to folder with "BIDS file compliant naming convention"

[ghisvail]

DWI connectome

Current specification

subjects/
└─ <participant_id>/
    └─ <session_id>/
        └─ dwi/
            └─ connectome_based_processing/
                ├─ <source_file>_space-{b0|T1w}_model-CSD_diffmodel.nii.gz
                │─ <source_file>_space-{b0|T1w}_model-CSD_tractography.tck
                └─ <source_file>_space-{b0|T1w}_model-CSD_parcellation-{desikan|destrieux}_connectivity.tsv

Motives for non-compliance

• Superfluous nesting.
• Superfluous model entity, always set to CSD.
• Do we really need the parcellation entity? Could it be placed in JSON metadata instead?

Proposal

dwi-connectome/
└─ <participant_id>/
    └─ <session_id>/
        └─ dwi-connectome/
              ├─ <source_entities>_space-{b0 | T1w}_model-CSD_diffmodel.nii.gz
              │─ <source_entities>_space-{b0 | T1w}_model-CSD_tractography.tck
              └─ <source_entities>_space-{b0 | T1w}_model-CSD_parcellation-{desikan | destrieux}_connectivity.tsv

[omar-rifai]

• BEP016 seems to recommend the model be specified as "suffix" (e.g bs, csd, or tensor)
• parcellation information as in T1Freesurfer?

[ghisvail]

I read BEP016 and agree that model should be the suffix, since it defines the nature of the recording within the NIfTI. We'd probably need to think further about the parcellation and tractography part. It depends whether they are considered contextual metadata to the model (like bvec / bval for dwi) or separate recordings which could be queried independently.

[ghisvail]

DWI preprocessing

Current specification

subjects/
﹂ <participant_id>/
   ﹂ <session_id>/
      ﹂ dwi/
         ﹂ preprocessing/
            ﹂ <participant_id>_<session_id>_dwi_space-<space>_brainmask.nii.gz
            ﹂ <participant_id>_<session_id>_dwi_space-<space>_preproc.bval
            ﹂ <participant_id>_<session_id>_dwi_space-<space>_preproc.bvec
            ﹂ <participant_id>_<session_id>_dwi_space-<space>_preproc.nii.gz

Motives for non-compliance

• Superfluous nesting
• Multiple suffixes (dwi, preproc)
• Is the space entity necessary?

Proposal

dwipreproc
﹂sub-<label>/
    ﹂ses-<label>/
        ﹂dwipreproc/
           ﹂[<source_entities>]_space-<label>_dwi.nii.gz
           ﹂[<source_entities>]_space-<label>_brainmask.nii.gz

[alexandreroutier]

I notice discrepency between command line name (dwi-preprocessing*) and resulting folder (dwipreproc without ing).

Is consistancy planned between CLI names and resulting folders?

[alexandreroutier]

Multiple suffixes (dwi, preproc)

One important point to clarify: "Multiple suffixes" occurs in any CAPS dataset. Given a BIDS file:

sub-<label>_ses-<label>[_acq-<label>][_ce-<label>][_rec-<label>][_run-<index>]_T1w.nii.gz
sub-<label>_ses-<label>[_acq-<label>][_dir-<label>][_run-<index>]_dwi.nii.gz
sub-<label>_ses-<label>[_task-<label>][_trc-<label>][_rec-<label>][_run-<index>]_pet.nii.gz

<source_file> in CAPS page is:

sub-<label>_ses-<label>[_acq-<label>][_ce-<label>][_rec-<label>][_run-<index>]_T1w
sub-<label>_ses-<label>[_acq-<label>][_dir-<label>][_run-<index>]_dwi
sub-<label>_ses-<label>[_task-<label>][_trc-<label>][_rec-<label>][_run-<index>]_pet

whereas <source_entities> in BIDS-Derivatives is I think:

sub-<label>_ses-<label>[_acq-<label>][_ce-<label>][_rec-<label>][_run-<index>]
sub-<label>_ses-<label>[_acq-<label>][_dir-<label>][_run-<index>]
sub-<label>_ses-<label>[_task-<label>][_trc-<label>][_rec-<label>][_run-<index>]

My asumption is based on this thread from 2018 explaining the final convention in BIDS-Derivatives.

[ghisvail]

Is consistancy planned between CLI names and resulting folders?

Reaching compliance is our top priority, consistency is secondary.

Current BIDS standards recommend the <pipeline>[-<variant>] naming scheme, with <pipeline> being alphanumerical. But it could be anything users specify as an output BIDS directory at the end of the day.

When we reach the stage where our sole issue is the naming of the output directory, we will consider ourselves pretty happy. We are only at the very beginning of this journey :-)

[ghisvail]

One important point to clarify: "Multiple suffixes" occurs in any CAPS dataset. Given a BIDS file:

You are correct. We have identified a number of issues common to all pipelines, such as:

• Multiple suffixes
• Superfluous entities
• Superfluous nesting
• etc...

whereas <source_entities> in BIDS-Derivatives is I think:

Indeed, <source_entities> is basically whatever prefixed the raw data suffix used to process the derivative.

[alexandreroutier]

• replace brainmask with mask
• use dwias suffix instead of preproc for bval and bvec as in example

From https://bids-specification.readthedocs.io/en/stable/05-derivatives/02-common-data-types.html:

Preprocessed bold NIfTI file in the original coordinate space of the original run. The location of the file in the original datasets is encoded in the RawSources metadata, and _desc- is used to prevent clashing with the original filename.

From https://bids-specification.readthedocs.io/en/stable/05-derivatives/02-common-data-types.html#preprocessed-or-cleaned-data:

<pipeline_name>/
   sub-<label>/
       <datatype>/
           <source_entities>[_space-<space>][_desc-<label>]_<suffix>.<ext>

The BIDS-Derivatives version could possibly be:

<dwipreproc>/
    sub-<label>/
        ses-<label>/
            dwi/
                <source_entities>_space-<space>_desc-preproc_dwi.{bval|bvec|nii.gz}
                <source_entities>_space-<space>_desc-preproc_mask.nii.gz

🤔
Whatever the final decision, the rationale should be applied for pet-volume in my opinion.

[ghisvail]

DWI DTI

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ dwi/
         └─ dti_based_processing/
            ├─ native_space/
            │  ├─ <source_file>_space-<space>_model-DTI_diffmodel.nii.gz
            │  ├─ <source_file>_space-<space>_{FA|MD|AD|RD}.nii.gz
            │  └─ <source_file>_space-<space>_DECFA.nii.gz
            │─ normalized_space/
            │  ├─ <source_file>_space-MNI152Lin_res-1x1x1_affine.mat
            │  ├─ <source_file>_space-MNI152Lin_res-1x1x1_deformation.nii.gz
            │  └─ <source_file>_space-MNI152Lin_res-1x1x1_{FA|MD|AD|RD}.nii.gz
            └─ atlas_statistics/
               └─ <source_file>_space-<space>_res-1x1x1_map-{FA|MD|AD|RD}_statistics.tsv

Motives for non-compliance:

• Superfluous nesting
• Superfluous res entity used, only value is 1x1x1.
• Superfluous model entity used, only value is DTI.
• Overuse of BIDS suffixes (FA, MD, AD, RD, DECFA)
  Proposal: Introduce a metric-<label> entity with a _dti or _map suffix instead?
• Superfluous nesting, likely because of some implicit metamodel?
  Proposal: Move everything under a single pipeline directory, say dwi-dti

Proposal

dti/
└─ <participant_id>/
   └─ <session_id>/
      └─ dti/
         ├─ <source_entities>_space-<b0 | T1w>_model-DTI_diffmodel.nii.gz
         ├─ <source_entities>_space-<b0 | T1w>_metric-<FA | MD | AD | RD | DECFA>_map.nii.gz
         ├─ <source_entities>_space-MNI152Lin_affine.mat
         ├─ <source_entities>_space-MNI152Lin_deformation.nii.gz
         ├─ <source_entities>_space-MNI152Lin_metric-<FA | MD | AD | RD | DECFA>_map.nii.gz
         └─ <source_entities>_space-<space>_metric-<FA | MD | AD | RD>_statistics.tsv

[omar-rifai]

• model entity needs to be replaced by tensorsuffix
• use the entity parameter with labels (FA | MD | AD ..) as per BEP016

[alexandreroutier]

Long story short, except statistics.tsv and deformation files, it followed BEP016 when it was a Google Doc.

Latest version provided by @omar-rifai shows that there were a lot of changes. However, it seems to be easy to convert (*space-<space>_parameter-{fa|md|ad|rd}_tensor.nii.gz+ *_space-<space>_tensor.nii.gz).

[ghisvail]

PET volume

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ pet/
         └─ preprocessing/
            └─ atlas_statistics/
                └─ <source_file>_space-<space>[_pvc-rbv]_suvr-<suvr>_statistics.tsv
            └─ <group_id>/
               ├─ <source_file>_space-T1w_pet.nii.gz
               ├─ <source_file>_space-T1w[_pvc-rbv]_pet.nii.gz
               ├─ <source_file>_space-Ixi549Space[_pvc-rbv]_pet.nii.gz
               ├─ <source_file>_space-Ixi549Space[_pvc-rbv]_suvr-<suvr>_pet.nii.gz
               ├─ <source_file>_space-Ixi549Space_brainmask.nii.gz
               └─ <source_file>_space-Ixi549Space[_pvc-rbv]_suvr-<suvr>_mask-brain_pet.nii.gz

Motives for non-compliance

• Superfluous nesting
• Multiple suffixes
• pvc and suvr entities used inconsistently
• suvr entitiy values could be shortened perhaps?

Proposal

petvolume/
└─ <participant_id>/
   └─ <session_id>/
      └─ petvolume/
        ├─ <source_entities>_space-<label>[_pvc-rbv][_suvr-<pons | cerebellumPons>]_statistics.tsv
        ├─ <source_entities>_group-<label>_space-<label>[_pvc-rbv][_suvr-<pons | cerebellumPons>][_mask-brain]_pet.nii.gz
        └─ <source_entities>_group-<label>_space-<label>_mod-pet_brainmask.nii.gz

Could we use shorter labels for the suvr entity?

[omar-rifai]

• pvc and suvr are not officially in the specs
• pvc is mentioned in BEP023
• No mention of suvr (leave a comment?)

[omar-rifai]

Does mask replace the brainmask suffix we use?

[ghisvail]

Does mask replace the brainmask suffix we use?

Yes as per this section.

[ghisvail]

PET surface

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ pet/
         └─ surface/
            ├─ atlas_statistics/
            │  └─ <source_file>_task-<label>_acq-<label>_pet_space-<space>_pvc-iy_suvr-<suvr>_statistics.tsv
            ├─ <source_file>_hemi-{left|right}_midcorticalsurface
            └─ <source_file>_hemi-{left|right}_task-rest_acq-<label>_pet_space-<space>_suvr-<suvr>_pvc-iy_hemi-{left|right}_fwhm-<label>_projection.mgh

Motives for non-compliance

• <source_file> includes source modality suffix.
• Superfluous nesting, consider flattening the hierarchy.
• Task entity likely in <source_entities> already, superfluous otherwise.
• Acquisition entity should be replaced by the PET tracer one.
• Duplicate hemisphere entity, likely in <source_entities> already, otherwise should be appended next.
• Inconsistent ordering of entities.
• Do we really need all these entities? FWHM looks like something that could be in a JSON sidecar instead.

Proposal

petsurface/
└─ <participant_id>/
   └─ <session_id>/
      └─ petsurface/
        ├─ <source_entities>_hemi-<left | right>_midcorticalsurface.mgh
        ├─ <source_entities>_space-<space_label>_pvc-iy_suvr-<suvr_label>_statistics.tsv
        └─ <source_entities>_space-<space_label>_pvc-iy_suvr-<suvr_label>_hemi-<left | right>_fwhm-<fwhm_label>_projection.mgh

[omar-rifai]

• fwhm has no equivalent in BIDS and _desc-sm4 and _desc-sm8 were proposed to distinguish between outputs generated with two different levels of smoothing
• what to do with the projection? (there doesn't seem to be an equivalent in BEPs)

[ghisvail]

Projection does not express the nature of the recording. I suspect they are also some form of discrete segmentation. If that's the case, then we have got the dseg suffix for that as per BIDS v1.7.0.

[ghisvail]

Statistics surface

Current specification

groups/
└─ <group_id>/
   └─ statistics/
      ├─ participants.tsv
      └─ surfstat_group_comparison/
         ├─ <group_id>_<group_1>-lt-<group_2>_measure-<measure>_fwhm-<label>_correctedPValue.jpg
         ├─ <group_id>_<group_2>-lt-<group_1>_measure-<measure>_fwhm-<label>_correctedPValue.jpg
         ├─ <group_id>_<group_1>-lt-<group_2>_measure-<measure>_fwhm-<label>_correctedPValue.mat
         ├─ <group_id>_<group_2>-lt-<group_1>_measure-<measure>_fwhm-<label>_correctedPValue.mat
         ├─ <group_id>_participants.tsv
         ├─ <group_id>_output.log
         └─ <group_id>_glm.json

groups/
└─ <group_id>/
   └─ statistics/
      ├─ participants.tsv
      └─ surfstat_correlation_analysis/
         ├─ <group_id>_correlation-<label>_contrast-{negative|positive}_measure-<measure>_fwhm-<label>_correctedPValue.jpg
         ├─ <group_id>_correlation-<label>_contrast-{negative|positive}_measure-<measure>_fwhm-<label>_correctedPValue.mat
         ├─ <group_id>_output.log
         ├─ <group_id>_participants.tsv
         └─ <group_id>_glm.json

Motives for non-compliance

• Superfluous nesting, consider flattening the hierarchy.
• Malformed "hypothesis" entity, use of multiple separators.
• Confusing metamodel going on here, why are group IDs repeated?

Proposal

Requires a proper rethinking of the pipeline's output model and how to map it to BIDS concepts.

[ghisvail]

Statistics volume

Current specification

groups/
└─ <group_id>/
   ├─ <group_id>_participants.tsv
   └─ statistics_volume/
      └─ group_comparison_measure-{graymatter|fdg|av45|<custom_user>}/
         ├─ <group_id>_{RPV|mask}.nii
         ├─ <group_id>_covariate-<covariate>_measure-<label>_fwhm-<n>_regressionCoefficient.nii
         ├─ <group_id>_<group_1>-lt-<group_2>_measure-<label>_fwhm-<n>_{TStatistics|contrast}.nii
         ├─ <group_id>_<group_2>-lt-<group_1>_measure-<label>_fwhm-<n>_{TStatistics|contrast}.nii
         ├─ <group_id>_report-1.png
         ├─ <group_id>_report-2.png
         └─ <group_id>_{mask|RPV|VarianceError}.nii

Motives for non-compliance

• Same as in statistics-surface.
• Invalid report suffix.
• Imaging files stored in non-compressed form.

Proposal

Requires a proper rethinking of the pipeline's output model and how to map it to BIDS concepts.

[ghisvail]

Spatial SVM

Current specification

subjects/
└─ <participant_id>/
   └─ <session_id>/
      └─ machine_learning/
         └─ input_spatial_svm/
            └─ <group_id>/
               ├─ <source_file_t1w>_segm-{graymatter|whitematter|csf}_space-Ixi549Space_modulated-on_spatialregularization.nii.gz
               └─ <source_file_pet>_space-Ixi549Space[_pvc-rbv]_suvr-<suvr>_spatialregularization.nii.gz

groups/
└─ <group_id>/
   └─ machine_learning/
      └─ input_spatial_svm/
         ├─ <group_id>_space-Ixi549Space_gram.npy
         └─ <group_id>_space-Ixi549Space_parameters.json

Motives for non-compliance

• Superfluous nesting, consider flattening the hierarchy.
• Superfluous space entity. Consider moving this piece of metadata to the JSON sidecar.
• Superfluous modulated entity. Consider moving this piece of metadata to the JSON sidecar.
• Superfluous new suffix definition, maybe re-use source suffix and introduce a custom entity.
• Group entity relationship with source subject / session entities is ambiguous or not explicit.

Proposal

Requires a proper rethinking of the pipeline's output model and how to map it to BIDS concepts.

[ghisvail]

Content of a statistics file

Current specification

<source_entities>_space-<space>_map-<map>_statistics.tsv

Areas for improvement

• The proposed parametric mapping entity uses the prefix name map which is quite generic and could be confused with other mapping processes in future modalities covered by the BIDS raw dataset spec. The specification explicitly warns us about custom entity naming. Perhaps using pmap or metric would guard us against that whilst being more explicit as well.
• Since statistics are linked with some modality, maybe use the mod entity to link it with a T1w, PET or DTI-based parametric map.
• Perhaps use stats as the suffix to reduce the file name length. Looks non-ambiguous and well established.

[omar-rifai]

ADNI:

• Use of the acq-<acquisition_label> entity for the dwi modality is superfluous, since no other acquisition label than axial is used within the dataset.

• Must use the trc-<tracer_label> entity to distinguish scans acquired with different PET tracers. The BIDS specification for PET recommends 11CPIB (pib), 18FAV45 (av45), 18FAV1451 (tau), 18FFBB (fbb) and 18FFDG (fdg) as tracer labels.

[ghisvail]

PET non-compliance is addressed in #593.

[ghisvail]

DWI non-compliance is addressed in #602.

[omar-rifai]

AIBL:

• Use of the task-<task_label> entity for the pet modality is superfluous, since no task-based fMRI is provided in the dataset.

• Same comment regarding the tracer entity. Proposed label for FLUTE: 18FFMM.

[ghisvail]

Addressed by #593