Skip to content

add code to create raw tsv and make a markdown table + add submodules #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Nov 8, 2023
Merged

add code to create raw tsv and make a markdown table + add submodules #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a3c0642
[DATALAD] Added subdataset
marcobarilari Oct 20, 2023
09a262a
[DATALAD] Added subdataset
marcobarilari Oct 20, 2023
c476e4d
[DATALAD] Save updated subdatasets
marcobarilari Oct 20, 2023
07f8060
add code to create raw tsv and make a markdown table
marcobarilari Oct 20, 2023
18f94a8
add gitignore
marcobarilari Oct 27, 2023
565d8f5
Apply suggestions from code review
marcobarilari Oct 27, 2023
638831e
rm __pycahce__ folder
marcobarilari Oct 27, 2023
80364bf
Merge remote-tracking branch 'cpp-lln-lab/marco_create-tsv' into marc…
marcobarilari Oct 27, 2023
9795726
add pre commit
Remi-Gau Nov 8, 2023
e8ec852
ignore cache
Remi-Gau Nov 8, 2023
c7ab0d8
'Refactored by Sourcery'
Nov 8, 2023
bf3e10e
Merge pull request #9 from cpp-lln-lab/sourcery/marco_create-tsv
Remi-Gau Nov 8, 2023
abc025e
update table
Remi-Gau Nov 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__pycache__
8 changes: 8 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[submodule "cpp-lln-lab_raw"]
path = cpp-lln-lab_raw
url = https://github.com/cpp-lln-lab/cpp-lln-lab_raw.git
datalad-url = https://github.com/cpp-lln-lab/cpp-lln-lab_raw.git
[submodule "cpp-lln-lab_derivatives"]
path = cpp-lln-lab_derivatives
url = https://github.com/cpp-lln-lab/cpp-lln-lab_derivatives.git
datalad-url = https://github.com/cpp-lln-lab/cpp-lln-lab_derivatives.git
15 changes: 15 additions & 0 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: check-yaml
- id: check-added-large-files
- id: end-of-file-fixer

- repo: https://github.com/psf/black
rev: 23.10.1
hooks:
- id: black
1 change: 1 addition & 0 deletions cpp-lln-lab_derivatives
Open in desktop
Submodule cpp-lln-lab_derivatives added at 629c65
1 change: 1 addition & 0 deletions cpp-lln-lab_raw
Open in desktop
Submodule cpp-lln-lab_raw added at 54fdad
6 changes: 3 additions & 3 deletions mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ site_name: CPP Lab datasets
repo_url: https://github.com/cpp-lln-lab/Datasets.git

nav:
- CPP Lab raw data:
- Home: README.md
- Contributing: CONTRIBUTING.md
- Home: README.md
- Contributing: CONTRIBUTING.md
- Raw datasets table: datasets_raw.md

theme:
name: material
Expand Down
2 changes: 1 addition & 1 deletion src/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@ Serving on http://127.0.0.1:8000/
```

- push your changes to your forked repository
- open a pull request
- open a pull request
25 changes: 25 additions & 0 deletions src/datasets_raw.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
| name | nb_subjects | modalities | sessions | tasks | fmriprep | freesurfer | mriqc |
|:----------------------------------------|--------------:|:-------------------------|:-------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------|:-------------|:--------|
| 2008_Montreal_BlindRestingState_OC_raw | 50 | ['anat', 'func'] | [] | ['rest'] | n/a | n/a | n/a |
| 2012_Montreal_AudVisMotion_GD_raw | 35 | ['anat', 'func'] | [] | ['audMotion', 'visMotion'] | n/a | n/a | n/a |
| 2014_Toronto_CataractsDiffusion_Gao_raw | 26 | ['anat', 'dwi'] | ['1'] | n/a | n/a | n/a | n/a |
| 2015_Trento_Diffusion_SM_raw | 30 | ['anat', 'dwi', 'func'] | [] | ['unknown'] | n/a | n/a | n/a |
| 2015_Trento_MultimConn_SM_raw | 29 | ['anat', 'dwi', 'func'] | ['01'] | ['Catgs'] | n/a | n/a | n/a |
| 2015_Trento_categs_SM_raw | 53 | ['anat', 'dwi', 'func'] | [] | ['Catgs'] | n/a | n/a | n/a |
| 2016_Trento_Categs_MR_raw | 29 | ['anat', 'dwi', 'func'] | ['01'] | ['Catgs'] | n/a | n/a | n/a |
| 2016_Trento_MultimodalMotion_MR_raw | 23 | ['anat', 'func'] | ['01'] | ['audMotion', 'motDecoding', 'visMotion'] | n/a | n/a | n/a |
| 2016_Trento_SpatiotopyDir_CB_raw | 33 | ['anat', 'func'] | [] | ['pRF'] | n/a | n/a | n/a |
| 2018_LLN_FPAS_FB_raw | 16 | ['eeg'] | [] | ['FPAS'] | n/a | n/a | n/a |
| 2018_Toronto_Cataract_SM_raw | 46 | ['anat', 'func'] | [] | ['categ', 'categBlur1', 'categBlur2'] | n/a | n/a | n/a |
| 2019_Nancy_iEEG_FB_raw | 1 | ['ieeg'] | ['1', '2', '3', '4'] | ['ERPaudmot', 'ERPbraille', 'ERPcategory', 'FPSaudmot', 'FPSemotion', 'FPSface', 'FPSlexical', 'FPSvismot', 'FPSvoice', 'Readingbraille', 'Restingstate', 'Speechtracking', 'Stimulation'] | n/a | n/a | n/a |
| 2021_Dijon_BabyFPAS_RPC_raw | 23 | ['eeg'] | [] | ['FPAS'] | n/a | n/a | n/a |
| 2021_LLN_FVDE_FB_raw | 20 | ['eeg'] | ['001'] | ['fvde'] | n/a | n/a | n/a |
| 2021_LLN_emotion_ST_raw | 14 | ['eeg'] | [] | ['emotion'] | n/a | n/a | n/a |
| 2021_SaintLuc_TmsMT_FB_raw | 16 | ['anat', 'beh', 'func'] | ['mri', 'sham', 'tms'] | ['audInstrumentDiscrimination', 'audMotionDirDiscrimination', 'auditoryLocalizer', 'visColourDiscrimination', 'visMotionDirDiscrimination', 'visualLocalizer'] | n/a | n/a | n/a |
| 2021_SaintLuc_combiEmo_FF_raw | 24 | ['anat', 'func'] | ['01', '02'] | ['correction', 'eventrelatedCombiemoAuditory', 'eventrelatedCombiemoBimdal', 'eventrelatedCombiemoBimodal', 'eventrelatedCombiemoVisual', 'facelocalizerCombiemo', 'facelocalizerCombiemoCombiemo', 'voicelocalizer', 'voicelocalizerCombiemo'] | n/a | n/a | n/a |
| 2022_SaintLuc_lipSpeech_AA_raw | 2 | ['anat', 'func'] | ['01', '02', '03'] | ['MVPAAud', 'MVPAVis', 'PhonoLoc', 'VisLoc'] | n/a | n/a | n/a |
| 2023_Liege_BLAM_MB_raw | 14 | ['anat', 'func'] | ['01', '02', '03', '04'] | ['BimodalMotionAud', 'BimodalMotionVis', 'audioV1', 'auditoryLocalizer', 'bimodalMotionAud', 'bimodalMotionVis', 'mtMstLocalizer', 'rdkBimodalMotion', 'restingState', 'visualLocalizer'] | n/a | n/a | n/a |
| 2023_SaintLuc_VisTacMotionFoR_IS_raw | 21 | ['anat', 'func'] | ['001', '002'] | ['handDown', 'handUp', 'mtMstLocalizer', 'tactileLocalizer2', 'visual', 'visualLocalizer2'] | n/a | n/a | n/a |
| 2023_Trento_plosBiology_YX_raw | 48 | ['anat', 'fmap', 'func'] | [] | ['judgement', 'resting'] | n/a | n/a | n/a |
| Toronto_VisMotionLocalizer_MR_raw | 3 | ['anat', 'func'] | ['01'] | ['visMotion'] | n/a | n/a | n/a |
| olf_blind_raw | 35 | ['anat', 'dwi', 'func'] | [] | ['olfid', 'olfloc', 'rest'] | n/a | n/a | n/a |
24 changes: 24 additions & 0 deletions tools/datasets_raw.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name nb_subjects has_participant_tsv has_participant_json participant_columns has_phenotype_dir modalities sessions tasks fmriprep freesurfer mriqc
2008_Montreal_BlindRestingState_OC_raw 50 True True ['participant_id', 'group', 'match', 'gender', 'age', 'handedness', 'onset_blindness', 'years_of_blindness', 'total_blindness', 'years_of_total_blindness'] False ['anat', 'func'] [] ['rest'] n/a n/a n/a
2012_Montreal_AudVisMotion_GD_raw 35 False False n/a False ['anat', 'func'] [] ['audMotion', 'visMotion'] n/a n/a n/a
2014_Toronto_CataractsDiffusion_Gao_raw 26 True True ['/annex/objects/MD5E-s436--a3ab24b813254f34bc5773ec7427c33c.tsv'] False ['anat', 'dwi'] ['1'] n/a n/a n/a n/a
2015_Trento_Diffusion_SM_raw 30 True True ['participant_id', 'group', 'group_counter', 'initials', 'recording'] False ['anat', 'dwi', 'func'] [] ['unknown'] n/a n/a n/a
2015_Trento_MultimConn_SM_raw 29 True True ['/annex/objects/MD5E-s567--adb7ca1aa64cfe8680a842000ec06a43.tsv'] False ['anat', 'dwi', 'func'] ['01'] ['Catgs'] n/a n/a n/a
2015_Trento_categs_SM_raw 53 False False n/a False ['anat', 'dwi', 'func'] [] ['Catgs'] n/a n/a n/a
2016_Trento_Categs_MR_raw 29 False False n/a False ['anat', 'dwi', 'func'] ['01'] ['Catgs'] n/a n/a n/a
2016_Trento_MultimodalMotion_MR_raw 23 False False n/a False ['anat', 'func'] ['01'] ['audMotion', 'motDecoding', 'visMotion'] n/a n/a n/a
2016_Trento_SpatiotopyDir_CB_raw 33 True True ['/annex/objects/MD5E-s724--cfd24d6ce2366dd29831a737b3f48752.tsv'] False ['anat', 'func'] [] ['pRF'] n/a n/a n/a
2018_LLN_FPAS_FB_raw 16 False False n/a False ['eeg'] [] ['FPAS'] n/a n/a n/a
2018_Toronto_Cataract_SM_raw 46 False False n/a False ['anat', 'func'] [] ['categ', 'categBlur1', 'categBlur2'] n/a n/a n/a
2019_Nancy_iEEG_FB_raw 1 True False ['participant_id'] False ['ieeg'] ['1', '2', '3', '4'] ['ERPaudmot', 'ERPbraille', 'ERPcategory', 'FPSaudmot', 'FPSemotion', 'FPSface', 'FPSlexical', 'FPSvismot', 'FPSvoice', 'Readingbraille', 'Restingstate', 'Speechtracking', 'Stimulation'] n/a n/a n/a
2021_Dijon_BabyFPAS_RPC_raw 23 False False n/a False ['eeg'] [] ['FPAS'] n/a n/a n/a
2021_LLN_FVDE_FB_raw 20 False False n/a False ['eeg'] ['001'] ['fvde'] n/a n/a n/a
2021_LLN_emotion_ST_raw 14 False False n/a False ['eeg'] [] ['emotion'] n/a n/a n/a
2021_SaintLuc_TmsMT_FB_raw 16 True True ['participant_id'] False ['anat', 'beh', 'func'] ['mri', 'sham', 'tms'] ['audInstrumentDiscrimination', 'audMotionDirDiscrimination', 'auditoryLocalizer', 'visColourDiscrimination', 'visMotionDirDiscrimination', 'visualLocalizer'] n/a n/a n/a
2021_SaintLuc_combiEmo_FF_raw 24 True True ['participant_id', 'education', 'bmi'] False ['anat', 'func'] ['01', '02'] ['correction', 'eventrelatedCombiemoAuditory', 'eventrelatedCombiemoBimdal', 'eventrelatedCombiemoBimodal', 'eventrelatedCombiemoVisual', 'facelocalizerCombiemo', 'facelocalizerCombiemoCombiemo', 'voicelocalizer', 'voicelocalizerCombiemo'] n/a n/a n/a
2022_SaintLuc_lipSpeech_AA_raw 2 False False n/a False ['anat', 'func'] ['01', '02', '03'] ['MVPAAud', 'MVPAVis', 'PhonoLoc', 'VisLoc'] n/a n/a n/a
2023_Liege_BLAM_MB_raw 14 True True ['participant_id', 'age', 'sex', 'group'] False ['anat', 'func'] ['01', '02', '03', '04'] ['BimodalMotionAud', 'BimodalMotionVis', 'audioV1', 'auditoryLocalizer', 'bimodalMotionAud', 'bimodalMotionVis', 'mtMstLocalizer', 'rdkBimodalMotion', 'restingState', 'visualLocalizer'] n/a n/a n/a
2023_SaintLuc_VisTacMotionFoR_IS_raw 21 True True ['participant_id', 'codename', 'date', 'age', 'gender'] False ['anat', 'func'] ['001', '002'] ['handDown', 'handUp', 'mtMstLocalizer', 'tactileLocalizer2', 'visual', 'visualLocalizer2'] n/a n/a n/a
2023_Trento_plosBiology_YX_raw 48 True False ['participant_id'] False ['anat', 'fmap', 'func'] [] ['judgement', 'resting'] n/a n/a n/a
Toronto_VisMotionLocalizer_MR_raw 3 True False ['participant_id'] False ['anat', 'func'] ['01'] ['visMotion'] n/a n/a n/a
olf_blind_raw 35 True True ['participant_id', 'Group', 'Sex', 'Age', 'Educational level', 'Smoker', 'Medication', 'Vision level', 'Use of guide dog', 'Use of white cane', 'Musical practice', 'Braille reading', 'Braille reading hand used', 'Age of total blindness onset', 'Blindness Reason', 'Handedness', 'DK_C1_Letter_Fluency', 'DK_C2_Category_Fluency', 'DK_C3_Category_SwitchingTC', 'DK_C3_Category_SwitchingACC', 'TEA_C2', 'TEA_C3', 'CVLT_T1_T5', 'CVLT_ImmediateFreeRecall', 'CVLT_ImmediateCuedRecall', 'CVLT_DelayedFreeRecall', 'CVLT_DelayedCuedRecall', 'CVLT_Recognition', 'SS_Iden_O1', 'SS_Iden_O2', 'SS_Iden_O3', 'SS_Iden_O4', 'SS_Iden_O5', 'SS_Iden_O6', 'SS_Iden_O7', 'SS_Iden_O8', 'SS_Iden_O9', 'SS_Iden_O10', 'SS_Iden_O11', 'SS_Iden_O12', 'SS_Iden_O13', 'SS_Iden_O14', 'SS_Iden_O15', 'SS_Iden_O16', 'SS_Pls_O1', 'SS_Pls_O2', 'SS_Pls_O3', 'SS_Pls_O4', 'SS_Pls_O5', 'SS_Pls_O6', 'SS_Pls_O7', 'SS_Pls_O8', 'SS_Pls_O9', 'SS_Pls_O10', 'SS_Pls_O11', 'SS_Pls_O12', 'SS_Pls_O13', 'SS_Pls_O14', 'SS_Pls_O15', 'SS_Pls_O16', 'SS_Int_O1', 'SS_Int_O2', 'SS_Int_O3', 'SS_Int_O4', 'SS_Int_O5', 'SS_Int_O6', 'SS_Int_O7', 'SS_Int_O8', 'SS_Int_O9', 'SS_Int_O10', 'SS_Int_O11', 'SS_Int_O12', 'SS_Int_O13', 'SS_Int_O14', 'SS_Int_O15', 'SS_Int_O16', 'SS_Threshold_rightN', 'SS_Threshold_leftN', 'SS_Discrimination_Total', 'SS_OM_Total', 'SS_OM_Hits', 'SS_OM_FalseAlarms', 'SS_OM_Miss', 'SS_OM_Correct_Rejection', 'SS_OM_O1', 'SS_OM_O2', 'SS_OM_O3', 'SS_OM_O4', 'SS_OM_O5', 'SS_OM_O6', 'SS_OM_O7', 'SS_OM_O8', 'SS_OM_O9', 'SS_OM_O10', 'SS_OM_O11', 'SS_OM_O12', 'SS_OM_O13', 'SS_OM_O14', 'SS_OM_O15', 'SS_OM_O16', 'IRM_Pleasantness_Eucalyptus', 'IRM_Pleasantness_Almond', 'IRM_Intensity_Eucalyptus', 'IRM_Intensity_Almond'] False ['anat', 'dwi', 'func'] [] ['olfid', 'olfloc', 'rest'] n/a n/a n/a
48 changes: 48 additions & 0 deletions tools/list_raw.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
"""List datasets contents on cpp-lln-lab_raw and write the results in a tsv file.

to do:

- [ ] Also checks for derivatives folders for mriqc, frmiprep and freesurfer.
"""

from pathlib import Path

import pandas as pd

from utils import init_dataset
from utils import list_datasets_in_dir

cpp_raw = Path(__file__).parent.parent / "cpp-lln-lab_raw"


# Overwrite the tsv file with the current raw datasets

DEBUG = False

datasets = init_dataset()
input_dir = cpp_raw
datasets = list_datasets_in_dir(datasets, input_dir, debug=DEBUG)

datasets_df = pd.DataFrame.from_dict(datasets)

datasets_df = datasets_df.sort_values("name")

root_dir = Path(__file__).parent.parent

output_file = Path(__file__).parent / "datasets_raw.tsv"

datasets_df.to_csv(output_file, index=False, sep="\t")

mk_file = Path(__file__).parent.parent / "src" / "datasets_raw.md"

datasets_df = datasets_df.drop(
columns=[
"has_participant_tsv",
"has_participant_json",
"has_phenotype_dir",
"participant_columns",
],
)

with open(mk_file, "w") as f:
datasets_df.to_markdown(f, index=False, mode="a")
15 changes: 15 additions & 0 deletions tools/print_dataset_listing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
"""Take the listing of raw datasets
and turns it into a markdown document with a series of markdown tables."""

from pathlib import Path
import pandas as pd
from bids import BIDSLayout

column_order = [
"name",
"description",
"datatypes",
"suffixes",
"link to full data",
"maintained by",
]
Loading