Project memory

[mriedel56]

Should we keep a running tally of project memory? i.e., raw storage, dset storage, derivatives storage (and subsequent folders within derivatives like mriqc, fmriprep, etc). could be more beneficial for more accurate estimates/reporting for storage needs in grants

[tsalo]

I think that makes sense. What are you thinking? How about:

• Total raw DICOM data (/data/nbc/DICOM/[project])
• Mean raw DICOM data (/data/nbc/DICOM/[project])
• Total raw nifti data (/data/nbc/[project]/dset/sub-*)
• Mean raw nifti data per subject (/data/nbc/[project]/dset/sub-*)
• Total MRIQC (/data/nbc/[project]/derivatives/mriqc)
• Mean MRIQC (/data/nbc/[project]/derivatives/mriqc)
• Total fMRIPrep (/data/nbc/[project]/derivatives/fmriprep)
• Mean fMRIPrep (/data/nbc/[project]/derivatives/fmriprep)
• Total Freesurfer, because it's run with fMRIPrep (/data/nbc/[project]/derivatives/freesurfer)
• Mean Freesurfer (/data/nbc/[project]/derivatives/freesurfer)

We can extend to include projects from other labs as well.

[mriedel56]

I was mainly thinking about totals for each project, but yeah, means would also be good.

[tsalo]

I've got code working for this, and would like to add it to this repository so others can use it, but want to check how other labs plan to organize their datasets and derivatives. I am assuming that the derivatives (MRIQC, Freesurfer, and fMRIPrep) will go outside the BIDS datasets. The cis-processing code outputs the MRIQC derivatives to a folder inside the BIDS dataset (e.g., /home/data/lab/project/dset/derivatives/), but I would like to change that to a folder outside (e.g., /home/data/lab/project/derivatives/). How does that sound? Tagging @anthonystevendick @mattfeld and @adamkimbler for your thoughts.

Given that one current plan is to only download dicoms from XNAT only for conversion to BIDS format (and then to clean up the tar files after), my code currently doesn't look for dicoms.

[anthonystevendick]

It would be better to have the bids folder self-contained because it may need to be uploaded to the NIH data archive. So having the downstream process folders outside the bids folder is better. Anthony

…

-- Anthony Steven Dick, Ph.D. Associate Professor Director, Cognitive Neuroscience Program and Graduate Certificate in Cognitive Neuroscience Florida International University MM Campus AHC4 454 Department of Psychology and Center for Children and Families<http://www.ccf.fiu.edu/>, an FIU Preeminent Program<https://beyondpossible.fiu.edu/preeminent-programs/> 11200 S.W. 8th Street Miami, FL 33199 Ph: 305-348-4202; Lab Ph: 305-348-9055; Fx: 305-348-3879 Email: adick@fiu.edu<mailto:adick@fiu.edu>; Webpage: http://dcn.fiu.edu<http://dcn.fiu.edu/>; Join the SSHD<http://www.sshdonline.org/> ##### Check out the new book<http://www.routledge.com/9781138960039>: Dick, A.S., & Müller, U. (2018). Advancing Developmental Science: Philosophy, Theory, and Method. Taylor & Francis. From: Taylor Salo <notifications@github.com> Reply-To: FIU-Neuro/cis-processing <reply@reply.github.com> Date: Saturday, September 15, 2018 at 2:00 PM To: FIU-Neuro/cis-processing <cis-processing@noreply.github.com> Cc: Anthony Dick <adick@fiu.edu>, Mention <mention@noreply.github.com> Subject: Re: [FIU-Neuro/cis-processing] Project memory (#19) I've got code working for this, and would like to add it to this repository so others can use it, but want to check how other labs plan to organize their datasets and derivatives. I am assuming that the derivatives (MRIQC, Freesurfer, and fMRIPrep) will go outside the BIDS datasets. The cis-processing code outputs the MRIQC derivatives to a folder inside the BIDS dataset (e.g., /home/data/lab/project/dset/derivatives/), but I would like to change that to a folder outside (e.g., /home/data/lab/project/derivatives/). How does that sound? Tagging @anthonystevendick<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_anthonystevendick&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=9VFxwiK9C7UXEW5AnVOyIEDdXK4M5Zp4pPxukidnLmU&e=> @mattfeld<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mattfeld&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=72aOMl88bYsxTcf5oUoVprWIUFiMQwhfqGQn614Yi6A&e=> and @adamkimbler<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_adamkimbler&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=pchHHHVnFFRCsng-z1Ic-mh4DtpLel-o0xLbaVlTfmo&e=> for your thoughts. Given that one current plan is to only download dicoms from XNAT only for conversion to BIDS format (and then to clean up the tar files after), my code currently doesn't look for dicoms. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_FIU-2DNeuro_cis-2Dprocessing_issues_19-23issuecomment-2D421608154&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=GsD7BPEGdTexcg5HHiJQ36g3Y-S4r_u2pwlHEubImaI&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AeBVSe5LNT-2DLZnLBWntratXLxImXAP-5FYks5ubUAWgaJpZM4WhJ0q&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=nBThDU7uihccvKH645UYGOpF86ZsTZ4gzgtFILTeY3E&e=>.

[tsalo]

@anthonystevendick That seems like a good rationale to keep them separate. Currently, we output to within the dataset folder, so this will require a change to cis_proc.py, but won't affect the BIDSifier image.

[tsalo]

I think we should move this to a separate repository. Something like cis-tracking. The check could be set up in separate cron jobs.