Reload the mri_upload dictionary before checking if a tarchive has been validated #783

cmadjar · 2022-08-10T19:25:09Z

Description

When a study is run through the whole dcm2bids pipeline, it exits after having run the DICOM validation with message:

The DICOM archive validation has failed for UploadID X. Either run the validation again and fix the problem or use --force to force the insertion of the NIfTI file.

The correct behaviour is to reload the mri_upload dictionary with the latest content for that upload so it reads properly the field IsTarchiveValidated and continues processing when that field has been set to 1 in the database.

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787)

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787)

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * set file number to undef if it is '' (#800) * [dcm2bids] Add missing get_scanner_candid function in imaging.py (#801) * add missing function called from database_config.py get_subject_ids function * add python function documentation * Add TaskName field to JSON side car file for modalities with task-* names (#802) * add taskName to JSON side car file for modalities with task-* names * add relevant comments * Add required IntendedFor BIDS field in fieldmap JSON files (#803) * add IntendedFor to fieldmaps automatically add visit_label in intended for paths * add method documentation * fix line too long * fix line too long * Update VERSION to 24.1.0 * Update release version in the README.md file * Fix batch_uploads_tarchive to redirect to dcm2bids pipeline when dcm2niix is set as a converter in the Configs (#804) * modify batch_uploads_tarchive so that it can run dcm2bids or dcm2mnc pipeline depending on the converter mentioned in the Config module * fix typo identified by Christine (#807) * [dcm2bids pipeline] Delete temporary directory created by the pipeline once it is done running (#806) * remove tmp dirs before exiting pipeline * add comments

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca>

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove print Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>

* Turn off column statistics option when running mysqldump (aces#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (aces#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (aces#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (aces#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * merge 24.0 release to 24.1-release (aces#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Turn off column statistics option when running mysqldump (aces#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (aces#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (aces#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (aces#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * merge 24.0 release to 24.1-release (aces#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (aces#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (aces#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (aces#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * set file number to undef if it is '' (aces#800) * [dcm2bids] Add missing get_scanner_candid function in imaging.py (aces#801) * add missing function called from database_config.py get_subject_ids function * add python function documentation * Add TaskName field to JSON side car file for modalities with task-* names (aces#802) * add taskName to JSON side car file for modalities with task-* names * add relevant comments * Add required IntendedFor BIDS field in fieldmap JSON files (aces#803) * add IntendedFor to fieldmaps automatically add visit_label in intended for paths * add method documentation * fix line too long * fix line too long * Update VERSION to 24.1.0 * Update release version in the README.md file * Fix batch_uploads_tarchive to redirect to dcm2bids pipeline when dcm2niix is set as a converter in the Configs (aces#804) * modify batch_uploads_tarchive so that it can run dcm2bids or dcm2mnc pipeline depending on the converter mentioned in the Config module * fix typo identified by Christine (aces#807) * [dcm2bids pipeline] Delete temporary directory created by the pipeline once it is done running (aces#806) * remove tmp dirs before exiting pipeline * add comments

* update the VERSION for the next bug fix release (aces#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (aces#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (aces#824) * fix some errors when RepetitionTime is not available in JSON file (aces#825) * Add capability to download file from s3 (aces#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (aces#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (aces#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (aces#830) * skip violation if not found on filesystem since it means the scan has been rerun (aces#831) * update VERSION file (aces#832) * do not push files to S3 when their path in the DB is already an S3 URL (aces#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (aces#834) * Merge 24.0 release into 24.1 release (aces#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (aces#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function * update version file (aces#847) * Chunk creation subprocess failure check (aces#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix aces#843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (aces#849) * remove prints in nifti_insertion_pipeline.py (aces#851) * fix permissoin denied upon deletion of tmp dir (aces#853) * update to next bug fix relesae (aces#854) * fix duplicated protocols error when same scan type returned (aces#856) * Add missing exit codes on the python's side (aces#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (aces#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (aces#860) * fix intended for bug when no acq time available (aces#861) * fix bug for intended for when getting the list of files needed IntendedFor (aces#862) * fix paths when there are not / at the end of the Config (aces#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (aces#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (aces#876) * Merge 24.0.4 into 24.1 release (aces#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (aces#872) * update version to 24.0.4 (aces#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (aces#881) * MATLAB 7.3 HDF5 Workaround (aces#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (aces#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (aces#884) * keep track of additional event data (24.1) (aces#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (aces#886) * change version to 24.1.8 (aces#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) * get SessioNID from the session_obj instead of the tarchive obj (#909) * Update VERSION to 24.1.10 (#910) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>

cmadjar added 2 commits August 10, 2022 15:16

reload mri_upload object

1669c8c

remove debugging exit and print statements

45e6118

cmadjar added Category: Bug Issue or PR that aims to report or fix a bug Add to release notes Priority: High Language: Python Issue or PR related to the Python codebase labels Aug 10, 2022

cmadjar added this to the 24.0.2 milestone Aug 15, 2022

cmadjar merged commit 941df57 into aces:24.0-release Aug 16, 2022

cmadjar added the Copied in release notes label Aug 18, 2022

cmadjar self-assigned this Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reload the mri_upload dictionary before checking if a tarchive has been validated #783

Reload the mri_upload dictionary before checking if a tarchive has been validated #783

Uh oh!

cmadjar commented Aug 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Reload the mri_upload dictionary before checking if a tarchive has been validated #783

Reload the mri_upload dictionary before checking if a tarchive has been validated #783

Uh oh!

Conversation

cmadjar commented Aug 10, 2022

Description

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant