Merge v24.1.9 into main (#905)

cmadjar · laemtl · Laetitia Fesselier · web-flow · commit 7d8fd24f6843 · 2023-03-10T18:11:32.000-05:00
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>
diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-24.1.8
+24.1.9
diff --git a/python/lib/database_lib/files.py b/python/lib/database_lib/files.py
@@ -62,7 +62,7 @@ def find_file_with_series_uid_and_echo_time(self, series_uid, echo_time, phase_e
             query += " AND PhaseEncodingDirection IS NULL "
 
         if echo_number:
-            query += " AND EchoNumber = % "
+            query += " AND EchoNumber = %s "
             args.append(echo_number)
         else:
             query += " AND EchoNumber IS NULL "
@@ -192,3 +192,18 @@ def get_files_inserted_for_tarchive_id(self, tarchive_id):
         query = "SELECT * FROM files WHERE TarchiveSource = %s"
 
         return self.db.pselect(query=query, args=(tarchive_id,))
+
+    def get_files_inserted_for_session_id(self, session_id):
+        """
+        Get the list of files that were inserted into the `files` table for a given `SessionID`.
+
+        :param session_id: `SessionID` to restrict the query on
+         :type session_id: int
+
+        :return: list of relative file path present in the `files` table associated to the `SessionID`
+         :rtype: list
+        """
+
+        query = "SELECT * FROM files WHERE SessionID = %s"
+
+        return self.db.pselect(query=query, args=(session_id,))
diff --git a/python/lib/database_lib/mri_candidate_errors.py b/python/lib/database_lib/mri_candidate_errors.py
@@ -49,3 +49,18 @@ def insert_mri_candidate_errors(self, field_value_dict):
             values=field_value_dict.values(),
             get_last_id=False
         )
+
+    def get_candidate_errors_for_tarchive_id(self, tarchive_id):
+        """
+        Get the list of MRI candidate errors logged in `MRICandidateErrors` for a given `TarchiveID`.
+
+        :param tarchive_id: `TarchiveID` to restrict the query on
+         :type tarchive_id: int
+
+        :return: list of files inserted into the `MRICandidateErrors` table
+         :rtype: list
+        """
+
+        query = "SELECT * FROM MRICandidateErrors WHERE TarchiveID = %s"
+
+        return self.db.pselect(query=query, args=(tarchive_id,))
diff --git a/python/lib/dcm2bids_imaging_pipeline_lib/nifti_insertion_pipeline.py b/python/lib/dcm2bids_imaging_pipeline_lib/nifti_insertion_pipeline.py
@@ -127,14 +127,15 @@ def __init__(self, loris_getopt_obj, script_name):
                 message = f"{self.nifti_path}'s acquisition protocol is 'unknown'."
                 self.log_error_and_exit(message, lib.exitcode.UNKNOWN_PROTOCOL, is_error="Y", is_verbose="N")
             else:
-                self.scan_type_name = self.imaging_obj.get_scan_type_name_from_id(self.scan_type_id)
+                self.loris_scan_type = self.imaging_obj.get_scan_type_name_from_id(self.scan_type_id)
         else:
+            self.scan_type_id = self.imaging_obj.get_scan_type_id_from_scan_type_name(self.loris_scan_type)
             if not self.scan_type_id:
                 self._move_to_trashbin()
                 self._register_protocol_violated_scan()
                 if self.nifti_s3_url:  # push violations to S3 if provided file was on S3
                     self._run_push_to_s3_pipeline()
-                message = f"{self.nifti_path}'s scan type {self.scan_type_name} provided to run_nifti_insertion.py" \
+                message = f"{self.nifti_path}'s scan type {self.loris_scan_type} provided to run_nifti_insertion.py" \
                           f" is not a valid scan type in the database."
                 self.log_error_and_exit(message, lib.exitcode.UNKNOWN_PROTOCOL, is_error="Y", is_verbose="N")
 
@@ -147,7 +148,7 @@ def __init__(self, loris_getopt_obj, script_name):
             self._register_protocol_violated_scan()
             if self.nifti_s3_url:  # push violations to S3 if provided file was on S3
                 self._run_push_to_s3_pipeline()
-            message = f"Scan type {self.scan_type_name} does not have BIDS tables set up."
+            message = f"Scan type {self.loris_scan_type} does not have BIDS tables set up."
             self.log_error_and_exit(message, lib.exitcode.UNKNOWN_PROTOCOL, is_error="Y", is_verbose="N")
 
         # ---------------------------------------------------------------------------------------------
@@ -435,8 +436,8 @@ def _determine_new_nifti_assembly_rel_path(self):
 
         # determine NIfTI file name
         new_nifti_name = self._construct_nifti_filename(file_bids_entities_dict)
-        already_inserted_filenames = self.imaging_obj.get_list_of_files_already_inserted_for_tarchive_id(
-            self.dicom_archive_obj.tarchive_info_dict["TarchiveID"]
+        already_inserted_filenames = self.imaging_obj.get_list_of_files_already_inserted_for_session_id(
+            self.dicom_archive_obj.tarchive_info_dict["SessionID"]
         )
         while new_nifti_name in already_inserted_filenames:
             file_bids_entities_dict['run'] += 1
diff --git a/python/lib/imaging.py b/python/lib/imaging.py
@@ -204,14 +204,26 @@ def insert_mri_candidate_errors(self, patient_name, tarchive_id, scan_param, fil
          :type reason: str
         """
 
+        series_uid = scan_param["SeriesUID"] if "SeriesUID" in scan_param.keys() else None
+        echo_time = scan_param["EchoTime"] if "EchoTime" in scan_param.keys() else None
+
+        existing_cand_errors = self.mri_cand_errors_db_obj.get_candidate_errors_for_tarchive_id(tarchive_id)
+
+        for row in existing_cand_errors:
+            if str(row['SeriesUID']) == str(series_uid) \
+                    and str(row['EchoTime']) == str(echo_time) \
+                    and str(row['PatientName']) == str(patient_name) \
+                    and str(row['Reason']) == str(reason):
+                return
+
         info_to_insert_dict = {
             "TimeRun": datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
-            "SeriesUID": scan_param["SeriesUID"] if "SeriesUID" in scan_param.keys() else None,
+            "SeriesUID": series_uid,
             "TarchiveID": tarchive_id,
             "MincFile": file_rel_path,
             "PatientName": patient_name,
             "Reason": reason,
-            "EchoTime": scan_param["EchoTime"] if "EchoTime" in scan_param.keys() else None
+            "EchoTime": echo_time
         }
         self.mri_cand_errors_db_obj.insert_mri_candidate_errors(info_to_insert_dict)
 
@@ -238,6 +250,7 @@ def insert_protocol_violated_scan(self, patient_name, cand_id, psc_id, tarchive_
 
         series_uid = scan_param["SeriesInstanceUID"] if "SeriesInstanceUID" in scan_param.keys() else None
         image_type = str(scan_param["ImageType"]) if "ImageType" in scan_param.keys() else None
+        echo_time = str(scan_param["EchoTime"]) if "EchoTime" in scan_param.keys() else None
         echo_number = repr(scan_param["EchoNumber"]) if "EchoNumber" in scan_param.keys() else None
         phase_encoding_dir = scan_param["PhaseEncodingDirection"] \
             if "PhaseEncodingDirection" in scan_param.keys() else None
@@ -247,7 +260,7 @@ def insert_protocol_violated_scan(self, patient_name, cand_id, psc_id, tarchive_
         for row in existing_prot_viol_scans:
             if row['SeriesUID'] == series_uid \
                     and row['PhaseEncodingDirection'] == phase_encoding_dir \
-                    and row['image_type'] == image_type \
+                    and row['TE_range'] == echo_time \
                     and row['EchoNumber'] == echo_number:
                 return
 
@@ -956,6 +969,26 @@ def get_list_of_files_already_inserted_for_tarchive_id(self, tarchive_id):
 
         return files_list
 
+    def get_list_of_files_already_inserted_for_session_id(self, session_id):
+        """
+        Get the list of filenames already inserted for a given SessionID.
+
+        :param session_id: the Session ID to process
+         :type session_id: int
+
+        :return: a list with file names already inserted in the files table for SessionID
+         :rtype: list
+        """
+
+        # get list files from a given tarchive ID
+        results = self.files_db_obj.get_files_inserted_for_session_id(session_id)
+
+        files_list = []
+        for entry in results:
+            files_list.append(os.path.basename(entry['File']))
+
+        return files_list
+
     def get_list_of_fmap_files_sorted_by_acq_time(self, files_list):
         """
         Get the list of fieldmap acquisitions that requires the IntendedFor field in their JSON file.
diff --git a/python/lib/physiological.py b/python/lib/physiological.py
@@ -826,8 +826,8 @@ def insert_event_file(self, event_data, event_file, physiological_file_id,
                 # table cols
                 add_event_fields = (
                     'PhysiologicalTaskEventID',
-                    'TaskName',
-                    'TaskValue'
+                    'PropertyName',
+                    'PropertyValue'
                 )
                 # each additional fields is a new entry
                 add_event_values = []
diff --git a/python/lib/utilities.py b/python/lib/utilities.py
@@ -14,7 +14,6 @@
 import requests
 import re
 import io
-
 import mat73
 import lib.exitcode
 
diff --git a/tools/gzip_nifti_files.pl b/tools/gzip_nifti_files.pl
@@ -166,7 +166,7 @@ =head2 Methods
     # create the gzipped NIfTI file
     my $gzip_nifti = &NeuroDB::MRI::gzip_file($nifti_full_path);
     unless ($gzip_nifti) {
-        print "WARNING: Failure to create $gzip_nifti on the filesystem\n";
+        print "WARNING: Failure to create $nifti_full_path.gz on the filesystem\n";
         next;
     }
 

Original file line number	Diff line number	Diff line change
`@@ -166,7 +166,7 @@ =head2 Methods`
`166`	`166`	`# create the gzipped NIfTI file`
`167`	`167`	`my $gzip_nifti = &NeuroDB::MRI::gzip_file($nifti_full_path);`
`168`	`168`	`unless ($gzip_nifti) {`
`169`		`- print "WARNING: Failure to create $gzip_nifti on the filesystem\n";`
	`169`	`+ print "WARNING: Failure to create $nifti_full_path.gz on the filesystem\n";`
`170`	`170`	`next;`
`171`	`171`	`}`
`172`	`172`