You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
New additions to the gain selection and sequencer scripts to include the gain selection stage in the next-day onsite analysis pipeline. The straightforward approach is to perform the gain selection for the whole night first thing the morning after data taking (triggered by a cron job).
As a first step:
Be able to run the gain-selection script with a date as an input argument instead of a list of dates (leave the possibility of also running the script with a list of dates).
Add cronjob to run the gain selection script for a given date.
Check job completion in gain selection. Use the existing --check option in the gain selection script. If everything finishes successfully, write a finished-flag file or write it in the osa database. This uses the SLURM's sacct output and lists the files in R0 and R0G respectively. Note: currently, sacct does not fetch correct information from SLURM jobs. This post-process of the gain selection should be checked by another cron job.
sequencer should first check whether this flag file exists or not to keep going with the processing. sequencer should be called then repeatedly by a cronjob (right now there is only one call in the morning). To avoid submitting same jobs multiple times we should check in the osa DB whether the processing has already started or not.
Maybe we should set in the cfg whether we want lstosa to start from R0 or from R0G to be flexible.
[OPTIONAL] It would be helpful to report the gain selection status in the sequencer output table as another column (but this is a second-order implementation).
Some aspects:
I guess we should keep writing the gain-selected files in R0G
Should we keep track of the gain_selection/DVR tool version as it is outside lstchain?
The gain selection script should take ~10 minutes for each subrun. The problem is when there are runs that cannot be gain selected, thus should be just copied, which can considerably increase the overall run time.
Note: Alternatively, in the future, we might include the gain selection stage and any other DVR-related stage inside datasequence.py where all lstchain scripts are piped. In this way, there should be a previous analysis level defined in the sequence of analysis steps, which right now are R0 to DL1a -> DL1a to DL1b -> DL1 datacheck -> DL1b to DL2
The text was updated successfully, but these errors were encountered:
New additions to the gain selection and sequencer scripts to include the gain selection stage in the next-day onsite analysis pipeline. The straightforward approach is to perform the gain selection for the whole night first thing the morning after data taking (triggered by a cron job).
As a first step:
--check
option in the gain selection script. If everything finishes successfully, write a finished-flag file or write it in the osa database. This uses the SLURM's sacct output and lists the files in R0 and R0G respectively. Note: currently,sacct
does not fetch correct information from SLURM jobs. This post-process of the gain selection should be checked by another cron job.sequencer
should be called then repeatedly by a cronjob (right now there is only one call in the morning). To avoid submitting same jobs multiple times we should check in the osa DB whether the processing has already started or not.Some aspects:
The gain selection script should take ~10 minutes for each subrun. The problem is when there are runs that cannot be gain selected, thus should be just copied, which can considerably increase the overall run time.
Note: Alternatively, in the future, we might include the gain selection stage and any other DVR-related stage inside
datasequence.py
where all lstchain scripts are piped. In this way, there should be a previous analysis level defined in the sequence of analysis steps, which right now are R0 to DL1a -> DL1a to DL1b -> DL1 datacheck -> DL1b to DL2The text was updated successfully, but these errors were encountered: