Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gain selection and DVR to OSA workflow #227

Open
6 tasks
morcuended opened this issue Jun 9, 2023 · 0 comments
Open
6 tasks

Add gain selection and DVR to OSA workflow #227

morcuended opened this issue Jun 9, 2023 · 0 comments

Comments

@morcuended
Copy link
Member

New additions to the gain selection and sequencer scripts to include the gain selection stage in the next-day onsite analysis pipeline. The straightforward approach is to perform the gain selection for the whole night first thing the morning after data taking (triggered by a cron job).

As a first step:

  • Be able to run the gain-selection script with a date as an input argument instead of a list of dates (leave the possibility of also running the script with a list of dates).
  • Add cronjob to run the gain selection script for a given date.
  • Check job completion in gain selection. Use the existing --check option in the gain selection script. If everything finishes successfully, write a finished-flag file or write it in the osa database. This uses the SLURM's sacct output and lists the files in R0 and R0G respectively. Note: currently, sacct does not fetch correct information from SLURM jobs. This post-process of the gain selection should be checked by another cron job.
  • sequencer should first check whether this flag file exists or not to keep going with the processing. sequencer should be called then repeatedly by a cronjob (right now there is only one call in the morning). To avoid submitting same jobs multiple times we should check in the osa DB whether the processing has already started or not.
  • Maybe we should set in the cfg whether we want lstosa to start from R0 or from R0G to be flexible.
  • [OPTIONAL] It would be helpful to report the gain selection status in the sequencer output table as another column (but this is a second-order implementation).

Some aspects:

  • I guess we should keep writing the gain-selected files in R0G
  • Should we keep track of the gain_selection/DVR tool version as it is outside lstchain?

The gain selection script should take ~10 minutes for each subrun. The problem is when there are runs that cannot be gain selected, thus should be just copied, which can considerably increase the overall run time.

Note: Alternatively, in the future, we might include the gain selection stage and any other DVR-related stage inside datasequence.py where all lstchain scripts are piped. In this way, there should be a previous analysis level defined in the sequence of analysis steps, which right now are R0 to DL1a -> DL1a to DL1b -> DL1 datacheck -> DL1b to DL2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant