Improvements for gain selection automation #293

marialainez · 2024-06-12T13:26:21Z

Main changes that have been made in this PR:

Write history files for each subrun (apart from the log files) + update them every time the gain selection script is launched
Allow launching the gain selection script several times without launching the jobs that are already running again (if the history file exists and is empty for a given subrun, it means that the job is running for that run, so it doesn't launch it again)
Change the check option so that it is done looking at the history files and not at the log files
Add simulate option to the gain selection script
Once the gain selection finishes correctly for all runs, the gain selection script is launched with the check option to see if there is any run missing that needs to be copied to R0G.

src/osa/job.py

src/osa/scripts/gain_selection.py

morcuended

Here are some more comments. Looks good overall. IMO some parts are too nested, but I understand that there are different cases you have to cope with

src/osa/scripts/gain_selection.py

morcuended · 2024-07-01T15:26:08Z

@marialainez, this would allow running the gain selection launcher several times and just re-doing jobs that did not previously succeed, right?

marialainez · 2024-07-01T15:37:51Z

@marialainez, this would allow running the gain selection launcher several times and just re-doing jobs that did not previously succeed, right?

yes, we have been testing it for the last days and it works as expected

morcuended

looks good overall, if you have tested it in the cluster for a couple of days I'd say it's good to go

src/osa/scripts/datasequence.py

src/osa/scripts/gain_selection.py

src/osa/scripts/datasequence.py

src/osa/scripts/gain_selection.py

morcuended

some changes in the log messages, to make following the logs easier. I would just leave some log info messages for general actions happening and maybe just give info at run level.

src/osa/scripts/gain_selection.py

src/osa/scripts/sequencer.py

src/osa/scripts/gain_selection.py

src/osa/configs/sequencer.cfg

morcuended

Find some more comments inline. Test it in the next period for several days and if works smoothly I'd say it's ready to go

src/osa/scripts/sequencer.py

morcuended · 2024-09-19T12:39:54Z

src/osa/scripts/sequencer.py

+            log.info(f"Sequencer is still running for date {date_to_iso(options.date)}. Try again later.")
+            sys.exit(0)
+
+        elif sequencer_finished(options.date) and not options.force:


when do you foresee using the force option?

morcuended

Some more comments (I hope the last ones)

src/osa/job.py

src/osa/utils/cliopts.py

src/osa/scripts/sequencer.py

morcuended · 2024-10-23T08:02:20Z

A problem with the mamba setup step in CI makes tests fail. Please relaunch the tests after #303.

morcuended · 2024-10-23T10:09:51Z

@marialainez please rebase the branch with main to make the tests run again

… needed

…entification

Co-authored-by: Daniel Morcuende <dmorcuende@iaa.es>

morcuended

Looks good! Thanks for all the work

marialainez marked this pull request as draft June 12, 2024 13:26

morcuended reviewed Jun 17, 2024

View reviewed changes

morcuended reviewed Jun 26, 2024

View reviewed changes

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

morcuended reviewed Jul 1, 2024

View reviewed changes

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

morcuended reviewed Jul 9, 2024

View reviewed changes

src/osa/scripts/datasequence.py Outdated Show resolved Hide resolved

src/osa/scripts/gain_selection.py Show resolved Hide resolved

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

morcuended reviewed Jul 9, 2024

View reviewed changes

src/osa/scripts/datasequence.py Outdated Show resolved Hide resolved

src/osa/scripts/datasequence.py Outdated Show resolved Hide resolved

morcuended reviewed Jul 10, 2024

View reviewed changes

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

morcuended reviewed Jul 10, 2024

View reviewed changes

src/osa/scripts/gain_selection.py Show resolved Hide resolved

morcuended reviewed Aug 1, 2024

View reviewed changes

marialainez marked this pull request as ready for review August 6, 2024 13:24

morcuended reviewed Aug 8, 2024

View reviewed changes

src/osa/scripts/gain_selection.py Outdated Show resolved Hide resolved

morcuended reviewed Aug 8, 2024

View reviewed changes

src/osa/configs/sequencer.cfg Outdated Show resolved Hide resolved

morcuended reviewed Sep 19, 2024

View reviewed changes

morcuended reviewed Oct 1, 2024

View reviewed changes

marialainez added 11 commits October 24, 2024 11:41

write a history file for each subrun and check its output

c472d4b

move run_sacct_j function to job.py

5aca805

add memory required by job to the cfg file

db98603

improve gain_selection script

c7a1c01

relaunch the jobs that finished in TIMEOUT

6b2957d

remove unnecessary imports

6d9350d

give a warning and exit if run summary is empty

2c31aeb

solve mistake

b89edcb

avoid creating a list with the files of all subruns of a run

ec39423

make the check looking at the history files (instead of logs)

f7625db

remove unnecessary imports and variables

5dbe639

marialainez and others added 26 commits October 24, 2024 11:41

remove unused import

1ea8719

allow watching the job status when launching sequencer with -s

513a2bb

stop gain selection check only in case FF heuristic identification is…

3840989

… needed

add option in the cfg to decide whether to use or not FF heuristic id…

cfd9b0b

…entification

avoid creating the flag file if there is no data

92bab93

solve mistake

6fddc64

check if sequencer already finished before submitting the jobs

ce62477

reduce memory requirement in the example cfg

e9f1029

adapt tests

f118abd

return False if no jobs were launched yet

e38508d

adapt warning message + create flag file even with warnings

82b72f6

rename runwise .history files as .closed

e32878d

update history files only if .closed file does not exist

bce3f41

Update src/osa/scripts/sequencer.py

a08c365

Co-authored-by: Daniel Morcuende <dmorcuende@iaa.es>

write the run number using always 5 digits

e0124b2

add necessary import

b364743

check if the R0G files can be opened with ctapipe_io_lst

2328828

remove unused variable

d9b1e72

remove unnecessary check

56a2735

check the last job id (e.g. for timeouts)

7e81b4f

rename function + rename force argument

2753bb9

Update src/osa/scripts/sequencer.py

f2ad14a

Co-authored-by: Daniel Morcuende <dmorcuende@iaa.es>

Update src/osa/job.py

5ef4b5e

Co-authored-by: Daniel Morcuende <dmorcuende@iaa.es>

Update src/osa/job.py

6e85dbe

Co-authored-by: Daniel Morcuende <dmorcuende@iaa.es>

change function name

424c073

fix name of force option

d32dbba

morcuended force-pushed the gainsel_automation branch from d19e19f to d32dbba Compare October 24, 2024 09:47

morcuended approved these changes Oct 24, 2024

View reviewed changes

morcuended merged commit fd8174e into main Oct 24, 2024
5 checks passed

morcuended deleted the gainsel_automation branch October 24, 2024 10:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements for gain selection automation #293

Improvements for gain selection automation #293

marialainez commented Jun 12, 2024 •

edited by morcuended

Loading

morcuended left a comment

morcuended commented Jul 1, 2024

marialainez commented Jul 1, 2024

morcuended left a comment

morcuended left a comment •

edited

Loading

morcuended left a comment

morcuended Sep 19, 2024

morcuended left a comment

morcuended commented Oct 23, 2024

morcuended commented Oct 23, 2024

morcuended left a comment

Improvements for gain selection automation #293

Improvements for gain selection automation #293

Conversation

marialainez commented Jun 12, 2024 • edited by morcuended Loading

morcuended left a comment

Choose a reason for hiding this comment

morcuended commented Jul 1, 2024

marialainez commented Jul 1, 2024

morcuended left a comment

Choose a reason for hiding this comment

morcuended left a comment • edited Loading

Choose a reason for hiding this comment

morcuended left a comment

Choose a reason for hiding this comment

morcuended Sep 19, 2024

Choose a reason for hiding this comment

morcuended left a comment

Choose a reason for hiding this comment

morcuended commented Oct 23, 2024

morcuended commented Oct 23, 2024

morcuended left a comment

Choose a reason for hiding this comment

marialainez commented Jun 12, 2024 •

edited by morcuended

Loading

morcuended left a comment •

edited

Loading