write status.csv as soon as study graph is constructed and between sy…#477
write status.csv as soon as study graph is constructed and between sy…#477atgrier wants to merge 2 commits into
Conversation
…nchronous login node jobs check for cancellation between synchronous login node jobs Signed-off-by: Alan Thomas Grier <grier1@llnl.gov>
| # updates, especially for login node jobs that complete | ||
| # synchronously. | ||
| if self._status_path: | ||
| self.write_status() |
There was a problem hiding this comment.
So I think this (and the cancellation) machinery should remain out at the conductor level, not in the job submission hot-loop inside the execution graph. Bench-marking the submission speed fix that's going into this release has the new floor at ~20-30 job submissions per second, and hitting the file system for this and cancellation that often is likely to cause problems (might even make the status cmd fail outright while jobs are being submitted due to io contention).
I've got an alternate local adapter + executor in the works to make it behave more like flux/slurm/etc schedulers (i.e. non-blocking) that should be a better path making things more responsive.
There was a problem hiding this comment.
I reverted the bulk of the changes, including the test file that was mostly for the cancellation changes. Now it's limited to a single addition of a status write at conclusion of initializing the conductor, along with a couple minor clerical edits.
2 participants
This PR addresses #476
Currently the status.csv is not written (such as for synchronous login node jobs) until all of the first job step jobs are completed, nor can cancellation be issued or checked until that point. Proposed changes: