Update Wrapper to Support Pegasus 5.0.0 #73
Conversation
|
I believe there are still a few pending changes here including:
|
…rs rather than XML to be inline with Pegasus5
Codecov Report
@@ Coverage Diff @@
## master #73 +/- ##
==========================================
+ Coverage 74.77% 76.42% +1.65%
==========================================
Files 13 13
Lines 547 560 +13
==========================================
+ Hits 409 428 +19
+ Misses 138 132 -6
Continue to review full report at Codecov.
|
…th the "write_workflow_description" function
|
|
||
| Currently the root directory should be be in your home directory and not on an NAS like `/nas/gaia/` as the submission will fail for an NFS reason. | ||
| The experiment directory can be (and ought to be) on such a drive, though. | ||
|
|
There was a problem hiding this comment.
Are there any restrictions on valid root dirs? Should the README include a statement that root dirs can now be on shared /nas/gaia or /nas/user/xyz ?
There was a problem hiding this comment.
There are no longer any restrictions on valid root dirs. There used to be a NFS related bug when trying to mount a shared NAS as the root dir for Pegasus but it has been resolved.
There was a problem hiding this comment.
@danielnapierski I've added a note to the readme that the root_dir can be set to any location on the NAS, and that a shared drive e.g. /nas/gaia may be preferred due to space limitations on the user home dirs. However if unspecified the script will default to a users /nas/home/xyz/ dir.
…ult to OS's reply to a users home dir.
pegasus_wrapper/workflow.py
Outdated
| @@ -243,14 +279,12 @@ def limit_jobs_for_category(self, category: str, max_jobs: int): | |||
| """ | |||
| self._category_to_max_jobs[category] = max_jobs | |||
|
|
|||
| def _conf_limits(self) -> str: | |||
| def _conf_limits(self) -> None: | |||
| """ | |||
| Return a Pegasus config string which sets the max jobs per category appropriately. | |||
There was a problem hiding this comment.
nit: not returning a string anymore so docstring should be updated
| handle.write( | ||
| f"{ckpt_name} file://{checkpoint_path} site={self._default_site}\n" | ||
| ) | ||
| checkpoint_pegasus_file = self.create_file( |
README.md
Outdated
| @@ -72,7 +66,8 @@ Partitions each have a max walltime associated with them. See the saga cluster w | |||
|
|
|||
| If you change code while a pipeline is runnning, the jobs will pick up the changes. This could be helpful if you notice an error and fix it before that code runs, but can also lead to some unexpected behavior. | |||
|
|
|||
| ## `No module named 'Pegasus'` | |||
| ## `No module named 'Pegasus'` (Version 4.9.3) | |||
| *This is believed to have been fixed for Pegasus Version 5. If this arrises please leave an issue* | |||
|
|
|||
| This is a weird one that pops up usually when first getting set up with Pegasus. First, if you see this please contact one of the maintainers (currently @joecummings or @spigo900). To fix this, install the following packages with these commands in this exact order - they are dependent on each other. | |||
There was a problem hiding this comment.
Can remove this fix. Comment you added is enough.
Closes #48