Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop reliance on a local filesystem #25

Open
gabbard opened this issue May 5, 2020 · 0 comments
Open

Drop reliance on a local filesystem #25

gabbard opened this issue May 5, 2020 · 0 comments

Comments

@gabbard
Copy link

gabbard commented May 5, 2020

To ultimately drop the reliance on a shared filesystem, we need to be able to use Pegasus File objects for our inputs and outputs. However, Pegasus by default is set up expecting the input and output files to be specified as job arguments. We would need to have a way of interpolating the local path to a file transferred by Pegasus on the execution machine into the parameter file ultimately passed to our executing script. That's it's own issue. For this issue, one of the Pegasus developers suggested that as a baby step we make the ___ckpt files themselves Pegasus Files. A job would depend on the Files corresponding the the checkpoint files of its "parent" jobs and its output would always be the File corresponding to its own checkpoint.

See #23 for context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants