-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expand file-based storage to all other storage types #2944
Conversation
Woot, this is great to see!
Since pickles are also stored as files (it's just the type of the contents of the file that is different), I'm not sure if that's the best kwarg. Perhaps something with |
Co-authored-by: Jim Crist-Harif <jcrist@users.noreply.github.com>
Co-authored-by: Jim Crist-Harif <jcrist@users.noreply.github.com>
Co-authored-by: Jim Crist-Harif <jcrist@users.noreply.github.com>
Co-authored-by: Jim Crist-Harif <jcrist@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Thanks for contributing to Prefect!
Please describe your work and make sure your PR:
changes/
directory (if appropriate)docs/outline.toml
for API reference docs (if appropriate)Note that your PR will not be reviewed unless all three boxes are checked.
What does this PR change?
This PR closes #2887 by expanding file-based storage (and hot reloading flows) to all other current storage types. This adds a
stored_as_file: bool
kwarg to the storage types that also support pickle storage.Example Workflow
Upload to S3:
Register with backend:
prefect register flow -f flow.py -p "Demo"
Opening this PR to get some feedback on the implementation. I plan on adding some more documentation around this pattern and making the storage documentation better. I also need to revamp the healthcheck.py for Docker storage because it is pickle-based right now (current implementation skips over healthcheck if using a file but will re-add it once working 👍)