-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
File-based flow storage #2840
Merged
Merged
File-based flow storage #2840
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…util, add default github secret
joshmeek
commented
Jun 22, 2020
cicdw
reviewed
Jun 23, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a few questions - exciting stuff!
jcrist
reviewed
Jun 23, 2020
cicdw
reviewed
Jun 24, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor doc tweak, otherwise LGTM!
joshmeek
commented
Jun 24, 2020
cicdw
approved these changes
Jun 24, 2020
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thanks for contributing to Prefect!
Please describe your work and make sure your PR:
changes/
directory (if appropriate)docs/outline.toml
for API reference docs (if appropriate)Note that your PR will not be reviewed unless all three boxes are checked.
What does this PR change?
Closes #2785
This is a first cut of an implementation of file based storage. There are a few new additions in this PR:
.py
files in github repos.py
filesprefect register flow
CLI command which takes a.py
file and registers a flow with a backendAs envisioned in this PR, the typical workflow for file based storage might look like:
Compose flow
.py
file where flow has GitHub storage:Push this
flow.py
file to themy/repo
repository under/flows/flow.py
.Call
prefect register -f flow.py
to register this flow with GitHub storage.Now as long as the flow structure is the same it does not need to be reregistered if the content of the flow itself changes. All that needs to happen is the flow file in the set location is overwritten (e.g. pushed to the git repo). This means flows may now be updated easily in CI/CD processes, development, mid run, between runs, etc...
A couple extra TODOs:
Why is this PR important?
Opens up the door for new file based storage, flow cloning, and "hot reloading" flows.