Prefect 2 (aka Orion) examples running self-contained in a local kubernetes cluster. Batteries (mostly) included. 🔋
Prerequisites:
- make
- node (required for pyright)
- python >= 3.10
- docker & docker compose
- k3d (for creating a local kubernetes cluster)
- kubectl
- helm
To start:
- Install the development environment:
make install
Flows:
- Dask flow using a dask local cluster
- Dask kubes flow using an ephemeral dask cluster on Kubernetes
- Parameterized flow using a custom Docker image containing additional modules
- Ray flow that runs on an existing ray cluster (see tekumara/ray-demo)
- Sub flow that is trigger by a parent flow
- Submit flow demonstrates the difference of running tasks with/without
.submit() - Retry flow demonstrates retries when a task fails.
- Failure flow shows how to handle a failing task.
- Map flow uses Task.map.
- Flatten flow demonstrates how to parallelise over a list of lists.
- Parent flow shows how to trigger a flow run from a deployment within a parent flow. The triggered flow is treated as a sub flow.
- Context flow accessing prefect context at runtime.
Deployments to Kubernetes created via:
- Deployment objects in python - both local and uploaded to remote s3 storage.
- prefect.yaml - both local and pushed to s3.
make param-flowormake dask-flowormake ray-flowormake sub-flowmake uithen navigate to http://localhost:4200/
The orion sqlite database is stored in ~/.prefect/prefect.db
Create k3d cluster with an image registry, minio (for remote storage), the prefect agent and api
make kubes
Create deployments that run on kubernetes
make deploy
Prefect UI: http://localhost:4200/
Minio UI: http://localhost:9001. User: minioadmin pass: minioadmin.
Prefect API: http://localhost:4200/api/
- Blocks - an overview and look into the database tables for Blocks.
- Deployment - an overview of the deployment process.
- Tutorials from which some of the examples are taken
To run flows with a cloud workspace set:
export PREFECT_API_URL=https://api.prefect.cloud/api/accounts/$accountid/workspaces/$workspaceid
export PREFECT_API_KEY=<your api key>
$accountid and $workspaceid are visible in the URL when you login to Prefect Cloud. The api key can be created from your user profile (bottom left).
Setting the environment variables is recommended. An alternative method is to login using:
prefect cloud login --key <your api key>
However be aware that this stores your api url and key as plain text ~/.prefect/profiles.toml.
Create a kubernetes ray cluster
make kubes-ray
Ray dashboard: http://localhost:8265
Major
Minor
- Replicate kube events to Prefect that occur before the pod starts
- Configuring different task runner for dev and prod deployments
- Workers only support local storage
- Prefect.Deployment doesn't have a property pull_steps
- Support environment variables read from kubernetes secrets
- Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError
- Deployment metadata
- Stream logs via CLI
- FileNotFoundError errors when running with a remote ray cluster #26
- Add Flow.submit interface for subflows on external infrastructure
- not a known member of module "prefect.runtime"
- mapped tasks aren't collapsed in the UI like they were in Prefect 1
- pushing to s3 is sequential and slower than using sf3s to parallel upload to an s3 storage block
See all roadmap tagged issues for planned work.
Check the logs of the agent/worker:
make kubes-logs