-
Notifications
You must be signed in to change notification settings - Fork 16
Getting a new Flow into On Call Alerts
You've made a new flow, and you'd like our on-call rotation to receive an alert if it fails. Great!
A few things to know before we get started:
-
Currently we only have a non-critical alert system for Data Flows. These jobs run, at the moment, nightly at the most and are exclusively used for model training. Until something in this repository supports a production service in real-time, we shouldn't need to wake anybody up :).
-
You should not need to change anything in PagerDuty itself to add a flow to the on-call alerts. PagerDuty is where you will go to add teammates to the on-call rotation or, if ever required, to create a new critical alert flow for Prefect. But you don't need it to either a) add a flow or b) change which channel in Slack gets these alerts (currently it's #log-data-product-alerts).
- Go to the cloud.prefect.io dashboard.
- Click "Automations" in the tab bar under where it says "Pocket."
- Scroll down and find the automation that says "When any run from ___ changes to ____, then Slack [...]" I know, this is weird. The passphrase is pinned the value in the #log-data-products-alerts channel, so if you get asked for this passphrase you can find it there.
- Click the automation.
- Click "Next"
- You now see a list of automations, with some of them outlined in pink. Those are the ones that currently post alerts on failure. Click on yours to add it to the list. You succeeded if it gets a pink outline.
- Click "Next" until it turns into a "save" button and then click "Save." You will know it succeeded when the wait spinner goes away and you return to the automations overview page where you clicked on the automation box in the first place.
Here is a 30 second video demonstrating the whole process; the video is stored with our work file manager and you'll have to be logged into your work account to view it.