Add prober to alert when the quantity of queued tasks reaches a threshold #3827
Description
Background
We have probers that run in an isolated environment and detect certain scenarios so that we can be alerted when the scenarios occur. Some probers use internal API endpoints to detect these scenarios.
Desired behavior
- An internal API endpoint should exist that returns the quantity of queued tasks (see this code for obtaining this)
- A prober should raise an exception when the quantity of returned by the endpoint is greater than 50
Value add
- Helps to alert us sooner when our task workers could get overwhelmed