You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We heavily rely on KEDA on our prod, had an incident and I’m looking for ways to circumvent such cases in the future.
We scale some of our deployments based on the number of requests they’re getting per second which works very well most of the time.
During this incident, we had an issue with our prometheus and all of our queries returned null values which it looks like the prometheus scaler interpreted as 0.
If that scaledobject was failing, we would have gotten alerts about it and figured out the issue, but because it didn’t happen, all of our deployments that relayed on the prometheus scaler scaled to the minimum number of replicas during peak hours and it wasn’t pretty.
Anything else?
No response
The text was updated successfully, but these errors were encountered:
@NaveLevi Just want to confirm, the query return null value, which means the result is empty list right, like below: { status: "success", isPartial: false, data: { resultType: "vector", result: [ ] } }
there how I got when prometheus target was lost in my case
Proposal
Add option to consider null values as errors
Use-Case
We heavily rely on KEDA on our prod, had an incident and I’m looking for ways to circumvent such cases in the future.
We scale some of our deployments based on the number of requests they’re getting per second which works very well most of the time.
During this incident, we had an issue with our prometheus and all of our queries returned null values which it looks like the prometheus scaler interpreted as 0.
If that scaledobject was failing, we would have gotten alerts about it and figured out the issue, but because it didn’t happen, all of our deployments that relayed on the prometheus scaler scaled to the minimum number of replicas during peak hours and it wasn’t pretty.
Anything else?
No response
The text was updated successfully, but these errors were encountered: