Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus scaler - add option to consider null values as errors #3065

Closed
NaveLevi opened this issue May 24, 2022 · 3 comments · Fixed by #3073
Closed

Prometheus scaler - add option to consider null values as errors #3065

NaveLevi opened this issue May 24, 2022 · 3 comments · Fixed by #3073
Assignees
Labels
feature-request All issues for new features that have not been committed to needs-discussion

Comments

@NaveLevi
Copy link

Proposal

Add option to consider null values as errors

Use-Case

We heavily rely on KEDA on our prod, had an incident and I’m looking for ways to circumvent such cases in the future.
We scale some of our deployments based on the number of requests they’re getting per second which works very well most of the time.
During this incident, we had an issue with our prometheus and all of our queries returned null values which it looks like the prometheus scaler interpreted as 0.
If that scaledobject was failing, we would have gotten alerts about it and figured out the issue, but because it didn’t happen, all of our deployments that relayed on the prometheus scaler scaled to the minimum number of replicas during peak hours and it wasn’t pretty.

Anything else?

No response

@NaveLevi NaveLevi added feature-request All issues for new features that have not been committed to needs-discussion labels May 24, 2022
@zroubalik
Copy link
Member

@bamboo12366 is willing to contribute this 🎉

@bamboo12366
Copy link
Contributor

@NaveLevi Just want to confirm, the query return null value, which means the result is empty list right, like below:
{ status: "success", isPartial: false, data: { resultType: "vector", result: [ ] } }
there how I got when prometheus target was lost in my case

@bamboo12366
Copy link
Contributor

@zroubalik @NaveLevi guys, could you kindly help me review this PR(#3073)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request All issues for new features that have not been committed to needs-discussion
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants