Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should we return error when target lost for prometheus scaler #3064

Closed
bamboo12366 opened this issue May 24, 2022 · 3 comments
Closed

Should we return error when target lost for prometheus scaler #3064

bamboo12366 opened this issue May 24, 2022 · 3 comments
Labels
bug Something isn't working

Comments

@bamboo12366
Copy link
Contributor

Report

when Prometheus target is lost, the prometheus query will return an empty result here:
https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L55

The hpa controller will get an 0 value and using the minReplicas. Maybe we should return an error when len(reslut) = 0? To let the hpa forget about this workload for a while until the prometheus is back to normal?

There are related issue regarding to this: #657. But I think we shouldn't patch the replicas when the dependency is not available. Maybe somebody can share more thoughts about that?

Expected Behavior

keda-metrics return 0 when prometheus is unavailable, hpa controller is patching the minReplicas to workload

Actual Behavior

keda-metrics should return error when prometheus is unavailable, hpa controller should maintain the same replicas

Steps to Reproduce the Problem

  1. Lost prometheus target

Logs from KEDA operator

example

KEDA Version

2.7.1

Kubernetes Version

1.20

Platform

Any

Scaler Details

No response

Anything else?

No response

@bamboo12366 bamboo12366 added the bug Something isn't working label May 24, 2022
@bamboo12366 bamboo12366 changed the title Should we return error when prometheus query missing Should we return error when target lost for prometheus scaler May 24, 2022
@zroubalik
Copy link
Member

zroubalik commented May 25, 2022

@bamboo12366 is this somehow related to #3065?

@bamboo12366
Copy link
Contributor Author

@bamboo12366 is this somehow related to #3065?

seems the same thing, can I give a try if no one is working on this?

@zroubalik
Copy link
Member

@bamboo12366 yeah that would be great, let me close this issue and continue on the other one.

Duplicate of #3065

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

No branches or pull requests

2 participants