-
Couldn't load subscription status.
- Fork 561
fix(collector): respect HPA scale; don't cap replicas at minReplicas #4401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@shuheiktgw , thank you for this PR. Please ensure that we don't have a regression to this #2585 . |
| }, | ||
| Spec: appsv1.DeploymentSpec{ | ||
| Replicas: manifestutils.GetInitialReplicas(params.OtelCol), | ||
| Replicas: manifestutils.GetDesiredReplicas(params.OtelCol), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe if GetDesiredReplicas will produce the same effect as described on #2585
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One idea would be adding a condition about the HPA here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you so much for your comment! I tried reproducing the behavior described in the original issue, but from what I can see, even when we change minReplicas, the Operator doesn’t seem to create a new ReplicaSet, and the Pods aren’t restarted. Could you let me know if I’m missing something here?
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: simplest
spec:
autoscaler:
minReplicas: 2
maxReplicas: 4
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
debug: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
NAME READY STATUS RESTARTS AGE
simplest-collector-c9fb74dcd-njqd8 1/1 Running 0 28s
simplest-collector-c9fb74dcd-plvd8 1/1 Running 0 28s
Update the minReplicas from 2 to 3.
spec:
autoscaler:
minReplicas: 3
maxReplicas: 4
Add one Pod without restarting the others.
NAME READY STATUS RESTARTS AGE
simplest-collector-c9fb74dcd-6rt44 1/1 Running 0 6s
simplest-collector-c9fb74dcd-njqd8 1/1 Running 0 52s
simplest-collector-c9fb74dcd-plvd8 1/1 Running 0 52s
|
@yuriolisa / maintainers: do you think we can merge this change? (This is the only fix I'm waiting on that's preventing me from upgrading the operator in my environment.) Regarding the open review comment: I believe at worst @shuheiktgw's change simply reverts the bug that was introduced (and in my opinion that's more important than fixing the original issue the buggy PR was intended to fix because in reality minReplicas on HPAs aren't changing unless it's a conscious, one-off decision. |
Description:
The issue was already well described in the original report. However, the operator currently always returns the values defined in spec, and autoscaler.minValues never scales up. I’ve fixed this problem. I also renamed the function to GetDesiredReplicas, since it’s not only called during initialization but also on every reconciliation.
Link to tracking Issue(s):
Testing:
Documentation: