Issues with Prometheus Configuration and Controller Crashes in Helm Chart v1.44.1

[yuxzhang97]

Description

We recently attempted to upgrade our kubernetes-ingress deployment to the latest image and Helm chart version (v1.44.1) and controller version (3.1). However, after deploying to our staging environment, we encountered multiple issues, particularly related to Prometheus configuration.

Observed Behavior

After deploying the updated chart to our staging environment, we noticed the following logs:

2025/03/04 19:16:08 INFO    handler/prometheus.go:128 [transactionID=2de9d597-d3ea-46c2-9c2e-efd12b67cfe2] reload required : creation/modification of prometheus endpoint
2025/03/04 19:16:08 DEBUG   handler/https.go:163 [transactionID=2de9d597-d3ea-46c2-9c2e-efd12b67cfe2] Cannot proceed with SSL Passthrough update, HTTPS is disabled
[NOTICE]   (69) : Reloading HAProxy
[NOTICE]   (69) : Initializing new worker (187)
[NOTICE]   (187) : haproxy version is 3.1.5-076df02
[WARNING]  (187) : config : parsing [/etc/haproxy/haproxy.cfg:60] : a 'monitor fail' rule placed after an 'http-request' rule will still be processed before.
[WARNING]  (187) : config : parsing [/etc/haproxy/haproxy.cfg:79] : a 'monitor fail' rule placed after an 'http-request' rule will still be processed before.
[WARNING]  (187) : frontend 'https' has no 'bind' directive. Please declare it as a backend if this was intended.
[NOTICE]   (69) : Loading success.
[WARNING]  (185) : Proxy healthz stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy http stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
Proxy healthz stopped (cumulated conns: FE: 0, BE: 0).
Proxy http stopped (cumulated conns: FE: 0, BE: 0).
Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy tcp-8080 stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy https stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy github_http stopped (cumulated conns: FE: 0, BE: 0).
[WARNING]  (185) : Proxy kube-dotcom-ingress-staging_default-local-service_http stopped (cumulated conns: FE: 0, BE: 0).
Proxy tcp-8080 stopped (cumulated conns: FE: 0, BE: 0).
Proxy https stopped (cumulated conns: FE: 0, BE: 0).
Proxy github_http stopped (cumulated conns: FE: 0, BE: 0).
Proxy kube-dotcom-ingress-staging_default-local-service_http stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE]   (69) : haproxy version is 3.1.5-076df02
[WARNING]  (69) : Former worker (185) exited with code 0 (Exit)
2025/03/04 19:16:08 DEBUG   haproxy/process/s6-overlay.go:63 [transactionID=2de9d597-d3ea-46c2-9c2e-efd12b67cfe2] [NOTICE]   (69) : Initializing new worker (187)
[NOTICE]   (187) : haproxy version is 3.1.5-076df02
[WARNING]  (187) : config : parsing [/etc/haproxy/haproxy.cfg:60] : a 'monitor fail' rule placed after an 'http-request' rule will still be processed before.
[WARNING]  (187) : config : parsing [/etc/haproxy/haproxy.cfg:79] : a 'monitor fail' rule placed after an 'http-request' rule will still be processed before.
[WARNING]  (187) : frontend 'https' has no 'bind' directive. Please declare it as a backend if this was intended.
[NOTICE]   (69) : Loading success.
[NOTICE]   (69) : haproxy version is 3.1.5-076df02
[WARNING]  (69) : Former worker (185) exited with code 0 (Exit)
2025/03/04 19:16:08 INFO    controller/controller.go:211 [transactionID=2de9d597-d3ea-46c2-9c2e-efd12b67cfe2] HAProxy reloaded
2025/03/04 19:16:08 DEBUG   haproxy/api/api.go:407 [transactionID=2de9d597-d3ea-46c2-9c2e-efd12b67cfe2] Pushing backends as previous successfully applied backends
2025/03/04 19:16:13 INFO    k8s/crs-monitor.go:124  Custom resource definition created, adding CR watcher for Defaults
2025/03/04 19:16:13 INFO    k8s/crs-monitor.go:124  Custom resource definition created, adding CR watcher for Defaults
2025/03/04 19:16:13 INFO    handler/prometheus.go:128 [transactionID=4c9c6daa-7269-4f38-834e-795653a1666b] reload required : creation/modification of prometheus endpoint
2025/03/04 19:16:13 DEBUG   handler/https.go:163 [transactionID=4c9c6daa-7269-4f38-834e-795653a1666b] Cannot proceed with SSL Passthrough update, HTTPS is disabled
[NOTICE]   (69) : Reloading HAProxy
[NOTICE]   (69) : Initializing new worker (189)

Additional trace logs

On start up:

2025/03/04 19:44:54 TRACE   store/events.go:124 [transactionID=380b8f7f-0b83-4d2a-9119-52821f278906] Treating endpoints event {SliceName:prometheus Namespace:kube-dotcom-ingress-staging Service:prometheus Ports:map[http:0xc00317f9f0] Status:ADDED}
2025/03/04 19:44:54 TRACE   store/events.go:128 [transactionID=380b8f7f-0b83-4d2a-9119-52821f278906] service prometheus : endpoints list map[http:{Addresses:map[127.0.0.1:{}] Port:6060}]
2025/03/04 19:44:54 TRACE   store/events.go:133 [transactionID=380b8f7f-0b83-4d2a-9119-52821f278906] service prometheus : number of already existing backend(s) in this transaction for this endpoint: 0
2025/03/04 19:44:54 INFO    handler/prometheus.go:128 [transactionID=380b8f7f-0b83-4d2a-9119-52821f278906] reload required : creation/modification of prometheus endpoint`

Then this in a loop

2025/03/04 19:44:59 TRACE   store/events.go:119 [transactionID=9db59d19-6652-47f8-aff2-4f7ec4219e00] [RUNTIME] [BACKEND] [SERVER] [No change] [EventEndpoints]. No change for ADDED prometheus prometheus
2025/03/04 19:44:59 INFO    handler/prometheus.go:128 [transactionID=9db59d19-6652-47f8-aff2-4f7ec4219e00] reload required : creation/modification of prometheus endpoint

Due to the following repeated log we determined there may be a problem with how prometheus is configured

reload required : creation/modification of prometheus endpoint

Investigation and Attempts to Fix

• We identified a commit that appears to address the issue. However, it is only available in the nightly build and not in an official release.

• Deploying the nightly build did not resolve the issue. We continue to see the same behavior.

• We attempted to disable Prometheus by setting service.enablePorts.promtheus: false, but discovered that the flag is not checked when the PrometheusEndpoint handler is instantiated, meaning the flag has no effect.

Expected Behavior

• Prometheus should not trigger constant reloads.
• Disabling Prometheus should work as expected when service.enablePorts.promtheus: false is set.

Request for Assitance

• Is the fix from this commit to resolve this or is further work needed?
• Is there an alternative work around to properly disable Prometheus?
• Is there a way to properly configure Prometheus so there is no longer constant reloads?

values.yaml

nameOverride: kubernetes-ingress-staging
controller:
  ingressClass: haproxytech-unicorn
  ingressClassResource:
    name: haproxytech-unicorn
  config:
    frontend-config-snippet: |-
      acl site_dead nbsrv(github_http) lt 53
      monitor-uri /_upstream_healthz
      monitor fail if site_dead
      option dontlog-normal
    scale-server-slots: "100"
  readinessProbe:
    httpGet:
      path: /_upstream_healthz
  livenessProbe:
    httpGet:
      path: /_upstream_healthz
  extraArgs:
    - --configmap-tcp-services=kube-dotcom-ingress-staging/tcpservices-unicorn
    - --disable-https
    - --disable-ipv6
    - --namespace-whitelist=github-production
  image:
    tag: nightly
  PodDisruptionBudget:
    enable: true
    maxUnavailable: 1

  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "glb-ingress"
    effect: "NoSchedule"

  containerPort:
    http: 8081
    https: 443
  config:
    timeout-connect: "100ms"
    timeout-client: "180s"
    timeout-server: "180s"
    timeout-queue: "120s"
    timeout-tunnel: "180s"
    timeout-client-fin: "5s"
    timeout-server-fin: "5s"
    nbthread: "1"

    load-balance: "leastconn"
    check: "true"
    check-interval: "5s"
    stats-config-snippet: |
      option dontlog-normal

  defaultTLSSecret:
    enabled: false
  lifecycle:
    preStop:
      exec:
        command: ["/bin/sh", "-c", "sleep 5"]
  extraEnvs:
  - name: GOMAXPROCS
    value: "1"

  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
    httpGet:
      port: 8081
      scheme: HTTP

  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 120
    periodSeconds: 60
    successThreshold: 1
    timeoutSeconds: 1
    httpGet:
      port: 8081
      scheme: HTTP

  logging:
    level: info
    traffic:
      address: stdout
      format: raw
      facility: daemon

  publishService:
    enabled: false

  replicaCount: 1
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 1024Mi

  service:
    enablePorts:
      http: false
      https: false
      quic: false
      prometheus: false
    enabled: true
    tcpPorts:
    - name: http-tcp
      port: 80
      targetPort: 8080
    type: NodePort
  serviceMonitor:
    enabled: false
defaultBackend:
  enabled: false

[aaronbbrown]

Bringing some context from @hdurand0710 in the haproxy Slack into this issue:

Helene Durand
  30 minutes ago
Hello,
Can you confirm that:
ingress-controller has `--namespace-whitelist=github-production` as argument
 ingress-controller pod is deployed in kube-dotcom-ingress-staging namespace ?

Aaron Brown
  [29 minutes ago](https://haproxy.slack.com/archives/CR9N5UNAZ/p1741785935417449?thread_ts=1741716178.433189&cid=CR9N5UNAZ)
:wave: yes, that's correct (edited) 


Helene Durand
  [26 minutes ago](https://haproxy.slack.com/archives/CR9N5UNAZ/p1741786100518119?thread_ts=1741716178.433189&cid=CR9N5UNAZ)
ok then I could reproduce the problem.
The issue is that the "fake" ingress created for Prometheus is discarded as it's created in ns kube-dotcom-ingress-staging as this namespace is not in the whislit.
Then the Prometheus Backend is never created.
Each time there is a change in k8s, the sync loop first tries to create the Prometheus (detects that backend needs to be created and ask for a reload) but later on it's not created as discarded.

Aaron Brown
  27 minutes ago
would it help if I added kube-dotcom-ingress-staging to the whitelist?


Helene Durand
  [26 minutes ago](https://haproxy.slack.com/archives/CR9N5UNAZ/p1741786176499729?thread_ts=1741716178.433189&cid=CR9N5UNAZ)
yes if it does not harm (handling unwanted ingresses/svc...)


Aaron Brown
  25 minutes ago
We can definitely try it experimentally to validate your hypothesis!  I need to read the docs on that parameter, but I'm not sure if that will be a secure configuration for production, though.

[hdurand0710]

Adding from the previous slack.

Aaron Brown

  Yesterday at 6:28 PM

We did a trial deployment with both namespaces listed and it seems to have stopped the reloads due to the prometheus endpoint. :tada:

[6:28](https://haproxy.slack.com/archives/CR9N5UNAZ/p1741800527163479?thread_ts=1741716178.433189&cid=CR9N5UNAZ)

Does --namespace-whitelist do anything except control which Pods and Services the controller watches? I'm trying to decide if this is a sufficient workaround for now or not.

[6:29](https://haproxy.slack.com/archives/CR9N5UNAZ/p1741800545065659?thread_ts=1741716178.433189&cid=CR9N5UNAZ)

We also were surprised that disabling the prometheus endpoint entirely didn't have any effect on this behavior.

To answer the questions.

1. For --namespace-whitelist
   Yes, --namespace-whitelist impacts the K8s watchers to add a filtering on the namespaces that the different watches watch.
   For all objects, not only Pods and Services, so also including Secrets, Ingresses....

2. It needs also indeed some investigation why the disabling on the prometheus endpoint did not stop the behavior.

2 actions:

• The root cause is identified. It now needs to be fixed in the IC code so that if Prometheus (same with pprof) is enabled, the whitelist has no impact of preventing it to be created.
• Investigate and fix why the disabling of prometheus endpoint did not stop the constant reloads.

[hdurand0710]

Hi @aaronbbrown @yuxzhang97

Both actions mentionned just before are fixed in commit (3.1) and similar commits in other branches.

It will be available in the next release.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.