Ingress DaemonSet deployment fails with metrics enabled

[ronnybremer]

I was able to successfully deploy the helm chart with this config:

controller:
  kind: DaemonSet
  service:
    daemonset:
      useHostPort: true
    nodePorts:
      http: 30000
      https: 30001
      stat: 30002
      admin: 30003
    ipFamilyPolicy: RequireDualStack

All nodes receive their HAProxy and its exposed to the outside on the specified ports.

Later I tried to enable Prometheus metrics support. According to the documentation, this has to be added:

controller:
...
  stats:
    enabled: true
  metrics:
    enabled: true
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
    metrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname
    ctrlMetrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname

However, helm generated a service ingress yaml, which fails to deploy:

# Source: kubernetes-ingress/templates/controller-service-metrics.yaml
apiVersion: v1
kind: Service
metadata:
  name: haproxy-kubernetes-ingress-metrics
  namespace: haproxy-controller
  labels:
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/instance: haproxy-kubernetes-ingress
    helm.sh/chart: kubernetes-ingress-1.46.1
    app.kubernetes.io/version: "3.1.14"
    app.kubernetes.io/managed-by: Helm
  annotations:
spec:
  type: ClusterIP
  ports:
    - name: stat
      port: 1024
      protocol: TCP
      targetPort: stat
      nodePort: 30002
  selector:
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/instance: haproxy-kubernetes-ingress

Its type is set to ClusterIP but it lists the nodePort, which is invalid. Also, the targetPort is stat, which should be admin.

Is this an error on my end?

Furthermore, according to the documentation of the Prometheus helm chart, the ServiceMonitor should be deployed into the namespace of Prometheus (in my case: monitoring), not in the haproxy namespace.

[dkorunic]

From the documentation in values.yaml:

  ## Controller Service configuration
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/
  service:
    enabled: true     # set to false when controller.kind is 'DaemonSet' and controller.daemonset.useHostPorts is true

Have you tried with service.enabled: false, as above?

[ronnybremer]

Thank you for your suggestion @dkorunic. I tried that , but unfortunately the HAproxy no longer binds to any host ports:

Labels:           app.kubernetes.io/instance=haproxy-kubernetes-ingress
                  app.kubernetes.io/name=kubernetes-ingress
                  controller-revision-hash=5d9c5697c6
                  pod-template-generation=1
Annotations:      <none>
Status:           Running
IP:               10.244.4.55
IPs:
  IP:           10.244.4.55
  IP:           fd00:1:2:5::90bb
Controlled By:  DaemonSet/haproxy-kubernetes-ingress
Containers:
  kubernetes-ingress-controller:
    Container ID:    containerd://497afb16e8a559d4a17e69fce9731bab20d3129c765bb24b9870b45cad4abefd
    Image:           haproxytech/kubernetes-ingress:3.1.14
    Image ID:        docker.io/haproxytech/kubernetes-ingress@sha256:2778d2c353934c9c1ab597dd25735f4bb97f85e1c9c7068c9aa13073d71ef22a
    Ports:           6060/TCP (admin), 8080/TCP (http), 8443/TCP (https), 1024/TCP (stat), 8443/UDP (quic)
    Host Ports:      0/TCP (admin), 0/TCP (http), 0/TCP (https), 0/TCP (stat), 0/UDP (quic)
    SeccompProfile:  RuntimeDefault
    Args:
      --default-ssl-certificate=haproxy-controller/haproxy-kubernetes-ingress-default-cert
      --configmap=haproxy-controller/haproxy-kubernetes-ingress
      --http-bind-port=8080
      --https-bind-port=8443
      --quic-bind-port=8443
      --quic-announce-port=443
      --ingress.class=haproxy
      --publish-service=haproxy-controller/haproxy-kubernetes-ingress
      --log=info
      --prometheus
      --pprof

[dkorunic]

OK, we will investigate on our part and report back here.

[ronnybremer]

I think the issue is a little bit more complex. Apparently, with my configuration posted initially I was actually opening up a nodePort on each worker node, forwarded to a service or headless service and then to the pods of haproxy-ingress.
By disabling the service this routing no longer works and I would need to use hostPorts (which I wanted to have from the beginning).

The newly deployed daemonset does not contain any port mappings, maybe thats what's missing right now:

# Source: kubernetes-ingress/templates/controller-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: haproxy-kubernetes-ingress
  namespace: haproxy-controller
  labels:
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/instance: haproxy-kubernetes-ingress
    helm.sh/chart: kubernetes-ingress-1.46.1
    app.kubernetes.io/version: "3.1.14"
    app.kubernetes.io/managed-by: Helm
spec:
  minReadySeconds: 0
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: kubernetes-ingress
      app.kubernetes.io/instance: haproxy-kubernetes-ingress
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kubernetes-ingress
        app.kubernetes.io/instance: haproxy-kubernetes-ingress
    spec:
      enableServiceLinks: true
      serviceAccountName: haproxy-kubernetes-ingress
      terminationGracePeriodSeconds: 60
      dnsPolicy: ClusterFirst
      securityContext:
        runAsNonRoot: true
        runAsUser:  1000
        runAsGroup: 1000
      containers:
        - name: kubernetes-ingress-controller
          image: "haproxytech/kubernetes-ingress:3.1.14"
          imagePullPolicy: IfNotPresent
          args:
          - --default-ssl-certificate=haproxy-controller/haproxy-kubernetes-ingress-default-cert
          - --configmap=haproxy-controller/haproxy-kubernetes-ingress
          - --http-bind-port=8080
          - --https-bind-port=8443
          - --quic-bind-port=8443
          - --quic-announce-port=443
          - --ingress.class=haproxy
          - --publish-service=haproxy-controller/haproxy-kubernetes-ingress
          - --log=info
          - --prometheus
          - --pprof
          securityContext:
            runAsNonRoot: true
            runAsUser:  1000
            runAsGroup: 1000
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            seccompProfile:
              type: RuntimeDefault
          ports:
            - name: admin
              containerPort: 6060
              protocol: TCP
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: https
              containerPort: 8443
              protocol: TCP
            - name: stat
              containerPort: 1024
              protocol: TCP
            - name: quic
              containerPort: 8443
              protocol: UDP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 1042
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 1042
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          startupProbe:
            failureThreshold: 20
            httpGet:
              path: /healthz
              port: 1042
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 1
            successThreshold: 1
            timeoutSeconds: 1
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          resources:
            requests:
              cpu: 250m
              memory: 400Mi
          volumeMounts:
            - name: tmp
              mountPath: /tmp
              subPath: tmp
            - name: tmp
              mountPath: /run
              subPath: run
      volumes:
        - name: tmp
          emptyDir:
            medium: Memory
            sizeLimit: 64Mi

My current config is:

controller:
  kind: DaemonSet
  service:
    enabled: false
    daemonset:
      useHostPort: true
      hostPorts:
        http: 30000
        https: 30001
        quic: 30001
        stat: 30002
        admin: 30003
    ipFamilyPolicy: RequireDualStack
  stats:
    enabled: true
  metrics:
    enabled: true
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
    metrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname
    ctrlMetrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname

Something is missing here....

[ronnybremer]

Ok, one step further. Yaml got me again, this time ... :/
In the config from the previous post the lines for daemonset: are indented too much, they should be parallel to service:, not children of it.

With that change I see the host ports correctly mapped:

Labels:           app.kubernetes.io/instance=haproxy-kubernetes-ingress
                  app.kubernetes.io/name=kubernetes-ingress
                  controller-revision-hash=76f698cd56
                  pod-template-generation=1
Annotations:      <none>
Status:           Running
IP:               10.244.3.220
IPs:
  IP:           10.244.3.220
  IP:           fd00:1:2:4::a036
Controlled By:  DaemonSet/haproxy-kubernetes-ingress
Containers:
  kubernetes-ingress-controller:
    Container ID:    containerd://d75884665dcfa98290433713a523bcf005380278f9dae8a2db750fa2b7109f3e
    Image:           haproxytech/kubernetes-ingress:3.1.14
    Image ID:        docker.io/haproxytech/kubernetes-ingress@sha256:2778d2c353934c9c1ab597dd25735f4bb97f85e1c9c7068c9aa13073d71ef22a
    Ports:           6060/TCP (admin), 8080/TCP (http), 8443/TCP (https), 1024/TCP (stat), 8443/UDP (quic)
    Host Ports:      30003/TCP (admin), 30000/TCP (http), 30001/TCP (https), 30002/TCP (stat), 30001/UDP (quic)
    SeccompProfile:  RuntimeDefault
    Args:
      --default-ssl-certificate=haproxy-controller/haproxy-kubernetes-ingress-default-cert
      --configmap=haproxy-controller/haproxy-kubernetes-ingress
      --http-bind-port=8080
      --https-bind-port=8443
      --quic-bind-port=8443
      --quic-announce-port=443
      --ingress.class=haproxy
      --publish-service=haproxy-controller/haproxy-kubernetes-ingress
      --log=info
      --prometheus
      --pprof

But the ports are not open yet on the node itself. I will investigate further.

[ronnybremer]

All right, the deployment now listens correctly on the host ports assigned. The issue is, that when selecting host ports they need to be outside the nodePort range of the K8s setup (at least with Cilium). Once I changed them to 80/443/8008/8009 the ports opened up as expected.

The new config:

controller:
  kind: DaemonSet
  service:
    enabled: false
    ipFamilyPolicy: RequireDualStack
  daemonset:
    useHostPort: true
    hostPorts:
      http: 80
      https: 443
      quic: 443
      stat: 8008
      admin: 8009
  stats:
    enabled: true
  metrics:
    enabled: true
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
    metrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname
    ctrlMetrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname

Now onto the monitoring part. As noted at the beginning, the service monitor object is deployed into the wrong namespace. I deleted it there and moved it to the monitoring namespace. Still had to replace the port name metrics with admin.

Prometheus doesn't pick it up though, as I assume the selector only looks for services, which I no longer have. I'll check on that.

[ronnybremer]

Ok, when deployed in daemonset mode a PodMonitor seems to be the only viable option. I manually deployed this config, which is based on the config generated by the helm chart:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: haproxy-kubernetes-ingress
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/instance: haproxy-kubernetes-ingress
    release: kube-prom-stack
spec:
  podMetricsEndpoints:
  - port: admin
    path: /metrics
    interval: 30s
    scheme: http
  namespaceSelector:
    matchNames:
    - haproxy-controller
  selector:
    matchLabels:
      app.kubernetes.io/name: kubernetes-ingress
      app.kubernetes.io/instance: haproxy-kubernetes-ingress

It gets picked up and Prometheus shows one target for each pod. I guess the helm chart needs to take this into account, when daemonset mode is selected, a corresponding PodMonitor needs to be deployed, in all other cases the ServiceMonitors makes sense.

[ronnybremer]

Small correction, the correct port in the yaml above should be stat, not admin, otherwise we get the metrics from the API ...

[ronnybremer]

Ok, got it all to work with this monitoring yaml:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: haproxy-ingress
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/instance: haproxy-ingress
    release: kube-prom-stack
spec:
  jobLabel: app.kubernetes.io/instance
  podMetricsEndpoints:
  - port: stat
    path: /metrics
    interval: 30s
    scheme: http
    relabelings:
    - replacement: k8s-prod
      targetLabel: cluster
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: hostname
  - port: admin
    path: /metrics
    interval: 30s
    scheme: http
    relabelings:
    - replacement: k8s-prod
      targetLabel: cluster
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: hostname
  namespaceSelector:
    matchNames:
    - haproxy-controller
  selector:
    matchLabels:
      app.kubernetes.io/name: kubernetes-ingress
      app.kubernetes.io/instance: haproxy-ingress

So in a nutshell (when using daemonset):

• controller.service needs to be enabled: false
• nodePorts needs to be hostPorts
• make sure hostPorts are outside the cluster node port range (< 30000)
• instead of ServiceMonitor we need PodMonitor
• two endpoints need to be monitored inside of PodMonitor (one for port stat and one for port admin)
• deploy to the monitoring namespace (unless Prometheus Operator is configured to scan all namespaces)
• set a jobLabel to match the expectations of the Grafana dashboard (this assumes the HAproxy ingress controller to be deployed via Helm with the name haproxy-ingress)

Maybe a few of these things could be done in the helm chart, when daemonset is requested, but I am not sure.