Collect metrics of QPS per backend

[yastij]

Hello, Currently I'm using Horizontal Pod Autoscaler, and I'm trying to scale based on QPS per backend.

Seems that currently we do not have something like :

ingress_http_hits_total{pod="foo",service="bar",ingress="baz",namespace="ns"}

[hoffoo]

@aledbf how should one go about this?

I imagine a goroutine here https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/metric/collector/vts.go which scrapes periodically to get request count and calculate rps.

[yastij]

I do not currently have enough bandwidth to work on this one, but if no one volunteers feel free to assign me

[hoffoo]

@yastij i would like to work on it, i just can't think of clean way to implement it. The only idea i have is querying nginx_status periodically since it doesnt export a raw qps. Would that idea be mergable:

1. loop periodically to get request counts per backend
2. calculate qps and serve along with the other metrics

edit: i made a prototype of what this might look like https://github.com/hoffoo/nginx-stats-exporter/blob/master/main.go

the metrics look like this:

# TYPE http_rps_per_backend gauge
http_rps_per_backend{server="10.221.0.13:8000",upstream="default-foo-8000"} 0
http_rps_per_backend{server="10.221.0.181:9091",upstream="default-bar-9091"} 63.78920086537701
http_rps_per_backend{server="10.221.0.196:5000",upstream="default-baz-5000"} 1.8700417191804337
http_rps_per_backend{server="10.221.0.197:5000",upstream="default-baz-5000"} 2.2856065456649746

let me know if this looks like the right idea

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[aledbf]

Closing. This was implemented in #2608

[huanghantao]

Hi, how to open this metrics? I have configured prometheus.io/scrape for ingress-nginx deployment, but it seems that rps per backend is missing.

The image is k8s.gcr.io/ingress-nginx/controller:v1.0.5.

My core configuration is as follows:

# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"

ports:
  - name: http
    containerPort: 80
    hostPort: 80
    protocol: TCP
  - name: https
    containerPort: 443
    hostPort: 443
    protocol: TCP
  - name: webhook
    containerPort: 8443
    protocol: TCP
  - name: prometheus
    containerPort: 10254
    protocol: TCP

I can confirm that prometheus has been able to fetch metrics but has not found rps per backend.