nginx_ingress_controller_requests is missing for Ingress that has had no requests

[lambchr]

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): 1.15

Environment:

• EKS k8s cluster

What happened:

The nginx_ingress_controller_requests metric was missing for Ingresses that have had 0 requests. I queried nginx_ingress_controller_requests == 0 for our prometheus metrics and found no time series.

What you expected to happen:

I expected ingresses that have had no requests sent to them to have a nginx_ingress_controller_requests metric with a count of 0 rather than not being present.

How to reproduce it:

• I deployed a test Ingress object (very basic config) with the host name set
• I waited for the Ingress controller to show the host and address, I've seen other bugs raised that state this metric won't show if the host isn't set but I don't think that applies here

$ kubectl --context <CONTEXT> -n <NAMESPACE> get ing test
NAME       HOSTS          ADDRESS                 PORTS     AGE
test       <DOMAIN>  <LB_ADDRESS>      <PORTS>   7h40m

• I looked for the metric in prometheus for this ingress but couldn't find it nginx_ingress_controller_requests{ingress="test"}
• I also exec'd to a pod on the cluster and hit the NGINX controller's /metrics endpoint to check if our prometheus stack was filtering out the metric, but it didn't appear there either

Anything else we need to know:

Please let me know if you need any other info from me, thanks for your time :)

/kind bug

[kundan2707]

/ assign

[strongjz]

Please upgrade to 0.46.0 and see if this is still an issue.

[strongjz]

/triage needs-information

[lambchr]

Hi @strongjz sorry for the delayed response. I upgrade to 0.47.0 and repeated my above test (created new debug ingress, waited for host and address to appear, checked if metric appeared) and could still not see any nginx_ingress_controller_requests metrics from the /metrics endpoint of any NGINX pod in the cluster. When I curled my new debug ingress and checked the NGINX pods /metrics endpoint again it showed me an nginx_ingress_controller_requests metric for that ingress. Please let me know if you need any more info from me :)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[lambchr]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[lambchr]

/remove-lifecycle stale

Hey @strongjz do you need any more information?

[philwelz]

i can confirm this behavior on version v1.0.4 / 4.0.6 where the metric nginx_ingress_controller_requests is missing until you do any request (for example curl) to an ingress resource.

[iamNoah1]

/triage accepted
/priority important-longterm

We are happy for any contributions
/help
/good-first-issue

[k8s-ci-robot]

@iamNoah1:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/triage accepted
/priority important-longterm

We are happy for any contributions
/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iamNoah1]

/remove needs-information

[iamNoah1]

/remove triage/needs-information

[iamNoah1]

/remove triage-needs-information

[gauravkghildiyal]

The issue seems to be a bit old, but I'm presenting my thoughts here to provide some closure to others who might see this.

I feel the problem at hand is that the metric itself could have multiple labels, and those labels could have multiple values. This makes it harder to initialize nginx_ingress_controller_requests for all combination of labels and values (maybe, some values might not even be known until the request arrives). Something like prometheus/client_golang#190 might be helpful but even that might have problems with dynamic values of labels.

[johurul000]

@iamNoah1 I am new to open source and Kubernetes, and I would like to work on this issue.
Extra help would be great. 🙂🙂

[longwuyuan]

@johurul000 can you write your own technical description of the problem to be solved here

[johurul000]

@longwuyuan no, since I am new, guidance is very helpful

[longwuyuan]

I mean is the metric missing in your cluster also ?

[kingli-crypto]

Hope it helps a fellow user, I was using wildcard domain and I have to add --metrics-per-host=false for this metric to work.

[Aut0R3V]

Hey, is anyone still working on this? I think I can take it up if someone can guide me on how to achieve this.

[DanielViniciusAlves]

Hey @strongjz and @lambchr, the problem appointed in this issue still need to be fixed? If so I would like to work on this issue.

[jmulcahyTSC]

I too am running into this issue when trying to implement Flagger which refers to this metric in the documents. Would love an update on this.

[SehiiRohoza]

I'm running into this issue while using NGINX Ingress Prometheus Overview in GCP Monitoring with the latest version.

[fyaretskiy]

imo this is a prometheus issue, they could introduce a function that returns a 0 if null. This isn't an issue specific to nginx_ingress_controller_requests.

[SehiiRohoza]

imo this is a prometheus issue, they could introduce a function that returns a 0 if null. This isn't an issue specific to nginx_ingress_controller_requests.

Have you tried to report this "issue" to Prometheus? If yes, please share the link to your report.

[fyaretskiy]

imo this is a prometheus issue, they could introduce a function that returns a 0 if null. This isn't an issue specific to nginx_ingress_controller_requests.

Have you tried to report this "issue" to Prometheus? If yes, please share the link to your report.

I haven't, I have read read somewhere it's the responsibility of the metric reporter to pre populate the 0's. I can't find the github issue page now.

[BenHesketh21]

I can easily reproduce this and can see that the metric doesn't appear until requests come through. However, I'm not sure there's an obvious fix. We could just set the metric to 0 when an ingress object is found, but the metric includes labels like: method, path and status (as in http status code). What should these be set too when we set the value to 0? I'm not sure there's a nice way to do this so that the metric appears right away and we don't end up with metrics that will always be 0 because the default labels we choose are never matched with a real request.

[StuxxNet]

@rikatz I know that I have a followup for the tests, but if you don't mind I would like to work on this issue :D

[rikatz]

/assign @StuxxNet
Go for it

[nisharyan]

Any update on this?