Multiple nginx instances

[libesz]

Is your feature request related to a problem? Please describe.
I am running a webhosting service with multiple nginx instances (one common for ingress and one for each hosted site).

Describe the solution you'd like
It would be great if I can collect all the metrics with one instance of this exporter. Also, attaching labels for each nginx target would be nice.

Describe alternatives you've considered
Run dedicated instances of this exporter and let prometheus to label all sources so I can differentiate between them.

Additional context

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[aavshr]

are there are any updates on this? I can work on this if the maintainers believe we should do it. We also have multiple nginx instances deployed and would be ideal to run only a single instance of the exporter.

[libesz]

@aavshr I don't have any. I also considered to start working on it myself, but don't have much free cycles nowadays to proceed. Anyways, I am more than happy to test/review it if there will be a PR.

[lucacome]

Hi @libesz sorry for the long wait for a response. From what I can tell this goes against Prometheus best practices.
From https://prometheus.io/docs/instrumenting/writing_exporters/#deployment

Each exporter should monitor exactly one instance application, preferably sitting right beside it on the same machine. That means for every HAProxy you run, you run a haproxy_exporter process. For every machine with a Mesos worker, you run the Mesos exporter on it, and another one for the master, if a machine has both.

The theory behind this is that for direct instrumentation this is what you’d be doing, and we’re trying to get as close to that as we can in other layouts. This means that all service discovery is done in Prometheus, not in exporters. This also has the benefit that Prometheus has the target information it needs to allow users probe your service with the blackbox exporter.

Then it goes on about special cases like blackbox but they use a specific pattern https://prometheus.io/docs/guides/multi-target-exporter/ that seems like a slightly different approach from the PR linked here.

So I'm not really sure about implementing this...what are the uses cases for this? Are you (or other people that liked and commented here) using other exporters that implement multi targets one way or another?
I'm trying to figure what's the best way to handle this and if there's like a common pattern used.

[Derioss]

If you use nginx as applicative router, you can have multiple instance on same vm, same docker.
Actually, i deploy one instance by nginx, i have 4 exporter on the same docker machine.
honnestly, i prefer deploy one exporter by docker machine, but is not mandatory, is not killing feature.
(and thanks for your's works).

[libesz]

@lucacome Thanks for the details! Generally I agree with the dedicated instanceusage and the decomposition as much as possible with containers/microservices.
The project where I would use this feature is a single-node web-hosting solution, where for example the HA requirements are not that hard. This single machine already runs 50+ containers for the sake of the proper isolation (runs dedicated nginx+fcgi workload containers of different web sites for security purposes). In this use-case, adding ~10 instances of the same specific exporter would not add much benefit, because the co-location to the app is already given and also the administrative domain is the same (web-site users does not really manage anything on the server).

I can accept if this is a marginal use-case and against the best practices. 👍

[pizycki]

So I'm not really sure about implementing this...what are the uses cases for this?

In my case, I have HA nginx hosted on kuberentes managed by HPA. By requesting an nginx for metrics via kubernetes service, I request only one of the running instances of nginx at the time.

When setting up a ServiceMonitor, I can set a selector field to specify which pods should be retrieved from cluster and requested directly by IP address (bypassing kubernetes service). In this way, I collect metrics from all app instances.

[pizycki]

Well, my problem (described above) can be solved with deploying nginx exporter as sidecar inside the pod along with nginx.

I no longer see a valid point for implementing this feature in nginx exporter, so I'm taking back my "👍 "

[brandon1024]

Edit

It turns out that the metrics exposed by nginx stub_status are global to the nginx instance, not individual server configurations. Exposing stub_status in the way I demonstrated below isn't all that useful because the metrics will be the same for each one.

Leaving my original comment for context :-)

Original Comment

Apologies for reopening this can of worms.

It's reasonably clear that it's generally better to run a single exporter for each monitored application. What I haven't seen discussed here is the case where we are monitoring a single nginx instance that proxies a few upstream servers. For example:

events {}
http {
        server_names_hash_bucket_size 64;

        server {
                listen 8080;
                listen [::]:8080;

                server_name s1.home.me.ca;

                include nginx.conf.d/letsencrypt.conf;

                location / {
                        proxy_pass http://n1.pi.lan:8081;
                        include nginx.conf.d/proxy.conf;
                }

               location = /stub_status {
                        stub_status;
               }
        }

        server {
                listen 8080;
                listen [::]:8080;

                server_name s2.home.me.ca;

                include nginx.conf.d/letsencrypt.conf;

                location / {
                        proxy_pass http://n1.pi.lan:8082;
                        include nginx.conf.d/proxy.conf;
                }

               location = /stub_status {
                        stub_status;
               }
        }
}

Here, we have a single instance with two metrics endpoints. Managing separate exporters for each one is a bit cumbersome--especially in my environment, where I have a single nginx instance proxying 10 or so origins.

I'd like to hear from those involved in this issue. Is best practice still to create a single exporter for each upstream? I had the same idea as @libesz to tackle this one myself and open an PR, but I don't want to start work on something that ultimately won't get merged.