Service Discovery for multiple ports/paths on same FQDN

[snarlysodboxer]

Some use cases may need to scrape multiple endpoints on the same FQDN, such as localhost.

For example, in a Kubernetes pod you might have multiple metrics endpoints to scrape, localhost:9100/metrics, localhost:9101/my-metrics, etc.

You can't just run another copy of the client and still get service discovery, since the client only forwards the FQDN.

What do you think about allowing the client to specify labels, ports, and metrics paths for discovery?

One way to do this might be changing --fqdn to --endpoint and maybe allowing it to be specified multiple times. An example --endpoint value might be http://localhost:9100/metrics?app=myapp, where app=myapp would be turned into labels for discovery.

I can work on a PR if you think this or similar is a good idea.

Additionally, the same values could optionally be used for security, as talked about in PR #41.

[brian-brazil]

What you need here is the actual FQDN, not just localhost.

[snarlysodboxer]

I'm assuming you mean run the client in a separate pod In Kubernetes, and then hook them up via a service. That can mean significantly more configuration, especially if using RBAC, but it's doable.

But it still doesn't give discovery for metrics paths or labels, which be significantly nicer than having to write custom relabeling config logic.

[brian-brazil]

If you're in Kubernetes you can use k8 service discovery, and there should be no need for pushprox as Prometheus will be on the same network.

[snarlysodboxer]

We actually have many Kubernetes clusters we need to monitor, and for security reasons we can't reach into them, only out of them, thus PushProx.

[brian-brazil]

The recommended way to monitor k8 is from inside k8, which will make all of this much simpler.

[snarlysodboxer]

I'm aware. The fact is we can't reach into these clusters to get metrics out, to view/alert on them, thus PushProx. The purpose of PushProx is to work around a network barrier just like we have to work around, there's no argument here.

This issue is around the idea of better service discovery, requiring less custom code to hit /clients and equivalent prometheus.yml entries, and less configuration to manage with the k8s objects.

[brian-brazil]

I'm saying to run Prometheus inside k8.

[snarlysodboxer]

How would I use Prometheus or hook it up to Grafana or other since I can't reach into the cluster? For security reasons, I cannot expose any ingress into the clusters.

[brian-brazil]

That's a bit of a self-inflicted problem, but you could use tunnels such as pushprox or ssh.

[snarlysodboxer]

These are clusters that we standup for clients and then remove our own access to but still need to monitor. It is part of what we do. I don't know why you're arguing with me about it, it seems like this is the kind of situation that pushprox is meant for.

Thank you for suggesting pushprox, that's why I'm here in the first place.

[brian-brazil]

Ultimately pushprox is meant to bypass network restrictions, not be an arbitrary SD system.

[snarlysodboxer]

I hear that, but it supports very limited service discovery for a narrow range of use cases. Is there no plan to allow it to broaden?

[brian-brazil]

Not currently, everything is machine-based.

[snarlysodboxer]

Even with separate FQDNs like my-svc-1.my-namespace.svc.cluster.local and my-svc-2.my-namespace.svc.cluster.local, if they happen to run their metrics on separate ports, I can no longer use service discovery, unless I setup 2 instances of the proxy and have 2 different config sections in the prometheus config and have 2 different file writers for file_sd_configs.

...It would be so much easier to just specify the port on the client side.

[snarlysodboxer]

With PushProx's current setup, I have to build automation of k8s Deployment and Service objects on the proxy client side for N clients in order to get differentiated hostnames and service discovery, essentially defeating the purpose of service discovery.

If instead the client could send it's FQDN/port/label list, no automation would be needed, other than writing the results of GETing /clients to a file for file_sd_configs.