Lack of server label - Multi-Target Support

[rafal-skurka]

Some of the metrics don't have a "server" label it's quite problematic in Multi-Target Support because we can't distinguish database servers and some useful graphs in the Grafana dashboard view incorrect data. In my opinion, that label should be added to every single metric to avoid mixing up data between different database servers.

Number of metrics with and without the "server" label

# with
➜  x git:(x) ✗ curl -s localhost:9187/metrics | grep -ci server 
    1465
#without 
➜  x git:(x) ✗ curl -s localhost:9187/metrics | grep -civ server 
    1338

Example metric without server label:

pg_database_size_bytes{container="prometheus-postgres-exporter", datname="green", endpoint="http", instance="192.168.226.57:9187", job="postgres-exporter-prometheus-postgres-exporter", namespace="monitoring", pod="postgres-exporter-prometheus-postgres-exporter-84675c5cfc-hbt77", service="postgres-exporter-prometheus-postgres-exporter"}
pg_database_size_bytes{container="prometheus-postgres-exporter", datname="blue", endpoint="http", instance="192.168.226.57:9187", job="postgres-exporter-prometheus-postgres-exporter", namespace="monitoring", pod="postgres-exporter-prometheus-postgres-exporter-84675c5cfc-hbt77", service="postgres-exporter-prometheus-postgres-exporter"}

Expected output from this metric:

pg_database_size_bytes{container="prometheus-postgres-exporter", datname="green",<trimmed>,server="green-server:5432"}
pg_database_size_bytes{container="prometheus-postgres-exporter", datname="blue",<trimmed>,server="blue-server:5432"}

[SuperQ]

Sounds like your relabel config isn't correct. The server label should come from the relabeling.

[portnov]

Hello
I have a similar issue.
Even without Prometheus by itself, I just run prometheus-exporter and go to localhost:9187/metrics. And I see that some metrics have server= label, and some do not. Moreover, some metrics appear twice, once with this label and once without it:

pg_database_size_bytes{datname="pgbench"} 5.1106607e+07
pg_database_size_bytes{datname="pgbench",server="host.docker.internal:5432"} 5.1106607e+07

(the value seem to be always equal for both records).
Going to /probe?target=... does not help, I see the same picture.
Since I see this in exporter output, I do not think it has something to do with prometheus config...

[KulykDmytro]

Hi
at 0.15 running for of RDS instance (in kubernetes) we observing following behaviour. on default settings + queries.yaml
These collectors provides server label, other not:

• all metrics passed over queries.yaml
• pg_settings
• pg_stat_activity
• pg_stat_archiver
• pg_stat_database (only for conflicts_confl section)
• pg_static