Skip to content

The instance label has a collision with the prometheus hidden label #1143

@LinPr

Description

@LinPr

When want to access the /scrapeall path to get metrics from multiple mongo instance, the returned result carrys a instance lable among all the original labels .
Something like this: if I use curl to get the original labels through the /scrapeall path , with is a aggerated API that we can fetch multiple mongo instance within one http call

curl localhost:9216/scrapeall 
.....
mongodb_collstats_storageStats_totalIndexSize{cl_id="67da7c3a47edda1452408a7b",cl_role="mongos",collection="users",database="hello",instance="172.28.69.53:27017",shard="shard-000"} 4.1013248e+07
...

Here we can see when I direct access the mongo_exporter /scrapeall API, I get the metrics with label instance="172.28.69.53:27017"

But the problem is, when you use prometheues to scrabe the metrics, the metrics lables are relabeled slitently, the instance key added by mongo_exporter has a collision with the prometheus hidden label,this can refer to https://stackoverflow.com/questions/71380816/replace-exported-instance-label-value-from-prometheus

So the actual result scraped by prometheus is like

mongodb_collstats_storageStats_totalIndexSize{cl_id="67c2edaba41b0c2b174f009c", cl_role="mongos", collection="data_logs", container="mongodb-exporter", database="hello", endpoint="metrics", exported_instance="172.28.69.53:27017", instance="10.244.7.49:9216", job="prometheus-mongodb-exporter", namespace="monitoring", pod="prometheus-mongodb-exporter-55fd9fdc5c-qjcws", service="prometheus-mongodb-exporter", shard="shard-001"}

We can see prometheus add some internal labels and relabeled the original instance lable fetched from mongo_exported to exported_instance, and the instance label now is actually rewrited by prometheus, which means the ip address represent where the metrics from, that is the mongo_exporter itself

So I suggest not to use the instance label in the source code to represent the mongodb instance, which is so confusing at the first time when I try to do some filter with PromQL. Probably use mongo_instance as the label in mongo_exporter is better to avoiding the collision with prometheus.

If my suggestion is reasonable, please give me some feedbacks, and I can make a PR for this issue. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions