RDS Kubernetes Endpoints include pod name

[tbuchier]

Hello,

We are using the rds endpoints to probe some statefulsets in kubernetes.
We would like to have the pod name available in the endpoint resource so we can add it in the metrics labels for clearer alerting.

Indeed statefulsets pod name are "fixed" whereas pod IP is prone to change so right now our temporary solution is to do some PromQL join to link the pod ip with it's name to print it correctly in the alert message / dashboard.

I saw that the k8s pods rds type does contain the name but it doesn't allow specifying container port and in our case there is multiple port and we select the good one via it's name so we have to use k8s endpoints.

Currently how endpoints are built:

resources = append(resources, &pb.Resource{
	Name:   proto.String(resName),
	Ip:     proto.String(addr.IP),
	Port:   proto.Int(port.Port),
	Labels: epi.Metadata.Labels,
})

The idea would be to add the targetRef.name located in the endpoint subset (eps) if exist.

      {
        "ip": "10.28.0.3",
        "nodeName": "gke-cluster-1-default-pool-abd8ad35-ccr7",
        "targetRef": {
          "kind": "Pod",
          "namespace": "default",
          "name": "cloudprober-test-577cf7bbcc-c7l5p",
          "uid": "61b2a6f7-f9cc-11e9-a334-42010a8a00f9",
          "resourceVersion": "82681480"
        }
      },

Thanks :)

[manugarg]

I think this can be done -- we can add a target-name label to endpoints, and you can add that label to your metrics in cloudprober.cfg.

I am not sure yet if we should call it pod-name instead (if target kind is pod), to make it more clear, as target-name may be a little confusing given cloudprober's use of 'targets' keyword. I'll think more about it. Let me know if you've any preference.

[tbuchier]

I'd prefer pod-name too, more clear

[manugarg]

I was out for a couple of days. I've sent a change internally (@google); I should be able to submit it in a day or two. By the way, I am adding two labels to endpoints resources -- "node" and "pod".

@tbuchier, if it's okay, can you please share where are you using cloudprober. I mean, which org/company it is, or if it's a home-setup.

[manugarg]

1027e15 adds "pod" and "node" labels to "endpoints" resources. You can try these changes using the "cloudprober/cloudprober:master" image.

[manugarg]

Closing this issue. Added it to the v0.11.3 milestone (release).