k8s operator: Unclear steps to enable Prometheus discovery on CockroachDB K8s Operator created clusters

[sheaffej]

When following our docs to create a CockroachDB cluster using the CockroachDB K8s operator, it's unclear how to cause Prometheus to discovery the CockroachDB nodes that were created by the operator.

There are pieces of information in multiple places, however it unclear to a user what steps are actually needed in order to succeed in Prometheus discovering the CockroachDB pods and scraping their metrics.

The documentation page below describes what I believe to be the necessary steps, however most of the steps on this page related to installing a new Prometheus server.

https://www.cockroachlabs.com/docs/v22.1/monitor-cockroachdb-kubernetes

Most users will already have a Prometheus server in operation, therefore the majority of these steps are not required, and if re-run would create problems in their Prometheus environment. And it is not clear which subset of steps are needed when a Prometheus server already exists.

The issue cockroachdb/cockroach-operator#469 on the operator discusses more what is required for Prometheus to properly discover the CockroachDB pods, which sheds some more light on the required steps.

The exact steps should be confirmed and then tested, however as filing this issue, these are the steps I believe to be required.

Pre-conditions:

• An existing Prometheus server is operating, and has successfully discovered other pods running in the k8s cluster (i.e. the Prometheus configuration is complete and correct)
• A CockroachDB cluster has been created using the CockroachDB K8s Operator

Steps:

Step 1: apply the label to the CockroachDB service:

kubectl label svc cockroachdb prometheus=cockroachdb

Step 2: Create a ServiceMonitor object in k8s to tell Prometheus to look for that label, and how to access the root certificate. Put this into a .yaml file and then use kubectl apply -f <filename>.yaml

# Select any services with the prometheus:cockroachdb label
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cockroachdb
  labels:
    app: cockroachdb
    prometheus: cockroachdb
spec:
  selector:
    matchLabels:
      prometheus: cockroachdb
  endpoints:
  - port: http
    path: /_status/vars
    tlsConfig:
      ca:
        secret:
          key: ca.crt
          # This is the secret name used by the CockroachDB Kubernetes Operator.
          # When using a custom CA, replace this with your secret name
          name: cockroachdb-node
      serverName: "127.0.0.1"

If there are field within the ServiceMonitory object definition that should be customized by the user, then the documentation should highlight those fields, and provide a description of what detail should be provided in those field.s