RHDEVDOCS-3924 - document option to add cluster id to off-cluster int…

…egrations

modules/monitoring-adding-cluster-id-labels-to-metrics.adoc

@@ -0,0 +1,20 @@
+// Module included in the following assemblies:
+//
+// * monitoring/configuring-the-monitoring-stack.adoc
+
+:_content-type: CONCEPT
+[id="adding-cluster-id-labels-to-metrics_{context}"]
+= Adding cluster ID labels to metrics
+
+If you manage multiple {product-title} clusters and use the remote write feature to send metrics data from these clusters to an external storage location, you can add cluster ID labels to identify the metrics data coming from different clusters. 
+You can then query these labels to identify the source cluster for a metric and distinguish that data from similar metrics data sent by other clusters.
+
+This way, if you manage many clusters for multiple customers and send metrics data to a single centralized storage system, you can use cluster ID labels to query metrics for a particular cluster or customer.
+
+Creating and using cluster ID labels involves three general steps:
+
+* Configuring the write relabel settings for remote write storage.
+
+* Adding cluster ID labels to the metrics.
+
+* Querying these labels to identify the source cluster or customer for a metric.

modules/monitoring-creating-cluster-id-labels-for-metrics.adoc

@@ -0,0 +1,95 @@
+// Module included in the following assemblies:
+//
+// * monitoring/configuring-the-monitoring-stack.adoc
+
+:_content-type: PROCEDURE
+[id="creating-cluster-id-labels-for-metrics_{context}"]
+= Creating cluster ID labels for metrics
+
+You can create cluster ID labels for metrics for default platform monitoring and for user workload monitoring.
+
+For default platform monitoring, you add cluster ID labels for metrics in the `write_relabel` settings for remote write storage in the `cluster-monitoring-config` config map in the `openshift-monitoring` namespace. 
+
+For user workload monitoring, you edit the settings in the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` namespace.
+
+.Prerequsites
+
+* You have installed the OpenShift CLI (`oc`).
+* You have configured remote write storage.
+* *If you are configuring default platform monitoring components:*
+** You have access to the cluster as a user with the `cluster-admin` role.
+** You have created the `cluster-monitoring-config` `ConfigMap` object.
+* *If you are configuring components that monitor user-defined projects:*
+** You have access to the cluster as a user with the `cluster-admin` role or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
+** You have created the `user-workload-monitoring-config` `ConfigMap` object.
+
+.Procedure
+
+. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
+----
++
+[NOTE]
+====
+If you configure cluster ID labels for metrics for the Prometheus instance that monitors user-defined projects, edit the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` namespace. 
+Note that the Prometheus component is called `prometheus` in this config map and not `prometheusK8s`, which is the name used in the `cluster-monitoring-config` config map. 
+====
+
+. In the `writeRelabelConfigs:` section under `data/config.yaml/prometheusK8s/remoteWrite`, add cluster ID relabel configuration values:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-monitoring-config
+  namespace: openshift-monitoring
+data:
+  config.yaml: |
+    prometheusK8s:
+      remoteWrite:
+      - url: "https://remote-write-endpoint.example.com"
+        <endpoint_authentication_credentials>
+        writeRelabelConfigs: <1>
+          - <relabel_config> <2>
+----
+<1> Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.
+<2> Substitute the label configuration for the metrics sent to the remote write endpoint.
++
+The following sample shows how to forward a metric with the cluster ID label `cluster_id` in default platform monitoring:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-monitoring-config
+  namespace: openshift-monitoring
+data:
+  config.yaml: |
+    prometheusK8s:
+      remoteWrite:
+      - url: "https://remote-write-endpoint.example.com"
+        writeRelabelConfigs:
+        - sourceLabels:
+          - __tmp_openshift_cluster_id__ <1>
+          targetLabel: cluster_id <2>
+          action: replace <3>
+----
+<1> The system initially applies a temporary cluster ID source label named `+++__tmp_openshift_cluster_id__+++`. This temporary label gets replaced by the cluster ID label name that you specify.
+<2> Specify the name of the cluster ID label for metrics sent to remote write storage. 
+If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label.
+For the label name, do not use `+++__tmp_openshift_cluster_id__+++`. The final relabeling step removes labels that use this name.
+<3> The `replace` write relabel action replaces the temporary label with the target label for outgoing metrics. 
+This action is the default and is applied if no action is specified.
+
+. Save the file to apply the changes to the `ConfigMap` object.
+The pods affected by the updated configuration automatically restart.
++
+[WARNING]
+====
+Saving changes to a monitoring `ConfigMap` object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.
+====

monitoring/configuring-the-monitoring-stack.adoc

@@ -97,6 +97,15 @@ include::modules/monitoring-supported-remote-write-authentication-settings.adoc[
 * See xref:../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets-about_nodes-pods-secrets[Understanding secrets] for steps to create and configure `Secret` objects in {product-title}.
 * See the xref:../rest_api/monitoring_apis/prometheus-monitoring-coreos-com-v1.adoc#spec-remotewrite-2[Prometheus REST API reference for remote write] for information about additional optional fields.
 
+// Configuring labels for outgoing metrics
+include::modules/monitoring-adding-cluster-id-labels-to-metrics.adoc[leveloffset=+1]
+include::modules/monitoring-creating-cluster-id-labels-for-metrics.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* For details about write relabel configuration, see xref:../monitoring/configuring-the-monitoring-stack.adoc#configuring_remote_write_storage_configuring-the-monitoring-stack[Configuring remote write storage].
+
 // Managing scrape sample limits for user-defined projects
 include::modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc[leveloffset=+1]
 include::modules/monitoring-setting-a-scrape-sample-limit-for-user-defined-projects.adoc[leveloffset=+2]
@@ -129,7 +138,6 @@ include::modules/monitoring-setting-query-log-file-for-prometheus.adoc[leveloffs
 
 [role="_additional-resources"]
 .Additional resources
-
 * See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps
 * See xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] for steps to enable user-defined monitoring.