Merge pull request openshift#10 from pavolloffay/distributed-tracing-3.0-prometheus-podmonitor

TRACING-3513 | Document that OTEL operator can create service monitors

Author: max-cx

modules/distr-tracing-otel-config-collector.adoc

@@ -26,10 +26,9 @@ metadata:
   namespace: tracing-system
 spec:
   mode: deployment
-  ports:
-  - name: promexporter
-    port: 8889
-    protocol: TCP
+  observability:
+    metrics:
+      enableMetrics: true
   config: |
     receivers:
       otlp:

modules/distr-tracing-otel-config-send-metrics-monitoring-stack.adoc

@@ -6,7 +6,35 @@ This module is included in the following assemblies:
 [id="distr-tracing-send-metrics-monitoring-stack_{context}"]
 = Sending metrics to the monitoring stack
 
-You can configure the monitoring stack to scrape OpenTelemetry Collector metrics endpoints and to remove duplicated labels that the monitoring stack has added during scraping.
+You can configure the OpenTelemetry Collector custom resource (CR) to create a Prometheus `ServiceMonitor` CR to scrape the collector's pipeline metrics and the enabled Prometheus exporters.
+
+.Example of the OpenTelemetry Collector custom resource with the Prometheus exporter
+[source,yaml]
+----
+spec:
+  mode: deployment
+  observability:
+    metrics:
+      enableMetrics: true <1>
+  config: |
+    exporters:
+      prometheus:
+        endpoint: 0.0.0.0:8889
+        resource_to_telemetry_conversion:
+          enabled: true # by default resource attributes are dropped
+    service:
+      telemetry:
+        metrics:
+          address: ":8888"
+      pipelines:
+        metrics:
+          receivers: [otlp]
+          exporters: [prometheus]
+----
+<1> Configures the operator to create the Prometheus `ServiceMonitor` CR to scrape the collector's internal metrics endpoint and Prometheus exporter metric endpoints. The metrics will be stored in the OpenShift monitoring stack.
+
+
+Alternatively, the Prometheus `PodMonitor` can be created manually, which offers more fine-grained control, for instance remove duplicated labels added during Prometheus scraping.
 
 .Sample `PodMonitor` custom resource (CR) that configures the monitoring stack to scrape Collector metrics
 [source,yaml]
@@ -18,7 +46,7 @@ metadata:
 spec:
   selector:
     matchLabels:
-      app.kubernetes.io/name: otel-collector
+      app.kubernetes.io/name: `<cr-name>-collector` <1>
   podMetricsEndpoints:
   - port: metrics <1>
   - port: promexporter <2>
@@ -35,5 +63,6 @@ spec:
     - action: labeldrop
       regex: job
 ----
-<1> The name of the internal metrics port for the OpenTelemetry Collector. This port name is always `metrics`.
-<2> The name of the Prometheus exporter port for the OpenTelemetry Collector. This port name is defined in the `.spec.ports` section of the `OpenTelemetryCollector` CR.
+<1> The name of the OpenTelemetry custom resource.
+<2> The name of the internal metrics port for the OpenTelemetry Collector. This port name is always `metrics`.
+<3> The name of the Prometheus exporter port for the OpenTelemetry Collector.