Merge pull request openshift#11 from pavolloffay/distributed-tracing-3.0-span-red

Document Tempo monitor tab and span RED metrics

Author: max-cx

distr_tracing/distr_tracing_tempo/distr-tracing-tempo-configuring.adoc

@@ -21,6 +21,8 @@ include::modules/distr-tracing-tempo-config-storage.adoc[leveloffset=+2]
 
 include::modules/distr-tracing-tempo-config-query-frontend.adoc[leveloffset=+2]
 
+include::modules/distr-tracing-tempo-config-spanmetrics.adoc[leveloffset=+2]
+
 [id="setting-up-monitoring-for-tempo"]
 == Setting up monitoring for the {TempoShortName}
 

modules/distr-tracing-otel-config-collector.adoc

@@ -14,6 +14,8 @@ Processors:: Optional. Processors run through the data between it is received an
 
 Exporters:: An exporter, which can be push or pull based, is how you send data to one or more back ends or destinations. By default, no exporters are configured. One or more exporters must be configured. Exporters can support one or more data sources. Exporters might be used with their default settings, but many exporters require configuration to specify at least the destination and security settings.
 
+Connectors:: A connector connects two pipelines: It consumes data as an exporter at the end of one pipeline and emits data as a receiver at the start of another pipeline. It can consume and emit data of the same or different data type. It can generate and emit data to summarize the consumed data, or it can merely replicate or route data.
+
 You can define multiple instances of components in a custom resource YAML file. When configured, these components must be enabled through pipelines defined in the `spec.config.service` section of the YAML file. As a best practice, only enable the components that you need.
 
 .Example of the OpenTelemetry Collector custom resource file
@@ -418,3 +420,30 @@ service:
 <4> The client password is configured as a client authenticator for the BasicAuth extension.
 <5> The authenticator configuration can be assigned to an OTLP receiver.
 <6> The authenticator configuration can be assigned to an OTLP exporter.
+
+[id="connectors_{context}"]
+=== Connectors
+
+[id="spanmetrics-connector_{context}"]
+==== Spanmetrics connector
+
+Aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from span data.
+
+* Support level: link:https://access.redhat.com/support/offerings/techpreview[Technology Preview]
+* Supported signals: traces
+
+.OpenTelemetry Collector custom resource with an enabled spanmetrics connector
+[source,yaml]
+----
+  config: |
+    connectors:
+      spanmetrics:
+        metrics_flush_interval: 15s <1>
+    service:
+      pipelines:
+        traces:
+          exporters: [spanmetrics]
+        metrics:
+          receivers: [spanmetrics]
+----
+<1>: Defines the flush interval of the generated metrics. Defaults to `15s`.

modules/distr-tracing-tempo-config-spanmetrics.adoc

@@ -0,0 +1,91 @@
+// Module included in the following assemblies:
+//
+// * distr_tracing_tempo/distr-tracing-tempo-configuring.adoc
+
+:_content-type: REFERENCE
+[id="distr-tracing-tempo-config-spanmetrics_{context}"]
+= Configuration of the monitor tab in Jaeger UI
+
+Trace data contains rich information, and the data is normalized across instrumented languages and frameworks.
+Therefore, additional metrics can be extracted from traces. These metrics are request count, duration, and error count (RED).
+The metrics can be visualized in Jaeger console in the *Monitor* tab.
+
+The metrics are derived from spans in the OpenTelemetry Collector that are scraped from the Collector by the Prometheus deployed in the user-workload monitoring stack.
+The Jaeger UI queries these metrics from the Prometheus endpoint and visualizes them.
+
+== OpenTelemetry Collector configuration
+
+The OpenTelemetry Collector requires configuration of the `spanmetrics` connector that derives metrics from traces and exports the metrics in the Prometheus format.
+
+.OpenTelemetry Collector custom resource for span RED
+[source,yaml]
+----
+kind: OpenTelemetryCollector
+apiVersion: opentelemetry.io/v1alpha1
+metadata:
+  name: otel
+spec:
+  mode: deployment
+  observability:
+    metrics:
+      enableMetrics: true <1>
+  config: |
+    connectors:
+      spanmetrics: <2>
+        metrics_flush_interval: 15s
+
+    receivers:
+      otlp: <3>
+        protocols:
+          grpc:
+          http:
+
+    exporters:
+      prometheus: <4>
+        endpoint: 0.0.0.0:8889
+        resource_to_telemetry_conversion:
+          enabled: true # by default resource attributes are dropped
+
+    service:
+      pipelines:
+        traces:
+          receivers: [otlp]
+          exporters: [otlp, spanmetrics] <5>
+        metrics:
+          receivers: [spanmetrics] <6>
+          exporters: [prometheus]
+----
+<1> Creates the `ServiceMonitor` custom resource to enable scraping of the Prometheus exporter.
+<2> The Spanmetrics connector receives traces and exports metrics.
+<3> The OTLP receiver to receive spans in the OpenTelemetry protocol.
+<4> The Prometheus exporter is used to export metrics in the Prometheus format.
+<5> The Spanmetrics connector is configured as exporter in traces pipeline.
+<6> The Spanmetrics connector is configured as receiver in metrics pipeline.
+
+== Tempo configuration
+
+The `TempoStack` custom resource must specify the following: the *Monitor* tab is enabled, and the Prometheus endpoint is set to the Thanos querier service to query the data from the user-defined monitoring stack.
+
+.TempoStack custom resource with the enabled Monitor tab
+[source,yaml]
+----
+kind:  TempoStack
+apiVersion: tempo.grafana.com/v1alpha1
+metadata:
+  name: simplest
+spec:
+  template:
+    gateway:
+      enabled: false <1>
+    queryFrontend:
+      jaegerQuery:
+        enabled: true
+        monitorTab:
+          enabled: true <2>
+          prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 <3>
+        ingress:
+          type: route
+----
+<1> Gateway must be enabled because the Monitor tab is not supported thorough the gateway deployment.
+<2> Enables the monitoring tab in the Jaeger console.
+<3> The service name for Thanos Querier from user-workload monitoring.