-
Notifications
You must be signed in to change notification settings - Fork 2.4k
/
README.md
147 lines (113 loc) · 7.88 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
# Prometheus Remote Write Exporter
<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [beta]: metrics |
| Distributions | [core], [contrib], [aws], [liatrio], [observiq] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fprometheusremotewrite) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fprometheusremotewrite) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@Aneurysm9](https://www.github.com/Aneurysm9), [@rapphil](https://www.github.com/rapphil) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta
[core]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[aws]: https://github.com/aws-observability/aws-otel-collector
[liatrio]: https://github.com/liatrio/liatrio-otel-collector
[observiq]: https://github.com/observIQ/observiq-otel-collector
<!-- end autogenerated section -->
Prometheus Remote Write Exporter sends OpenTelemetry metrics
to Prometheus [remote write compatible
backends](https://prometheus.io/docs/operating/integrations/)
such as Cortex, Mimir, and Thanos.
By default, this exporter requires TLS and offers queued retry capabilities.
:warning: Non-cumulative monotonic, histogram, and summary OTLP metrics are
dropped by this exporter.
A [design doc](DESIGN.md) is available to document in detail
how this exporter works.
## Getting Started
The following settings are required:
- `endpoint` (no default): The remote write URL to send remote write samples.
By default, TLS is enabled and must be configured under `tls:`:
- `insecure` (default = `false`): whether to enable client transport security for
the exporter's connection.
As a result, the following parameters are also required under `tls:`:
- `cert_file` (no default): path to the TLS cert to use for TLS required connections. Should
only be used if `insecure` is set to false.
- `key_file` (no default): path to the TLS key to use for TLS required connections. Should
only be used if `insecure` is set to false.
The following settings can be optionally configured:
- `external_labels`: map of labels names and values to be attached to each metric data point
- `headers`: additional headers attached to each HTTP request.
- *Note the following headers cannot be changed: `Content-Encoding`, `Content-Type`, `X-Prometheus-Remote-Write-Version`, and `User-Agent`.*
- `namespace`: prefix attached to each exported metric name.
- `add_metric_suffixes`: If set to false, type and unit suffixes will not be added to metrics. Default: true.
- `send_metadata`: If set to true, prometheus metadata will be generated and sent. Default: false.
- `remote_write_queue`: fine tuning for queueing and sending of the outgoing remote writes.
- `enabled`: enable the sending queue (default: `true`)
- `queue_size`: number of OTLP metrics that can be queued. Ignored if `enabled` is `false` (default: `10000`)
- `num_consumers`: minimum number of workers to use to fan out the outgoing requests. (default: `5`)
- `resource_to_telemetry_conversion`
- `enabled` (default = false): If `enabled` is `true`, all the resource attributes will be converted to metric labels by default.
- `target_info`: customize `target_info` metric
- `enabled` (default = true): If `enabled` is `true`, a `target_info` metric will be generated for each resource metric (see https://github.com/open-telemetry/opentelemetry-specification/pull/2381).
- `export_created_metric`:
- `enabled` (default = false): If `enabled` is `true`, a `_created` metric is
exported for Summary, Histogram, and Monotonic Sum metric points if
`StartTimeUnixNano` is set.
- `max_batch_size_bytes` (default = `3000000` -> `~2.861 mb`): Maximum size of a batch of
samples to be sent to the remote write endpoint. If the batch size is larger
than this value, it will be split into multiple batches.
Example:
```yaml
exporters:
prometheusremotewrite:
endpoint: "https://my-cortex:7900/api/v1/push"
wal: # Enabling the Write-Ahead-Log for the exporter.
directory: ./prom_rw # The directory to store the WAL in
buffer_size: 100 # Optional count of elements to be read from the WAL before truncating; default of 300
truncate_frequency: 45s # Optional frequency for how often the WAL should be truncated. It is a time.ParseDuration; default of 1m
resource_to_telemetry_conversion:
enabled: true # Convert resource attributes to metric labels
```
Example:
```yaml
exporters:
prometheusremotewrite:
endpoint: "https://my-cortex:7900/api/v1/push"
external_labels:
label_name1: label_value1
label_name2: label_value2
```
## Advanced Configuration
Several helper files are leveraged to provide additional capabilities automatically:
- [HTTP settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md)
- [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
- [Retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md), note that the exporter doesn't support `sending_queue` but provides `remote_write_queue`.
## Metric names and labels normalization
OpenTelemetry metric names and attributes are normalized to be compliant with Prometheus naming rules. [Details on this normalization process are described in the Prometheus translator module](../../pkg/translator/prometheus/).
## Setting resource attributes as metric labels
By default, resource attributes are added to a special metric called `target_info`. To select and group by metrics by resource attributes, you [need to do join on `target_info`](https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches). For example, to select metrics with `k8s_namespace_name` attribute equal to `my-namespace`:
```promql
app_ads_ad_requests_total * on (job, instance) group_left target_info{k8s_namespace_name="my-namespace"}
```
Or to group by a particular attribute (for ex. `k8s_namespace_name`):
```promql
sum by (k8s_namespace_name) (app_ads_ad_requests_total * on (job, instance) group_left(k8s_namespace_name) target_info)
```
This is not a common pattern, and we recommend copying the most common resource attributes into metric labels. You can do this through the transform processor:
```yaml
processor:
transform:
metric_statements:
- context: datapoint
statements:
- set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
- set(attributes["container"], resource.attributes["k8s.container.name"])
- set(attributes["pod"], resource.attributes["k8s.pod.name"])
```
After this, grouping or selecting becomes as simple as:
```promql
app_ads_ad_requests_total{namespace="my-namespace"}
sum by (namespace) (app_ads_ad_requests_total)
```
[beta]:https://github.com/open-telemetry/opentelemetry-collector#beta
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[core]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol