Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve conversion of resource attributes to/from prometheus #2381

Merged
merged 10 commits into from
Mar 5, 2022
Merged

Improve conversion of resource attributes to/from prometheus #2381

Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
af0881b
update resource attributes used by prometheus, and round-trip other r…
dashpole Feb 24, 2022
0fb5fca
address feedbck
dashpole Feb 25, 2022
44ba0a8
address feedbck
dashpole Feb 25, 2022
063af17
map target_info to resource, and use service.instance.id for instance
dashpole Feb 25, 2022
631cd41
formatting
dashpole Feb 25, 2022
57e947a
be clear on metric family vs metric point
dashpole Feb 25, 2022
5ce1868
Update specification/metrics/datamodel.md
dashpole Feb 25, 2022
738ef83
change deliniator to /
dashpole Mar 1, 2022
e0c8708
fix formatting
dashpole Mar 2, 2022
d153406
Merge branch 'main' into prom_resource
bogdandrutu Mar 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 56 additions & 10 deletions specification/metrics/datamodel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1170,18 +1170,38 @@ Prometheus Cumulative metrics do not include the start time of the metric. When

#### Resource Attributes

When scraping a Prometheus endpoint, resource attributes MUST be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1).
When scraping a Prometheus endpoint, resource attributes MUST be added to the
scraped metrics to distinguish them from metrics from other Prometheus
endpoints. In particular, `service.name` and `service.instance.id`, are needed
to ensure Prometheus exporters can disambiguate metrics using
[`job` and `instance` labels](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances)
as [described below](#resource-attributes-1).

The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes:
The following attributes MUST be associated with scraped metrics as resource
attributes, and MUST NOT be added as metric attributes:

| OTLP Resource Attribute | Description |
| ----------------------- | ----------- |
| `service.name` | The configured name of the service that the target belongs to |
| `job` | Identical to `service.name` |
| `instance` | The <host>:<port> of the target's URL that was scraped. |
| `host.name` | `instance` The <host> portion of `instance` |
| `port` | `instance` The <port> portion of `instance` |
| `scheme` | `http` or `https` |
| `service.instance.id` | A unique identifier of the target. By default, it should be the <host>:<port> of the scraped URL |
dashpole marked this conversation as resolved.
Show resolved Hide resolved

The following attributes SHOULD be associated with scraped metrics as resource
attributes, and MUST NOT be added as metric attributes:

| OTLP Resource Attribute | Description |
| ----------------------- | ----------- |
| `net.host.name` | The <host> portion of the target's URL that was scraped |
dashpole marked this conversation as resolved.
Show resolved Hide resolved
| `net.host.port` | The <port> portion of the target's URL that was scraped |
dashpole marked this conversation as resolved.
Show resolved Hide resolved
| `http.scheme` | `http` or `https` |

In addition to the attributes above, the
["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
metric family is used to supply additional resource attributes. If present,
"target" info MUST be dropped from the batch of metrics, and all labels from
the "target" info metric family MUST be converted to resource attributes
attached to all other metrics which are part of the scrape. By default, label
keys and values MUST NOT be altered (such as replacing `_` with `.` characters
in keys).
bogdandrutu marked this conversation as resolved.
Show resolved Hide resolved

### OTLP Metric points to Prometheus

Expand Down Expand Up @@ -1240,9 +1260,35 @@ OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https:/

#### Resource Attributes

In SDK Prometheus (pull) exporters, all resource attributes MUST be dropped, and MUST NOT be attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping.

In the Collector's Prometheus pull and push (remote-write) exporters, it is possible for metrics from multiple targets to be sent together, so targets must be disambiguated from one another. However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector Prometheus exporters, the `job` and `instance` resource attributes MUST be converted to Prometheus metric labels, and other resource attributes SHOULD NOT be converted to metric labels.
In SDK Prometheus (pull) exporters, resource attributes SHOULD be converted to the ["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems) metric family; otherwise, they MUST be dropped, and MUST NOT be attached as labels to other metric families. The "target" info metric family MUST be an info-typed metric family whose labels MUST include the resource attributes, and MUST NOT include any other labels. There MUST be at most one "target" info metric family exposed on a Prometheus endpoint.

In the Collector's Prometheus pull and push (remote-write) exporters, it is
possible for metrics from multiple targets to be sent together, so targets must
be disambiguated from one another. However, the Prometheus exposition format
and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto)
formats do not include a notion of resource, and expect metric labels to
distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances)
labels distinguish targets and are expected to be present on metrics exposed on
a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/)
Prometheus endpoint) or pushed via Prometheus remote-write. In OTLP, the
`service.name`, `service.namespace`, and `service.instance.id` triplet is
[required to be unique](https://github.com/open-telemetry/opentelemetry-specification/blob/da74730bf835010229a9612c281f052e26553218/specification/resource/semantic_conventions/README.md#service),
which makes them good candidates to use to construct `job` and `instance`. In
the collector Prometheus exporters, the `service.name` and `service.namespace`
attributes MUST be combined as `<service.namespace>/<service.name>`, or
`<service.name>` if namespace is empty, to form the `job` metric label. The
`service.instance.id` attribute, if present, MUST be converted to the
`instance` label; otherwise, `instance` should be added with an empty value.
Other resource attributes SHOULD be converted to a
["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
metric family, or MUST be dropped. The "target" info metric family is an info-typed metric family
whose labels MUST include the resource attributes, and MUST NOT include any
other labels other than `job` and `instance`. There MUST be at most one
"target" info metric point exported for each unique combination of `job` and `instance`.

If info-typed metric families are not yet supported by the language Prometheus client library, a gauge-typed metric family named "target" info with a constant value of 1 MUST be used instead.

To convert OTLP resource attributes to Prometheus labels, string Attribute values are converted directly to labels, and non-string Attribute values MUST be converted to string attributes following the [attribute specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/common.md#attribute).

## Footnotes

Expand Down