Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add processed/exported Span metrics. #184

Closed
wants to merge 4 commits into from
Closed

Add processed/exported Span metrics. #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
60eee99
Add processed/exported Span metrics.
carlosalberto Jul 10, 2023
f65095c
Merge branch 'main' into add-span-dropped-metrics
carlosalberto Aug 24, 2023
8679683
Merge branch 'main' into add-span-dropped-metrics
carlosalberto Sep 6, 2023
3ce6cd1
Merge branch 'main' into add-span-dropped-metrics
carlosalberto Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/otel/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# OTel Components semantic conventions

**Status**: [Experimental][DocumentStatus]

This document defines semantic conventions for OTel components (such as processors, exporters, etc).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTel SDK components?


OTel Component semantic conventions are defined for the following metrics:

* [Export](export-metrics.md): For export level metrics.

[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.22.0/specification/document-status.md
64 changes: 64 additions & 0 deletions docs/otel/export-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<!--- Hugo front matter used to generate the website version of this page:
linkTitle: OTel Export
--->

# Semantic Conventions for OTel Export Metrics

**Status**: [Experimental][DocumentStatus]

This document describes instruments and attributes for OTel
Export level metrics. Consider the [general metric semantic
conventions](README.md#general-metric-semantic-conventions) when creating
instruments not explicitly defined in the specification.

<!-- Re-generate TOC with `markdown-toc --no-first-h1 -i` -->

<!-- toc -->

- [Metric Instruments](#metric-instruments)
* [Metric: `otel.processor.spans`](#metric-otelprocessorspans)
* [Metric: `otel.exporter.spans`](#metric-otelexporterspans)

<!-- tocstop -->

## Metric Instruments

### Metric: `otel.processor.spans`

This metric is [required][MetricRequired].

<!-- semconv metric.otel.processor.spans(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `otel.processor.spans` | Counter | `{span}` | Measures the number of processed Spans. |
<!-- endsemconv -->

<!-- semconv metric.otel.processor.spans(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `processor.dropped` | boolean | Whether the Span was dropped or not. [1] | | Required |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not included un-sampled spans, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, this doesn't include un-sampled Spans. Will clarify that in a comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for reference: https://github.com/jaegertracing/jaeger-client-go/blob/8d8e8fcfd04de42b8482476abac6a902fca47c18/metrics.go#L22-L106

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this missing the otel. prefix? So otel.processor.dropped instead of processor.dropped?

| `processor.type` | string | Type of processor being used. | `BatchSpanProcessor` | Recommended |

**[1]:** Spans may be dropped if the internal buffer is full.
<!-- endsemconv -->

### Metric: `otel.exporter.spans`

This metric is [required][MetricRequired].

<!-- semconv metric.otel.exporter.spans(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `otel.exporter.spans` | Counter | `{span}` | Measures the number of exported Spans. |
<!-- endsemconv -->

<!-- semconv metric.otel.exporter.spans(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `exporter.dropped` | boolean | Whether the Span was dropped or not. [1] | | Required |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 states from the exporter's view:

  1. delivered (the exporter has received the confirmation from the ingestion)
  2. dropped (the exporter decided to drop the data knowing that the ingestion didn't accept it)
  3. unknown (the exporter has sent the data, but it has no idea whether the data is accepted or not by the ingestion)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good one, yes. So I guess my initial approached took for granted this:

  • dropped=false: Delivered, with confirmation from the ingestion step.
  • dropped=true: Either consecutive attempts were exhausted OR it's not confirmed ingestion was successful (including error on the server side).

Think it would make sense to have 3 values instead of two here (as suggested by your comment)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(On a related note: I will follow up this PR, once it's merged, but some additional metrics, such as retries count at the export level, etc)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to understand how to use this counter and what "dropped' actually means.

Does dropped mean you never tried to export, or that you did try and gave up?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes. It means that either the spans were dropped for an irrecoverable error (e.g. 4xx), or the max retries count has been reached. You are looking into this clarification or are thinking about some other scenarios as well? (Clarify this as well in this PR in a bit)

| `exporter.type` | string | Type of exporter being used. | `OtlpGrpcSpanExporter` | Recommended |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this essentially the same as the instrumentation scope name?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would different instances of the same "exporter" get a different type, or the same type? E.g. if I have two OTLP grpc span exporters, would I be able to tell how many spans each was exporting? In the collector today, I believe we can tell the difference between two instances of the same component.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same type ;) I don't think we have a (clear) notion regarding differentiating two processors/exporters of the same type (at least in the SDK).


**[1]:** Spans may be dropped in case of failed ingestion, e.g. network problem or the exported endpoint being down.
<!-- endsemconv -->

[MetricRequired]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.22.0/specification/metrics/metric-requirement-level.md#required
38 changes: 38 additions & 0 deletions model/metrics/otel.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
groups:
- id: metric.otel.exporter.spans
type: metric
metric_name: otel.exporter.spans
brief: "Measures the number of exported Spans."
instrument: counter
unit: "{span}"
attributes:
- id: exporter.dropped
type: boolean
requirement_level: required
brief: "Whether the Span was dropped or not."
note: >
Spans may be dropped in case of failed ingestion, e.g. network problem
or the exported endpoint being down.
- id: exporter.type
type: string
requirement_level: recommended
brief: "Type of exporter being used."
examples: ["OtlpGrpcSpanExporter"]
- id: metric.otel.processor.spans
type: metric
metric_name: otel.processor.spans
brief: "Measures the number of processed Spans."
instrument: counter
unit: "{span}"
attributes:
- id: processor.dropped
type: boolean
requirement_level: required
brief: "Whether the Span was dropped or not."
note: >
Spans may be dropped if the internal buffer is full.
- id: processor.type
type: string
requirement_level: recommended
brief: "Type of processor being used."
examples: ["BatchSpanProcessor"]
Loading