WIP: Pipeline monitoring metrics

text/metrics/0238-pipeline-monitoring.md

@@ -0,0 +1,364 @@
+# OpenTelemetry Telemetry Pipeline metrics
+
+Propose a uniform standard for telemetry pipeline metrics generated by
+OpenTelemetry SDKs and Collectors.
+
+WIP: this is a work-in-progress draft, derived from OTEP 238 which
+was closed pending further development.
+
+Tl;DR: this proposes TWO metric instrument semantics, three overall
+considering SDKs and collectors:
+
+- `otelcol_consumed_items`: Received and inserted data items (Collector)
+- `otelcol_produced_items`: Exported, dropped, and discarded items (Collector)
+- `otelsdk_produced_items`: Exported, dropped, and discarded items (SDK)
+
+## Motivation
+
+OpenTelemetry desires to standardize conventions for the metrics
+emitted by SDKs about success and failure of telemetry reporting. At
+the same time, the OpenTelemetry Collector has existing conventions
+which are expected to connect with metrics emitted by SDKs and have
+similar definitions.
+
+## Explanation
+
+We use the term "pipeline" to describe an arrangement of system
+components which produce, consume, and process telemetry on its way
+from the point of origin to the endpoint(s) in its journey.  Pipeline
+components included in this specification are:
+
+- OpenTelemetry SDKs: As telemetry producers, these components are the
+  start of a pipeline.
+- OpenTelemetry Collectors: The OpenTelemetry collector contains an
+  arrangement of components which act as both consumers and producers.
+
+The terms "following"/"follower" and "preceding"/"preceder" refer to
+the relative components after or before another component in a
+pipeline.  The preceding component ("preceder") produces data that is
+consumed by the following component ("follower").
+
+An arrangement of pipeline components acting as a single unit, such as
+implemented by the OpenTelemetry Collector, is called a segment.  Each
+segment consists of a receiver, zero or more processors, and an
+exporter.  The terms "following" and "preceding" apply to pipeline
+segments with the same meaning as for components.  For example, a
+agent pipeline segment forwards to a gateway pipeline.
+
+### Detailed design
+
+#### Producer and Consumer instruments
+
+We choose to specify two metric instruments for use counting outcomes,
+one instrument to account for producer outcomes and one to account for
+consumer outcomes.  In an ideal pipeline, a conservation rule exists
+between what goes in (i.e., is consumed) and what goes out (i.e., is
+produced).  The use of producer and consumer metric instruments is
+designed to enable this form of consistency check.  When the pipeline
+is properly functioning and instrumented, we expect the sum of
+producer and consumer across all outcomes to be equal.
+
+The alternative, which uses one metric instrument per producer outcome
+and one metric instrument per consumer outcome, has known
+difficulties.  To define a ratio between any one outcome and the total
+requires a metric formula defined by all the outcomes.  On other hand,
+it is common practice using OpenTelemetry metrics to aggrgate by
+attribute.  It possible and convenient, when a single metric
+instrument is used per outcome, to define ratios and build area charts
+from a single metric instruments.
+
+The use of exclusive counters, one per outcome, is also logically
+confusing.  Existing OpenTelemetry collector metrics for exporters
+have both `sent` and `send_failed` metrics.  However, `sent` only
+counts success outcomes. A user could easily believe that the failure
+ratio is defined as `send_failed / sent`, since (logically) something
+has to be sent before the send can fail.  The correct failure ratio,
+using exclusive counters, is `send_failed / (sent + send_failed)`, but
+from experience, users can easily miss this detail.  Moreover, when
+exclusive counters have been defined in this manner, it is impossible
+to define new outcomes, as every formula would need to be updated.
+
+#### Processors count as producers and consumers
+
+As specified, processors are responsible for counting items only when
+the number changes while passing through a processor.  Processors are
+responsible for counting producer outcomes when they remove an item of
+telemetry from a request, including:
+
+- `discarded` outcomes, which do not pass the data and return success
+- `dropped` outcomes, which do not pass the data and return failure.
+- `deferred:dropped` outcomes, which (eventually) do not pass the data
+  but (immediately) return success.
+
+Data that passes through a processor component, otherwise, should not
+be counted as produced because following components are responsible
+for counting the outcome.  Considering the combined producer outcomes
+for a pipeline segment, the total will include `discarded` and
+`dropped`, subtotals from processors combined with all potential
+outcomes from the exporter component.
+
+Data that is inserted by a processor component, meaning new items of
+telemetry that were not consumed from the previous component in the
+pipeline, should be counted as consumer outcomes by the component that
+inserted them, since the preceding component does not know about them.
+Processors that insert items will:
+
+1. Wait until the next component returns from the Consume() operation.
+2. Count a consumer outcome according to the return value for the
+   number of points inserted.
+
+By these rules, a processor that inserts data and then immediately
+drops or discards the same data will raise the count equally for both
+both consumer and producer metrics.
+
+#### Distinct prefixes for SDKs and Collectors
+
+There is a potential to use the same metric names to describe SDK
+producers and Collector producers.  However, we find two reasons this
+unification should be avoided.
+
+First, we seek to avoid aggregations combining first-class SDKs
+producers, SDK consumer components (i.e., bridges) and Collector
+producers, which do not have corresponding consumer metrics, and
+Collector producers.
+
+Second (really a specific case of the first), we seek to avoid
+aggregations that combine OpenTelemetry Collector pipeline metrics
+with SDK pipeline metrics in the same process, when the OpenTelemetry
+SDK instruments the OpenTelemetry Collector.
+
+#### Pipeline equations
+
+The behavior of a pipeline section consisting of one or more elements
+can be reduced to distinct operation categories.  In all cases (i.e.,
+for both producer and consumer metric instruments), the item of
+telemetry has a definite outcome determined by a single component,
+with a list of outcomes specified below.  Each item also has a
+definite success or failure boolean property.
+
+The consumer categories, leading to the first pipeline segment equation:
+
+- **Received**: An item of telemetry was exported from a preceding pipeline segment
+- **Inserted**: An item of telemetry was inserted by this pipeline segment
+
+The first equation:
+
+```
+Consumed(Segment) == Recieved(Segment) + Inserted(Segment)
+```
+
+The producer categories, leading to the second pipeline segment equation:
+
+- **Exported**: An attempt was made to export the telemetry to a following pipeline segment
+- **Discarded**: Considered success, an item of telemetry was eliminated (i.e., export never attempted)
+- **Dropped**: Considered failure, an item of telemetry was eliminated (i.e., export never attempted)
+
+The second equation:
+
+```
+Produced(Segment) == Discarded(Segment) + Dropped(Segment) + Exported(Segment)
+```
+
+The third equation states that the sum of all items-consumed outcomes
+for a pipeline segment equals the sum of all items-produced outcomes
+for that segment.
+
+```
+Consumed(Segment) == Produced(Segment)
+```
+
+These invariants are an idealization, because the consumer and
+producer operations happen independently, without ordering
+requirements.  Nevertheless, after a pipeline has been drained and the
+individual components shut down, we expect the producer and consumer
+instrument values to match exactly.
+
+#### Producer and consumer outcomes are asymmetric
+
+There are a several of reasons why the producer and consumer outcomes
+counted by pipeline monitoring will reflect contradictory information.
+
+For example, when timeouts are configured independently, as for
+example when a preceding segment's timeout is smaller than a following
+stage's timeout.  The preceding exporter's timeout, if less than the
+following exporter's timeout may cause consumer `timeout` outcomes
+without corresponding producer `timeout` outcomes.
+
+#### Outcomes may be deferred
+
+In some configurations, Collector pipeline segments have asynchronous
+elements, in an arrangement where the `Consume()` operation called by
+the preceder on the follower returns success, and responsibility for
+delivery transfers to the follower.  When deferred outcomes are in
+use, we consumer metrics will generally indicate 100% `accepted`
+outcomes.
+
+For these cases, exporters are expected to use the `deferred:` outcome
+categories to signal to monitoring systems especially the failure 
+outcomes that were not seen by producers.
+
+#### WIP
+
+Considering an Exporter and Receiver pair connecting two OpenTelemetry
+Collector pipeline segments, we expect:
+
+```
+Exported(Preceder) == Received(Follower)
+```
+
+Dropped and discarded are special
+
+#### Resource-exhausted is special
+
+Be specific about this one, it impacts SLOs.  Dot apply to "this"
+component.
+
+### WIP
+
+The specified counter names are:
+
+- `otelcol_consumed_items`
+- `otelcol_produced_items`
+- `otelsdk_produced_items`
+
+
+### Recommended conventional attributes
+
+- `otel.success` (boolean): This is true or false depending on whether the
+  component considers the outcome a success or a failure.
+- `otel.outcome` (string): This describes the outcome in a more specific
+  way than `otel.success`, with recommended values specified below.
+- `otel.signal` (string): This is the name of the signal (e.g., "logs",
+  "metrics", "traces")
+- `otel.component` (string): Name of the component in a pipeline.
+- `otel.pipeline` (string): Name of the pipeline in a collector.
+
+### Specified `otel.outcome` attribute values
+
+The `otel.outcome` attribute indicates extra information about a
+success or failure.  A set of standard conventional attribute values
+is supplied and is considered a closed set.  If these outcomes do not
+accurately explain the reason for a success or failure outcome, they
+SHOULD be extended by OpenTelemetry.
+
+For success=true:
+
+- `accepted`: Indicates a normal, synchronous request success case.
+  The item was consumed by the next stage of the pipeline, which
+  returned success.  Note the item could have been deferred by a
+  subsequent component, but as far as this component knows, the 
+  request successful.
+- `discarded`: Indicates a successful outcome in which the next stage
+  of the pipeline does not handle the event, as by a sampling
+  processor.
+- `deferred:<failure outcome>`: Deferred cases are where the
+  caller receives a success response and the true outcome is failure,
+  but this is not known until later.  The item is counted as
+  `deferred:` combined with the failure outcome that would otherwise
+  have been counted.
+
+For success=false, transient and potentially retryable cases:
+
+- `dropped`: The component introduced an original failure and did not
+  send to the next stage in the pipeline.
+- `timeout`: The item was in the process of being sent but the request
+  timed out, or its deadline was exceeded.  In this case, it
+  undetermined whether the consuming pipeline saw the item or not.
+- `exhausted`: The item was handled by the next stage of the pipeline,
+  which returned an error code indicating that it was overloaded.  If
+  the resource being exhausted is local and the item was not handled
+  by the next stage of the pipeline, record the item `dropped` and
+  return a resource-exhausted status code to the producer, who will
+  record a `exhausted` outcome.
+- `retryable`: The item was handled by the next stage of the pipeline,
+  which returned a retryable error status not covered by any of the
+  above values.
+
+For success=false, permanent cases:
+
+- `rejected`: The item was handled by the next stage of the pipeline,
+  which returned a permanent error status or partial success status
+  indicating that some items could not be accepted.
+- `unknown`: May be used when the component is suppressing errors and
+  not actually counting successes and failures.  As a special case,
+  the outcome `deferred:unknown` indicates that a success response 
+  was given and no information about the actual outcome is available.
+
+##
+
+
+
+#### Success, Outcome matrix
+
+| Outcome            | Export Attempted? | Caller Success? | Metrics Success? | Meaning                                                       |
+|--------------------|-------------------|-----------------|------------------|---------------------------------------------------------------|
+| accepted           | true              | true            | true             | Data (successfully) sent                                      |
+| discarded          | false             | true            | true             | Data (successfully) discarded                                 |
+| dropped            | false             | false           | false            | Request never started, error returned                         |
+| timeout            | true              | false           | false            | Request started, timed out, error returned                    |
+| exhausted          | true              | false           | false            | Request started, insufficient resources, error returned       |
+| retryable          | true              | false           | false            | Request started, retryable error status, error returned       |
+| rejected           | true              | false           | false            | Request completed, permanent error status, error returned     |
+| deferred:dropped   | false             | true            | false            | Request never started, error NOT returned                     |
+| deferred:timeout   | true              | true            | false            | Request started, timed out, error NOT returned                |
+| deferred:exhausted | true              | true            | false            | Request started, insufficient resources, error NOT returned   |
+| deferred:retryable | true              | true            | false            | Request started, retryable error status, error NOT returned   |
+| deferred:rejected  | true              | true            | false            | Request completed, permanent error status, error NOT returned |
+| deferred:unknown   | true              | true            | false            | Request has unknown outcome, error NOT returned               |
+
+#### Examples of each outcome
+
+##### Success, Accepted
+
+This is the common success case.  The item(s) were sent to the next
+stage in the pipeline while blocking the producer.
+
+##### Success, Dropped
+
+A processor was configured with instructions not to pass certain data.
+
+##### Success, Deferred-Accepted
+
+A component returned success to its producer, and later the outcome
+was successful.
+
+##### Failure, Dropped and Success, Deferred-Dropped
+
+(If deferred: A component returned success to its producer, then ...)
+
+The component never sent the item(s) due to limits in effect.  For
+example, shutdown was ordered and the queue could not be drained in
+time due to a limit on parallelism.
+
+##### Failure, Deadline exceeded and Success, Deferred-Deadline exceeded
+
+(If deferred: A component returned success to its producer, then ...)
+
+The component attempted sending the item(s), but the item(s) did not
+succeed before the deadline expired.  If there were attempts to retry,
+this is outcome of the final attempt.
+
+##### Failure, Resource exhausted and Success, Deferred-Resource exhausted
+
+(If deferred: A component returned success to its producer, then ...)
+
+The component attempted sending the item(s), but the consumer
+indicated its (or its consumers') resources were exceeded.  If there
+were attempts to retry, this is outcome of the final attempt.
+
+##### Failure, Retryable and Success, Deferred-Retryable
+
+(If deferred: A component returned success to its producer, then ...)
+
+A component returned success to its producer, and then it attempted
+sending the item(s), but the consumer indicated some kind of transient
+condition other than deadline- or resource-related (e.g., connection
+not accepted).  If there were attempts to retry, this is outcome of
+the final attempt.
+
+##### Failure, Rejected and Success, Deferred-Rejected
+
+(If deferred: A component returned success to its producer, then ...)
+
+A compmnent returned success to its producer, and then it attempted
+sending the item(s), but the consumer returned a permanent error.