Transform Processor

Status
Stability	alpha
Supported pipeline types	traces, metrics, logs
Distributions	contrib
Warnings	Unsound Transformations, Identity Conflict, Orphaned Telemetry, Other

The transform processor modifies telemetry based on configuration using the OpenTelemetry Transformation Language.

For each signal type, the processor takes a list of statements associated to a Context type and executes the statements against the incoming telemetry in the order specified in the config. Each statement can access and transform telemetry using functions and allow the use of a condition to help decide whether the function should be executed.

Config

The transform processor allows configuring multiple context statements for traces, metrics, and logs. The value of context specifies which OTTL Context to use when interpreting the associated statements. The statement strings, which must be OTTL compatible, will be passed to the OTTL and interpreted using the associated context. Each context will be processed in the order specified and each statement for a context will be executed in the order specified.

The transform processor also allows configuring an optional field, error_mode, which will determine how the processor reacts to errors that occur while processing a statement.

error_mode	description
ignore	The processor ignores errors returned by statements and continues on to the next statement. This is the recommended mode.
propagate	The processor returns the error up the pipeline. This will result in the payload being dropped from the collector.

If not specified, propagate will be used.

transform:
  error_mode: ignore
  <trace|metric|log>_statements:
    - context: string
      statements:
        - string
        - string
        - string
    - context: string
      statements:
        - string
        - string
        - string

Proper use of contexts will provide increased performance and capabilities. See Contexts for more details.

Valid values for context are:

Signal	Context Values
trace_statements	resource, scope, span, and spanevent
metric_statements	resource, scope, metric, and datapoint
log_statements	resource, scope, and log

Example

The example takes advantage of context efficiency by grouping transformations with the context which it intends to transform. See Contexts for more details.

Example configuration:

transform:
  error_mode: ignore
  trace_statements:
    - context: resource
      statements:
        - keep_keys(attributes, ["service.name", "service.namespace", "cloud.region", "process.command_line"])
        - replace_pattern(attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***")
        - limit(attributes, 100, [])
        - truncate_all(attributes, 4096)
    - context: span
      statements:
        - set(status.code, 1) where attributes["http.path"] == "/health"
        - set(name, attributes["http.route"])
        - replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}")
        - limit(attributes, 100, [])
        - truncate_all(attributes, 4096)

  metric_statements:
    - context: resource
      statements:
      - keep_keys(attributes, ["host.name"])
      - truncate_all(attributes, 4096)
    - context: metric
      statements:
        - set(description, "Sum") where type == "Sum"
    - context: datapoint
      statements:
        - limit(attributes, 100, ["host.name"])
        - truncate_all(attributes, 4096)
        - convert_sum_to_gauge() where metric.name == "system.processes.count"
        - convert_gauge_to_sum("cumulative", false) where metric.name == "prometheus_metric"
        
  log_statements:
    - context: resource
      statements:
        - keep_keys(resource.attributes, ["service.name", "service.namespace", "cloud.region"])
    - context: log
      statements:
        - set(severity_text, "FAIL") where body == "request failed"
        - replace_all_matches(attributes, "/user/*/list/*", "/user/{userId}/list/{listId}")
        - replace_all_patterns(attributes, "/account/\\d{4}", "/account/{accountId}")
        - set(body, attributes["http.route"])

Grammar

You can learn more in-depth details on the capabilities and limitations of the OpenTelemetry Transformation Language used by the transform processor by reading about its grammar.

Contexts

The transform processor utilizes the OTTL's contexts to transform Resource, Scope, Span, SpanEvent, Metric, DataPoint, and Log telemetry. The contexts allow the OTTL to interact with the underlying telemetry data in its pdata form.

• Resource Context
• Scope Context
• Span Context
• SpanEvent Context
• Metric Context
• DataPoint Context
• Log Context

Each context allows transformation of its type of telemetry.
For example, statements associated to a resource context will be able to transform the resource's attributes and dropped_attributes_count.

Contexts NEVER supply access to individual items "lower" in the protobuf definition.

• This means statements associated to a resource WILL NOT be able to access the underlying instrumentation scopes.
• This means statements associated to a scope WILL NOT be able to access the underlying telemetry slices (spans, metrics, or logs).
• Similarly, statements associated to a metric WILL NOT be able to access individual datapoints, but can access the entire datapoints slice.
• Similarly, statements associated to a span WILL NOT be able to access individual SpanEvents, but can access the entire SpanEvents slice.

For practical purposes, this means that a context cannot make decisions on its telemetry based on telemetry "lower" in the structure. For example, the following context statement is not possible because it attempts to use individual datapoint attributes in the condition of a statements that is associated to a metric

metric_statements:
- context: metric
  statements:
  - set(description, "test passed") where datapoints.attributes["test"] == "pass"

Context ALWAYS supply access to the items "higher" in the protobuf definition that are associated to the telemetry being transformed.

• This means that statements associated to a datapoint have access to a datapoint's metric, instrumentation scope, and resource.
• This means that statements associated to a spanevent have access to a spanevent's span, instrumentation scope, and resource.
• This means that statements associated to a span/metric/log have access to the telemetry's instrumentation scope, and resource.
• This means that statements associated to a scope have access to the scope's resource.

For example, the following context statement is possible because datapoint statements can access the datapoint's metric.

metric_statements:
- context: datapoint
  statements:
    - set(metric.description, "test passed") where attributes["test"] == "pass"

Whenever possible, associate your statements to the context that the statement intend to transform. Although you can modify resource attributes associated to a span using the span context, it is more efficient to use the resource context. This is because contexts are nested: the efficiency comes because higher-level contexts can avoid iterating through any of the contexts at a lower level.

Supported functions:

Since the transform processor utilizes the OTTL's contexts for Traces, Metrics, and Logs, it is able to utilize functions that expect pdata in addition to any common functions. These common functions can be used for any signal.

• OTTL Functions

In addition to OTTL functions, the processor defines its own functions to help with transformations specific to this processor:

Metrics only functions

• convert_sum_to_gauge
• convert_gauge_to_sum
• convert_summary_count_val_to_sum
• convert_summary_sum_val_to_sum

convert_sum_to_gauge

convert_sum_to_gauge()

Converts incoming metrics of type "Sum" to type "Gauge", retaining the metric's datapoints. Noop for metrics that are not of type "Sum".

NOTE: This function may cause a metric to break semantics for Gauge metrics. Use at your own risk.

Examples:

• convert_sum_to_gauge()

convert_gauge_to_sum

convert_gauge_to_sum(aggregation_temporality, is_monotonic)

Converts incoming metrics of type "Gauge" to type "Sum", retaining the metric's datapoints and setting its aggregation temporality and monotonicity accordingly. Noop for metrics that are not of type "Gauge".

aggregation_temporality is a string ("cumulative" or "delta") that specifies the resultant metric's aggregation temporality. is_monotonic is a boolean that specifies the resultant metric's monotonicity.

NOTE: This function may cause a metric to break semantics for Sum metrics. Use at your own risk.

Examples:

• convert_gauge_to_sum("cumulative", false)

• convert_gauge_to_sum("delta", true)

convert_summary_count_val_to_sum

convert_summary_count_val_to_sum(aggregation_temporality, is_monotonic)

The convert_summary_count_val_to_sum function creates a new Sum metric from a Summary's count value.

aggregation_temporality is a string ("cumulative" or "delta") representing the desired aggregation temporality of the new metric. is_monotonic is a boolean representing the monotonicity of the new metric.

The name for the new metric will be <summary metric name>_count. The fields that are copied are: timestamp, starttimestamp, attibutes, and description. The new metric that is created will be passed to all functions in the metrics statements list. Function conditions will apply.

NOTE: This function may cause a metric to break semantics for Sum metrics. Use at your own risk.

Examples:

• convert_summary_count_val_to_sum("delta", true)

• convert_summary_count_val_to_sum("cumulative", false)

convert_summary_sum_val_to_sum

convert_summary_sum_val_to_sum(aggregation_temporality, is_monotonic)

The convert_summary_sum_val_to_sum function creates a new Sum metric from a Summary's sum value.

aggregation_temporality is a string ("cumulative" or "delta") representing the desired aggregation temporality of the new metric. is_monotonic is a boolean representing the monotonicity of the new metric.

The name for the new metric will be <summary metric name>_sum. The fields that are copied are: timestamp, starttimestamp, attibutes, and description. The new metric that is created will be passed to all functions in the metrics statements list. Function conditions will apply.

NOTE: This function may cause a metric to break semantics for Sum metrics. Use at your own risk.

Examples:

• convert_summary_sum_val_to_sum("delta", true)

• convert_summary_sum_val_to_sum("cumulative", false)

Contributing

See CONTRIBUTING.md.

Warnings

The transform processor's implementation of the [OpenTelemetry Transformation Language]https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/processing.md#opentelemetry-transformation-language) (OTTL) allows users to modify all aspects of their telemetry. Some specific risks are listed below, but this is not an exhaustive list. In general, understand your data before using the transform processor.

• Unsound Transformations: Several Metric-only functions allow you to transform one metric data type to another or create new metrics from an existing metrics. Transformations between metric data types are not defined in the metrics data model. These functions have the expectation that you understand the incoming data and know that it can be meaningfully converted to a new metric data type or can meaningfully be used to create new metrics.
  • Although the OTTL allows the set function to be used with metric.data_type, its implementation in the transform processor is NOOP. To modify a data type you must use a function specific to that purpose.
• Identity Conflict: Transformation of metrics have the potential to affect the identity of a metric leading to an Identity Crisis. Be especially cautious when transforming metric name and when reducing/changing existing attributes. Adding new attributes is safe.
• Orphaned Telemetry: The processor allows you to modify span_id, trace_id, and parent_span_id for traces and span_id, and trace_id logs. Modifying these fields could lead to orphaned spans or logs.