Skip to content

[Backport 3.2] Fix errors in the pipeline code #10827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 26, 2025
Merged

[Backport 3.2] Fix errors in the pipeline code #10827

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions _data-prepper/common-use-cases/trace-analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@

There are three processors for the trace analytics feature:

* otel_traces_raw -- The *otel_traces_raw* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace-source/), and performs stateful processing, extraction, and completion of trace-group-related fields.
* otel_traces -- The *otel_traces* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace-source/), and performs stateful processing, extraction, and completion of trace-group-related fields.

Check failure on line 41 in _data-prepper/common-use-cases/trace-analytics.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _data-prepper/common-use-cases/trace-analytics.md#L41 

[Vale.Terms] Use 'OTel' instead of 'otel'.
Raw output 
{"message": "[Vale.Terms] Use 'OTel' instead of 'otel'.", "location": {"path": "_data-prepper/common-use-cases/trace-analytics.md", "range": {"start": {"line": 41, "column": 3}}}, "severity": "ERROR"}
* otel_traces_group -- The *otel_traces_group* processor fills in the missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/298e7931aa3b26130048ac3bde260e066857df54/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend.
* service_map_stateful -- The *service_map_stateful* processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards.
* service_map -- The *service_map* processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards.


### OpenSearch sink
Expand All @@ -49,8 +49,8 @@

The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics:

* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-traces/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map/) processor.
* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-traces/) processor.
* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map/) processor.

## Trace tuning

Expand All @@ -64,12 +64,12 @@

#### Buffer

The total number of trace requests processed by Data Prepper is equal to the sum of the `buffer_size` values in `otel-trace-pipeline` and `raw-pipeline`. The total number of trace requests sent to OpenSearch is equal to the product of `batch_size` and `workers` in `raw-trace-pipeline`. For more information about `raw-pipeline`, see [Trace analytics pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/pipelines).
The total number of trace requests processed by Data Prepper is equal to the sum of the `buffer_size` values in `otel-trace-pipeline` and `raw-trace-pipeline`. The total number of trace requests sent to OpenSearch is equal to the product of `batch_size` and `workers` in `raw-trace-pipeline`. For more information about `raw-trace-pipeline`, see [Trace analytics pipeline]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/pipelines).


We recommend the following when making changes to buffer settings:
* The `buffer_size` value in `otel-trace-pipeline` and `raw-pipeline` should be the same.
* The `buffer_size` should be greater than or equal to `workers` * `batch_size` in the `raw-pipeline`.
* The `buffer_size` value in `otel-trace-pipeline` and `raw-trace-pipeline` should be the same.
* The `buffer_size` should be greater than or equal to `workers` * `batch_size` in the `raw-trace-pipeline`.


#### Workers
Expand Down Expand Up @@ -114,13 +114,13 @@

The following example demonstrates how to build a pipeline that supports the [OpenSearch Dashboards Observability plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines serve two different purposes and write to different OpenSearch indexes. The first pipeline prepares trace data for OpenSearch and enriches and ingests the span documents into a span index within OpenSearch. The second pipeline aggregates traces into a service map and writes service map documents into a service map index within OpenSearch.

Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_traces_raw_prepper` processor. The `otel_traces_raw` processor replaces the `otel_traces_raw_prepper` processor and supports some of Data Prepper's recent data model changes. Instead, you should use the `otel_traces_raw` processor. See the following YAML file example:
Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_traces_prepper` processor. The `otel_traces` processor replaces the `otel_traces_prepper` processor and supports some of Data Prepper's recent data model changes. Instead, you should use the `otel_traces` processor. See the following YAML file example:

```yml
entry-pipeline:
delay: "100"
source:
otel_traces_source:
otel_trace_source:
ssl: false
buffer:
bounded_blocking:
Expand All @@ -131,7 +131,7 @@
name: "raw-trace-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
raw-trace-pipeline:
source:
pipeline:
name: "entry-pipeline"
Expand All @@ -140,7 +140,7 @@
buffer_size: 10240
batch_size: 160
processor:
- otel_traces_raw:
- otel_traces:
sink:
- opensearch:
hosts: ["https://localhost:9200"]
Expand All @@ -158,7 +158,7 @@
buffer_size: 10240
batch_size: 160
processor:
- service_map_stateful:
- service_map:
sink:
- opensearch:
hosts: ["https://localhost:9200"]
Expand All @@ -176,7 +176,7 @@

```yaml
source:
otel_traces_source:
otel_trace_source:
#record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0
ssl: true
sslKeyCertChainFile: "/full/path/to/certfile.crt"
Expand All @@ -202,7 +202,7 @@
# default value is 3_000 ms
delay: "100"
source:
otel_traces_source:
otel_trace_source:
#record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0
ssl: false # Change this to enable encryption in transit
authentication:
Expand All @@ -223,10 +223,10 @@
name: "raw-trace-pipeline"
- pipeline:
name: "entry-pipeline"
raw-pipeline:
raw-trace-pipeline:
# Configure same as the otel-trace-pipeline
workers: 8
# We recommend using the default value for the raw-pipeline.
# We recommend using the default value for the raw-trace-pipeline.
delay: "3000"
source:
pipeline:
Expand All @@ -242,7 +242,7 @@
# With 64 as batch size each worker thread could process upto 3200 spans (64 * 50)
batch_size: 64
processor:
- otel_traces_raw:
- otel_traces:
- otel_traces_group:
hosts: [ "https://localhost:9200" ]
# Change to your credentials
Expand Down Expand Up @@ -276,7 +276,7 @@
pipeline:
name: "entry-pipeline"
processor:
- service_map_stateful:
- service_map:
# The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships.
# The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes.
# Set higher value if your applications have higher latency.
Expand Down Expand Up @@ -369,9 +369,9 @@

Starting with Data Prepper version 1.4, trace processing uses Data Prepper's event model. This allows pipeline authors to configure other processors to modify spans or traces. To provide a migration path, Data Prepper version 1.4 introduced the following changes:

* `otel_traces_source` has an optional `record_type` parameter that can be set to `event`. When configured, it will output event objects.
* `otel_traces_raw` replaces `otel_traces_raw_prepper` for event-based spans.
* `otel_trace_source` has an optional `record_type` parameter that can be set to `event`. When configured, it will output event objects.
* `otel_traces` replaces `otel_traces_prepper` for event-based spans.
* `otel_traces_group` replaces `otel_traces_group_prepper` for event-based spans.

In Data Prepper version 2.0, `otel_traces_source` will only output events. Data Prepper version 2.0 also removes `otel_traces_raw_prepper` and `otel_traces_group_prepper` entirely. To migrate to Data Prepper version 2.0, you can configure your trace pipeline using the event model.
In Data Prepper version 2.0, `otel_trace_source` will only output events. Data Prepper version 2.0 also removes `otel_traces_prepper` and `otel_traces_group_prepper` entirely. To migrate to Data Prepper version 2.0, you can configure your trace pipeline using the event model.