Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tail sampling fails to start when decision wait is lower than 1s #26354

Closed
jpkrohling opened this issue Aug 31, 2023 · 1 comment · Fixed by #26561
Closed

Tail sampling fails to start when decision wait is lower than 1s #26354

jpkrohling opened this issue Aug 31, 2023 · 1 comment · Fixed by #26561
Assignees
Labels
bug Something isn't working processor/tailsampling Tail sampling processor

Comments

@jpkrohling
Copy link
Member

Component(s)

processor/tailsampling

What happened?

When setting the decision_wait parameter to a value lower than 1s, the following error message is shown during the startup:

2023/08/31 09:28:53 collector server run finished with error: failed to build pipelines: failed to create "tail_sampling" processor, in pipeline "traces/export": invalid number of batches, it must be greater than zero

A workaround is to use a decision wait of at least 1s.

Collector version

v0.84.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:

processors:
  tail_sampling:
    decision_wait: 500ms
    num_traces: 50_000
    expected_new_traces_per_sec: 10_000
    policies:
      [
          {
            name: only-10-percent,
            type: probabilistic,
            probabilistic: {sampling_percentage: 10}
          },
      ]

exporters:
  logging:

connectors:

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - tail_sampling
      exporters:
        - logging

Log output

2023-08-31T09:31:48.873-0300    info    service/telemetry.go:84 Setting up own telemetry...
2023-08-31T09:31:48.874-0300    info    service/telemetry.go:201        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
2023-08-31T09:31:48.874-0300    info    exporter@v0.83.0/exporter.go:275        Development component. May change in the future.        {"kind": "exporter", "data_type": "traces", "name": "logging"}
Error: failed to build pipelines: failed to create "tail_sampling" processor, in pipeline "traces": invalid number of batches, it must be greater than zero
2023/08/31 09:31:48 collector server run finished with error: failed to build pipelines: failed to create "tail_sampling" processor, in pipeline "traces": invalid number of batches, it must be greater than zero

Additional context

No response

@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2023

Pinging code owners for processor/tailsampling: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

codeboten pushed a commit that referenced this issue Sep 14, 2023
This PR changes the tail-sampling processor to allow sub-second decision
wait times. Before this PR, the batcher size would be determined based
on the decision time, which would be 0 for sub-second values. This has
now a minimum value of 1.

Fixes #26354

---------

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working processor/tailsampling Tail sampling processor
Projects
None yet
2 participants