Skip to content

Commit d03b033

Browse files
authored
Check context prior to delaying retry in OTLP exporters (#7678)
Fix #7673 [Issue being addressed](#7673 (comment)): > 1. [`fn`](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry_test.go#L163-L165) is [called](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L87) > 2. It [returns an error](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry_test.go#L165) > 3. The code [checks if the error is retryable](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L92), it [always is](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry_test.go#L149) > 4. [Time delay is checked](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L97-L108) > - [Max elsapsed time](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry_test.go#L156-L157) is 10 ms > - Initial [delay is 1ms](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry_test.go#L151) > - Delay is determined to be 1ms > - The program proceeds to waiting > 5. [Wait is called](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L110-L112) > 6. The [wait select statement is evaluated](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L127-L138) > - On slow systems both `case`s are true > - [Non-deterministically](https://go.dev/ref/spec#:~:text=If%20one%20or,communications%20can%20proceed.) the [timer channel `case`](https://github.com/open-telemetry/opentelemetry-go/blob/1bc9713ac6dc8cbe2fd04fd6dc716d316059eb90/exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go#L137) is selected > - The retry function is re-run and and second iteration is recorded causing the failure > - On fast systems only the context cancel is true > - The retry stops here with only `1` execution Do not rely on non-deterministic `select` statement to catch ended context prior to waiting for a retry delay. Explicitly check the context prior to entering the wait. This resolves the flaky test and ensure in normal operation that requests with canceled context are ended without having to wait for any additional delays.
1 parent 61765e7 commit d03b033

File tree

7 files changed

+35
-0
lines changed

7 files changed

+35
-0
lines changed

exporters/otlp/otlplog/otlploggrpc/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporters/otlp/otlplog/otlploghttp/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporters/otlp/otlpmetric/otlpmetricgrpc/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporters/otlp/otlpmetric/otlpmetrichttp/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporters/otlp/otlptrace/otlptracegrpc/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

exporters/otlp/otlptrace/otlptracehttp/internal/retry/retry.go

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

internal/shared/otlp/retry/retry.go.tmpl

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,11 @@ func (c Config) RequestFunc(evaluate EvaluateFunc) RequestFunc {
9494
return err
9595
}
9696

97+
// Check if context is canceled before attempting to wait and retry.
98+
if ctx.Err() != nil {
99+
return fmt.Errorf("%w: %w", ctx.Err(), err)
100+
}
101+
97102
if maxElapsedTime != 0 && time.Since(startTime) > maxElapsedTime {
98103
return fmt.Errorf("max retry time elapsed: %w", err)
99104
}

0 commit comments

Comments
 (0)