Skip to content

Commit

Permalink
Adds duration sampler distinct from latency in supplying two bounds (#…
Browse files Browse the repository at this point in the history
…26115)

**Description:** Adds a bounded duration sampling processor, distinct
from the existing latency one in that it has both lower and upper bounds

Apologies for this appearing as a pull request out of nothing, my intent
had actually been to create a review area against my own fork and raise
an issue asking if you'd accept the PR. I think the need here is pretty
obvious from the context, though I think it's easy to imagine preferring
this to be a change to the existing processor. I raised as a new one as
I thought it might make existing behavior cleaner to retain.

**Link to tracking Issue:** As above this is a bit of a premature PR
since I intended to raise as an issue, and thus there isn't one, but I
think it's easy enough to deal with here so leaving open for now and
have learned GitHub's ways for the future (I rarely use github).

**Testing:** New module so associated tests are added showing all
relevant behavior, and passing.

**Documentation:** Updated README and example config

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
garry-cairns and dependabot[bot] authored Nov 2, 2023
1 parent f509060 commit ae6d36b
Show file tree
Hide file tree
Showing 9 changed files with 138 additions and 14 deletions.
27 changes: 27 additions & 0 deletions .chloggen/bounded_duration_sampler.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: processor/tailsampling

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: adds optional upper bound duration for sampling

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [26115]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,4 @@ integration-coverage.html

go.work*

/result
4 changes: 2 additions & 2 deletions processor/tailsamplingprocessor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The following configuration options are required:

Multiple policies exist today and it is straight forward to add more. These include:
- `always_sample`: Sample all traces
- `latency`: Sample based on the duration of the trace. The duration is determined by looking at the earliest start time and latest end time, without taking into consideration what happened in between.
- `latency`: Sample based on the duration of the trace. The duration is determined by looking at the earliest start time and latest end time, without taking into consideration what happened in between. Supplying no upper bound will result in a policy sampling anything greater than `threshold_ms`.
- `numeric_attribute`: Sample based on number attributes (resource and record)
- `probabilistic`: Sample a percentage of traces. Read [a comparison with the Probabilistic Sampling Processor](#probabilistic-sampling-processor-compared-to-the-tail-sampling-processor-with-the-probabilistic-policy).
- `status_code`: Sample based upon the status code (`OK`, `ERROR` or `UNSET`)
Expand Down Expand Up @@ -77,7 +77,7 @@ processors:
{
name: test-policy-2,
type: latency,
latency: {threshold_ms: 5000}
latency: {threshold_ms: 5000, upper_threshold_ms: 10000}
},
{
name: test-policy-3,
Expand Down
2 changes: 1 addition & 1 deletion processor/tailsamplingprocessor/and_helper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ func TestAndHelper(t *testing.T) {
require.NoError(t, err)

expected := sampling.NewAnd(zap.NewNop(), []sampling.PolicyEvaluator{
sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 100),
sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 100, 0),
})
assert.Equal(t, expected, actual)
})
Expand Down
4 changes: 2 additions & 2 deletions processor/tailsamplingprocessor/composite_helper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,11 @@ func TestCompositeHelper(t *testing.T) {

expected := sampling.NewComposite(zap.NewNop(), 1000, []sampling.SubPolicyEvalParams{
{
Evaluator: sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 100),
Evaluator: sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 100, 0),
MaxSpansPerSecond: 250,
},
{
Evaluator: sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 200),
Evaluator: sampling.NewLatency(componenttest.NewNopTelemetrySettings(), 200, 0),
MaxSpansPerSecond: 500,
},
}, sampling.MonotonicClock{})
Expand Down
6 changes: 5 additions & 1 deletion processor/tailsamplingprocessor/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,13 +87,15 @@ type AndSubPolicyCfg struct {
sharedPolicyCfg `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct
}

// TraceStateCfg holds the common configuration for trace states.
type TraceStateCfg struct {
// Tag that the filter is going to be matching against.
Key string `mapstructure:"key"`
// Values indicate the set of values to use when matching against trace_state values.
Values []string `mapstructure:"values"`
}

// AndCfg holds the common configuration to all and policies.
type AndCfg struct {
SubPolicyCfg []AndSubPolicyCfg `mapstructure:"and_sub_policy"`
}
Expand Down Expand Up @@ -126,8 +128,10 @@ type PolicyCfg struct {
// LatencyCfg holds the configurable settings to create a latency filter sampling policy
// evaluator
type LatencyCfg struct {
// ThresholdMs in milliseconds.
// Lower bound in milliseconds. Retaining original name for compatibility
ThresholdMs int64 `mapstructure:"threshold_ms"`
// Upper bound in milliseconds.
UpperThresholdmsMs int64 `mapstructure:"upper_threshold_ms"`
}

// NumericAttributeCfg holds the configurable settings to create a numeric attribute filter
Expand Down
17 changes: 11 additions & 6 deletions processor/tailsamplingprocessor/internal/sampling/latency.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,19 @@ import (
)

type latency struct {
logger *zap.Logger
thresholdMs int64
logger *zap.Logger
thresholdMs int64
upperThresholdMs int64
}

var _ PolicyEvaluator = (*latency)(nil)

// NewLatency creates a policy evaluator sampling traces with a duration higher than a configured threshold
func NewLatency(settings component.TelemetrySettings, thresholdMs int64) PolicyEvaluator {
func NewLatency(settings component.TelemetrySettings, thresholdMs int64, upperThresholdMs int64) PolicyEvaluator {
return &latency{
logger: settings.Logger,
thresholdMs: thresholdMs,
logger: settings.Logger,
thresholdMs: thresholdMs,
upperThresholdMs: upperThresholdMs,
}
}

Expand All @@ -47,6 +49,9 @@ func (l *latency) Evaluate(_ context.Context, _ pcommon.TraceID, traceData *Trac
}

duration := maxTime.AsTime().Sub(minTime.AsTime())
return duration.Milliseconds() >= l.thresholdMs
if l.upperThresholdMs == 0 {
return duration.Milliseconds() >= l.thresholdMs
}
return (l.thresholdMs < duration.Milliseconds() && duration.Milliseconds() <= l.upperThresholdMs)
}), nil
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
)

func TestEvaluate_Latency(t *testing.T) {
filter := NewLatency(componenttest.NewNopTelemetrySettings(), 5000)
filter := NewLatency(componenttest.NewNopTelemetrySettings(), 5000, 0)

traceID := pcommon.TraceID([16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16})
now := time.Now()
Expand Down Expand Up @@ -71,6 +71,93 @@ func TestEvaluate_Latency(t *testing.T) {
}
}

func TestEvaluate_Bounded_Latency(t *testing.T) {
filter := NewLatency(componenttest.NewNopTelemetrySettings(), 5000, 10000)

traceID := pcommon.TraceID([16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16})
now := time.Now()

cases := []struct {
Desc string
Spans []spanWithTimeAndDuration
Decision Decision
}{
{
"trace duration shorter than lower bound",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 4500 * time.Millisecond,
},
},
NotSampled,
},
{
"trace duration is equal to lower bound",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 5000 * time.Millisecond,
},
},
NotSampled,
},
{
"trace duration is within lower and upper bounds",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 5001 * time.Millisecond,
},
},
Sampled,
},
{
"trace duration is above upper bound",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 10001 * time.Millisecond,
},
},
NotSampled,
},
{
"trace duration equals upper bound",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 10000 * time.Millisecond,
},
},
Sampled,
},
{
"total trace duration is longer than threshold but every single span is shorter",
[]spanWithTimeAndDuration{
{
StartTime: now,
Duration: 3000 * time.Millisecond,
},
{
StartTime: now.Add(2500 * time.Millisecond),
Duration: 3000 * time.Millisecond,
},
},
Sampled,
},
}

for _, c := range cases {
t.Run(c.Desc, func(t *testing.T) {
decision, err := filter.Evaluate(context.Background(), traceID, newTraceWithSpans(c.Spans))

assert.NoError(t, err)
assert.Equal(t, decision, c.Decision)
})
}
}

type spanWithTimeAndDuration struct {
StartTime time.Time
Duration time.Duration
Expand Down
2 changes: 1 addition & 1 deletion processor/tailsamplingprocessor/processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ func getSharedPolicyEvaluator(settings component.TelemetrySettings, cfg *sharedP
return sampling.NewAlwaysSample(settings), nil
case Latency:
lfCfg := cfg.LatencyCfg
return sampling.NewLatency(settings, lfCfg.ThresholdMs), nil
return sampling.NewLatency(settings, lfCfg.ThresholdMs, lfCfg.UpperThresholdmsMs), nil
case NumericAttribute:
nafCfg := cfg.NumericAttributeCfg
return sampling.NewNumericAttributeFilter(settings, nafCfg.Key, nafCfg.MinValue, nafCfg.MaxValue, nafCfg.InvertMatch), nil
Expand Down

0 comments on commit ae6d36b

Please sign in to comment.