[Bug]: Sampling strategies not working as expected

[Sucharitha95]

What happened?

Configured the Sampling strategies as per the Jaeger documentation: https://www.jaegertracing.io/docs/1.34/sampling/

Sampling strategy does not seem to affect even after configuring the strategy using .json file. I can also see in the collector logs that the input json file is read but the output is not obtained as expected.
Following is the configuration .json file used by us by default:
{
"service_strategies": [],
"default_strategy": {
"type": "probabilistic",
"param": 0.1
}
}

Even after providing the sampling configuration as above keeping the type as probabilistic and parameter as 0.1.
When 100 spans are generated all of those spans are seen on UI and no filtration of spans is done.
Tested with the parameter as 0, even then I could see all the spans are stored and seen on UI.

Steps to reproduce

1. Select a Sampling strategy and Parameter[As above case Probablistic and 0.1]
2. Configure the sampling_strategies.json file using the method selected.
3. Check if the spans are sampled as per the strategy provided.

Expected behavior

Sampling has to be performed as per the input configuration file.

Relevant log output

Log that Samplingstrategies file is being read:
{"level":"info","ts":1663994011.7913685,"caller":"static/strategy_store.go:138","msg":"Loading sampling strategies","filename":"/etc/sampling/samplingstrategies.json"}

Jaeger backend version

v1.34.0

Storage backend

Issue is seen with both Cassandra and Elastic search as backends.

Operating system

Ubuntu 18.04.6

[yurishkuro]

Expected behavior
Sampling has to be performed as per the input configuration file.

How are you verifying that?

[Sucharitha95]

Hi @yurishkuro ,
We have checked it in the Query UI. When 1000 traces are generated all of them can be queried out from the Query and all the 1000 traces were seen on Query UI.
We are also verifying it in the Collector logs where the following is seen for all the 1000 traces:
{"level":"debug","ts":1664348103.0998297,"caller":"app/span_processor.go:162","msg":"Span written to the storage by the collector","trace-id":"0dc7d9051509e0c0","span-id":"0dc7d9051509e0c0"}
{"level":"debug","ts":1664348103.09983,"caller":"app/span_processor.go:162","msg":"Span written to the storage by the collector","trace-id":"3ce311d349f53fb2","span-id":"3ce311d349f53fb2"}
{"level":"debug","ts":1664348103.099846,"caller":"app/span_processor.go:162","msg":"Span written to the storage by the collector","trace-id":"70958f6754f35127","span-id":"70958f6754f35127"}

[yurishkuro]

Jaeger backend does not perform sampling, sampling happens in the SDK. Which SDK are you using? Which sampler is the SDK configured to use?

[Sucharitha95]

Hi @yurishkuro ,
Thanks for your reply.
We are using Go-SDK and we are configuring the sampling in SDK as per the below link: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/trace
The sampler we are using from the SDK is AlwaysSample() in our client side configuration[ https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/main.go ]

In the collector we are configuring file sampling as decribed in the below link :https://www.jaegertracing.io/docs/1.36/sampling/#file-sampling
However, Sampling is not considered by the collector which is provided using strategies.json file.
Can you please let us know what else need to be configured, for the collector to consider the .json file sampling.
Also, As you mentioned that Sampling happens on sdk then what can be the exact use of sampling_strategies.json file which is provided as input in Collector configuration.Could you please let us know the complete flow of work for sampling from Client side to the Collector.

[Sucharitha95]

Hi @yurishkuro,
Can you please help us with this issue that is mentioned in above comment.

[yurishkuro]

AlwaysSample means it will sample all requests unconditionally. It's no surprise that the sampling configuration in Jaeger would have no effect on that. Backend sampling configuration only works with so-called "remote" samplers. All Jaeger SDKs supported those, but not all OTEL SDKs do, even though I believe it is called out as a requirement in the OTEL spec (the jaeger_remote sampler type). Looking at OTEL Go SDK, I don't think it supports Jaeger remote sampler out of the box, but there is a contrib module that does: open-telemetry/opentelemetry-go-contrib#936

[Sucharitha95]

Hi @yurishkuro ,
We have configured our trace generator using GO SDK remote sampling configuration similar to way mentioned in https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/samplers/jaegerremote.

jaegerRemoteSampler := jaegerremote.New(
"your-service-name",
jaegerremote.WithSamplingServerURL("http://{sampling_service_host_name}:5778"),
jaegerremote.WithSamplingRefreshInterval(10time.Second),
jaegerremote.WithInitialSampler(trace.TraceIDRatioBased(0.5)),
)
tp := trace.NewTracerProvider(
trace.WithSampler(jaegerRemoteSampler),
...
)
otel.SetTracerProvider(tp)
ticker := time.Tick(time.Second)
for {
<-ticker
fmt.Printf("\n Jaeger Remote Sampler %v\n\n", time.Now())
spewCfg := spew.ConfigState{
Indent: " ",
DisablePointerAddresses: true,
}
spewCfg.Dump(jaegerRemoteSampler)
}
}

After configuring the SDK using the above configuration,samplingstrategies.json file configuration is successfully fetched.
But when we tried generating the traces, the sampling_strategies.json file of collector is still not applied and did not show any effect on the sampling of traces.
What is the reason for not applying the fetched sampling configuration, are we missing anything ? could you please help us?

[yurishkuro]

What is printed when you do spewCfg.Dump(jaegerRemoteSampler)?

[Sucharitha95]

Hi @yurishkuro,
logs.txt
Please find the output log of spewCfg.Dump(jaegerRemoteSampler) in above file.
Apart from this I have one another query regarding sampling,
Using 5778 we are performing the sampling, is there any way of configuring the sampling without using 5778 port and directly configuring the sampling configuration using Jaeger collector.

[yurishkuro]

What tags do you get in the root span? The original Jaeger samplers would record the sampling method in the tags, not sure if OTEL SDK does that.

The output of the sampler state shows default probability of 0.7

[Sucharitha95]

Hi @yurishkuro ,
We have tried to check for the tags in the Query UI but could not find anything of such in the UI.
Also,Can you please let us know is there any port of sampling for Collector other than applying sampling using Agent 5778 port as we are trying to remove agent and directly connect to Collector we want to know is there any way to perform Sampling using Jaeger collector.

[yurishkuro]

We have an open ticket for it, but nobody stepped up to implement it. #1420

[yurishkuro]

My bad, it's been already implemented: #1971

[Sucharitha95]

Hi @yurishkuro ,
Thanks for the support, we have gone through the implementation in #1990, as per this sampling endpoint in collector was previously implemented using the Configmanager in pkg/clientcfg/clientcfghttp/cfgmgr.go and same is added in the cmd/collector/main.go, but currently the implementation is changed in cmd/collector/main.go and is not implemented using pkg/clientcfg/clientcfghttp. so is the latest code( cmd/collector/main.go) still can be used for adding sampling endpoint in collector?, May I know how it can be implemented.
In #1971 we have seen a discussion regarding JAEGER_SAMPLING_ENDPOINT environment variable but it was no where mentioned in the Jaeger documentation about the environment variable and it's usage.Can you please let us know how we can include and use the JAEGER_SAMPLING_ENDPOINT variable.