[Tracing Framework] Head Based Sampling Support

[devagarwal1803]

OpenSearch sampling strategy

As we consider the instrumentation of generic constructs such as rest controllers, transport actions, and task managers, a notable issue arises. Our system encompasses more than 200+ transport actions and a variety of rest actions, leading to the potential generation of numerous spans. However, each span incurs a non-negligible cost. Hence, it is crucial to develop a Sampling strategy that enables us to select a subset of spans from the total pool, effectively representing the entire population or system.

Head Based OpenSearch sampling

We should cover the following scenarios in head based sampling.

1. Blanket Probabilistic Sampling rate - We can use the OTel probabilistic sampler. Rate should be configurable through telemetry settings.
2. Override for on demand - We will override the blanket probabilistic sampler for the requests where header has "trace=true" attribute set.
3. Override for action/operation through settings - We can provide one more option for users to override the blanket probabilistic sampling for certain actions. They should be able to configure the action and sampling rate through logger like affix setting e.g. "trace.action-name=0.2". This will be needed for some internal actions, like background jobs, health checks, etc.
4. Custom processor - We would also require a custom processor which should be able to discard the parent spans in case the current span is not sampled. This is a bit tricky and requires a lot of assumptions and scenarios to be considered. I will post more on this separately.

More details on Sampling and other possible approaches of sampling are described in #8918