[processor/transform] Add Function to convert Exponential Histograms to normal Histograms

[daidokoro]

Component(s)

processor/transform

Is your feature request related to a problem? Please describe.

The Coralogix platform presently does not support ingesting metrics in the form of Exponential Histograms. We have clients currently facing this limitation while ingesting metrics from receivers that specifically only support generating Exponential Histograms. For example, the statsdreceiver

Describe the solution you'd like

We have created a solution which adds a custom conversion function to the transform processor, which handles converting exponential histograms to normal histograms.

A brief description of the key features of this function:

• Takes a single argument, the user-defined Explicit Boundaries to be used in the conversion.
• Non-Exponential Histogram metrics passed to this function are ignored.
• Only works within the metric context

Describe alternatives you've considered

We considered addressing the issue in the statsdreceiver and potentially add support for normal histograms there, however, this would only fix the issue for one receiver.

Having a dedicated function in the transform processor allows us to mitigate the issue for *all receivers and external metric sources.

Additional context

We've created a PR for this potential change: #33824

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[kentquirk]

I think some people will find this very useful, although I think it should be covered in warnings that the conversion is lossy and should only be used when there is no alternative. The results will not be identical.

I took a quick look at the draft PR, and it seems plausible but it needs:

• More comments on the algorithm used (for example, a note about how the bucket values are converted). I believe that all values in a bucket are assumed to be the max value of the bucket -- is that a good assumption, or would it be better to use the arithmetic or even geometric mean?
• Many more test cases. A single test case with two values is definitely not sufficient to show this is correct.

[crobert-1]

Removing needs triage based on code owner's response.

[daidokoro]

Hey @kentquirk

Thanks for your response and for having an initial look at the draft.

I'm currently working on adding more testing cases. I've also updated the transform processor README.md in the draft to reflect your recommendations for adding a usage warning.

To clarify the approach/algorithm:

Buckets are calculated based on a combination of the Explicit Boundaries that are passed to the function and the upper boundary of each exponential bucket.

calculateBucketCounts function calculates the bucket counts for a given exponential histogram data point. The algorithm is inspired by the logExponentialHistogramDataPoints function used to Print Exponential Histograms in Otel.

• factor is calculated as
  $factor = \ln(2) \times 2^{-\text{scale}}$
  math.Ldexp(math.Ln2, -scale)

• next we iterate the bucket counts and positions (pos) in the exponential histogram datapoint.

  • the index is calculated by adding the exponential offset to the positive bucket position (pos)
    $index = offset + pos$

  • the factor is then used to calculate the upper bound of the bucket which is calculated as
    upper = math.Exp((index+1) * factor)

At this point we know that the upper bound represents the highest value that can be in this bucket, so we take the upper bound and compare it to each of the explicit boundaries provided by the user until we find a boundary that fits, that is, the first instance where upper bound <= explicit boundary.

For eg.

If we have an explicit boundary of [0, 10, 20, 30] and an upper bound of 11, the count would be added to the explicit bound at 20, as it is the 1st value in which the upper bound is <= a given explicit boundary.

Technically, the explicit values of the histogram are never known in this conversion, we only calculate the upper boundaries and use them to determine the bucket based on the Explicit Boundaries defined by the user.

If the user provides Explicit Boundaries that do not fit the datapoints, this will result in imprecise conversions.

[daidokoro]

/label processor/transform needs-triage

Hey @kentquirk,

I've done the following:

• Added a few more test cases to the PR, if you believe more should be added, could you recommend additional cases.
• Added a description of the function to the README as well as a warning as you've described.
• Tweaked the algorithm used to convert exponential histograms and added comments in the code describing how it is done.

The PR has been set to active from draft.

Let me know if anything else is required.

Thanks.

[github-actions]

Pinging code owners for processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[kentquirk]

This is not stale, just waiting for the PR to be approved and merged.

[kentquirk]

/label -Stale

[oertl]

Maybe helpful:

In DynaHist, we have implemented a generic mapping between histograms with different bucket layouts by first considering the reconstruction of the value. We have implemented 4 strategies (lower, upper, midpoint, and uniform) there. We don't have any random reconstruction, as we require reproducibility. We first create an object describing the reconstructed values in ascending order depending on the chosen strategy. This can then be used to efficiently insert the reconstructed values (without explicitly calculating all of them) into the other histogram with a different layout. In this way, we could nicely decouple the reconstruction of values from the insertion code (see https://github.com/dynatrace-oss/dynahist/blob/94608772e16a1dbaed4594eeb38eaa240a89fbce/src/main/java/com/dynatrace/dynahist/AbstractMutableHistogram.java#L120).

The reconstruction strategy can also be used to specify quantile estimation. In Dynahist, quantile estimation is defined as a combination of a reconstruction strategy and a sample quantile estimation method (see https://github.com/dynatrace-oss/dynahist/blob/94608772e16a1dbaed4594eeb38eaa240a89fbce/src/main/java/com/dynatrace/dynahist/AbstractHistogram.java#L233). This enables better reproducibility of the reported quantiles as quantile estimation based on individual values (not to mention histograms) is already ambiguous enough (cf. https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample).