[processor/cumulativetodelta] Heavy memory usage of histograms

[balintzs]

Problem Description:
The Cumulative to Delta processor needs to record all numerical values of cumulative datapoints in order to calculate the difference between them. Values are stored based on their identity, which contains all resource, scope and datapoint attributes.

For histograms, the sum, the count and all bucket counts are stored as separate values. This causes huge spikes of heap memory usage when a large number of histogram datapoints pass through the processor.

The following screenshots show the heap memory allocation bytes and working set bytes for the same amount of data:
A) With Cumulative to Delta Processor turned on:

B) With Cumulative to Delta Processor turned off:

Proposed Solution:
After discussing the issue with @mistodon, we believe that instead of treating each numerical value in a histogram datapoint separately, we should store them together under a single entity, and add handling for calculating the delta of each value contained within.

[balintzs]

Link to issue requesting histogram support in Cumulative to Delta Processor: #12423

[TylerHelmuth]

@balintzs is this something you plan to work on?

[balintzs]

@TylerHelmuth, yes we have already started working on it.

[balintzs]

we found the cause of the memory issue and opened a PR with the fix: #14729

[balintzs]

We have been running OTel with the above fix in place (using a custom image we built) and it seems that this issue is now effectively resolved.
See the following charts for the before and after (fix deployed on Oct 5 at 1pm):