Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark exemplars as stable. #3870

Merged
merged 8 commits into from
Apr 20, 2024
Merged

Conversation

jsuereth
Copy link
Contributor

@jsuereth jsuereth commented Feb 8, 2024

Fixes #3756

Changes

  • Marks Exemplar specification as stable.

For non-trivial changes, follow the change proposal process.

@cijothomas
Copy link
Member

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

@reyang
Copy link
Member

reyang commented Feb 14, 2024

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

@cijothomas
Copy link
Member

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

Those link does not tell much...For example, in JavaScript some exemplar related classes are there, but its not connected to SDK - its as good as saying JS does not support Exemplars... Which is why I was hoping every languages who implemented it will update the matrix, so we know if we have 3 languages implementing the spec. (or we can ask around as well :) )

OTel C++ also has some exemplars..

@MrAlias
Copy link
Contributor

MrAlias commented Feb 14, 2024

Go has an experimental implementation. It does not expose any API for configuration outside of environment variables.

Copy link

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Feb 22, 2024
Copy link

Closed as inactive. Feel free to reopen if this PR is still being worked on.

@github-actions github-actions bot closed this Feb 29, 2024
@reyang reyang reopened this Feb 29, 2024
@reyang reyang removed the Stale label Feb 29, 2024
Copy link

github-actions bot commented Mar 8, 2024

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@reyang
Copy link
Member

reyang commented Mar 15, 2024

The TC discussed this during the Mar. 13th, 2024 meeting:

  1. There is only one blocking issue (more details can be found from Tracking: Exemplar Specification - Stabilization #3756), @jack-berg is working on it Clarify that exemplar reservoir default may change in a minor version #3943.
  2. After PR Clarify that exemplar reservoir default may change in a minor version #3943 is merged, we should be ready to move this PR forward. We're targeting mid Apr. for the Metrics Exemplar spec to be marked Stable (spec release will happen early May).

jack-berg added a commit that referenced this pull request Mar 20, 2024
…#3943)

Related to #3756. 

See this
[conversation](#3870 (comment))
for context.

---------

Co-authored-by: Reiley Yang <reyang@microsoft.com>
@jsuereth jsuereth marked this pull request as ready for review March 21, 2024 14:31
@jsuereth jsuereth requested review from a team March 21, 2024 14:31
@jsuereth
Copy link
Contributor Author

This is now ready for review/merge

Copy link
Member

@jack-berg jack-berg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was giving the exemplar spec a final read and noticed an issue.

specification/metrics/sdk.md Show resolved Hide resolved
@reyang
Copy link
Member

reyang commented Apr 11, 2024

#3994 would hopefully be the last piece before we mark Exemplar spec stable. Please refer to the #3944 PR description for the Technical Committee discussion details.

@cijothomas would you check if it unblocks this PR? Thanks!

@reyang reyang reopened this Apr 11, 2024
@github-actions github-actions bot removed the Stale label Apr 11, 2024
@cijothomas
Copy link
Member

#3994 would hopefully be the last piece before we mark Exemplar spec stable. Please refer to the #3944 PR description for the Technical Committee discussion details.

@cijothomas would you check if it unblocks this PR? Thanks!

Sure. Will check and post back here.

Copy link
Member

@cijothomas cijothomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on #3994 (comment), languages are free to chose a different default, and changing the default is considered non-breaking.
All the blocking issues are hence, addressable post stable as well.

@reyang
Copy link
Member

reyang commented Apr 16, 2024

@open-telemetry/specs-approvers @open-telemetry/specs-metrics-approvers @open-telemetry/technical-committee PTAL, I think this is ready to be merged by end of Fri. this week (Apr. 19th, 2024, PT).

@jack-berg @jmacd @jsuereth FYI

@cijothomas
Copy link
Member

open-telemetry/opentelemetry-dotnet#5545 (comment) OTel .NET's performance hit when enabling exemplars is shown here. And OTel .NET is planning to make Exemplars opt-in for now.

@jack-berg Do you have some perf numbers from Java handy that can be shared?

@jmacd jmacd self-requested a review April 19, 2024 18:24
@jmacd
Copy link
Contributor

jmacd commented Apr 19, 2024

I measured the Lightstep metrics SDK implementation of exemplars here and the performance hit is consistent with what has been reported for .NET. For example, a fast-path that was 45ns becomes 60ns, for a 33% hit. I believe this is acceptable, and if there are applications for which a few 10s or 100s of nanoseconds of additional cost per metrics event are limiting, there may be other paths to optimization other than simply disabling exemplars.

lightstep/otel-launcher-go#576 (comment)

@jack-berg
Copy link
Member

Here are the benchmarks from the java repo. Sharing these without having recently dug into what elements of the exemplar implementation drive perf.

The benchmark tests a variety of dimension:

  • Single thread vs. multiple threads (in "Benchmark" column)
  • Configuration of attributes which receive measurements (in "Benchmark" column)
  • The instrument being recorded to (counter, histogram, etc, in the "(opBuilder)" column)
  • The sdk configuration (noop api, exemplars enabled, examplars disabled, temporality, in the "(sdk)" column)
Benchmark                                                                     (opBuilder)             (sdk)  Mode  Cnt     Score     Error   Units
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd          API_ONLY  avgt   10     0.982 ±   0.034   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10  1241.016 ±  11.347   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd    SDK_CUMULATIVE  avgt   10  1235.555 ±  21.709   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd         SDK_DELTA  avgt   10   983.321 ±  12.307   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd          API_ONLY  avgt   10     1.006 ±   0.072   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10   961.077 ±   9.234   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd    SDK_CUMULATIVE  avgt   10  1235.765 ±  16.153   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.005 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd         SDK_DELTA  avgt   10  1224.222 ±  27.974   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord          API_ONLY  avgt   10     2.716 ±   0.085   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1276.578 ±  10.122   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  1270.384 ± 114.901   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    14.956 ±  12.737  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     2.439 ±   2.380    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.time                DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord         SDK_DELTA  avgt   10  1292.251 ±   4.334   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord          API_ONLY  avgt   10     2.046 ±   0.006   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1313.262 ±   9.764   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord    SDK_CUMULATIVE  avgt   10  1290.082 ±  34.783   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord         SDK_DELTA  avgt   10  1259.403 ±  39.749   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord         SDK_DELTA  avgt   10    14.872 ±   2.581  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord         SDK_DELTA  avgt   10     2.433 ±   0.366    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord         SDK_DELTA  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.time                  LongHistogramRecord         SDK_DELTA  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd          API_ONLY  avgt   10     1.027 ±   0.020   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10   987.828 ±  23.329   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd    SDK_CUMULATIVE  avgt   10   969.901 ±  11.455   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd         SDK_DELTA  avgt   10  1239.477 ±  23.283   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd          API_ONLY  avgt   10     0.964 ±   0.007   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10  1258.761 ±  21.210   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10   972.530 ±  19.704   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd         SDK_DELTA  avgt   10  1237.105 ±   7.030   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord          API_ONLY  avgt   10     3.551 ±   0.129   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1413.131 ±  79.087   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  1293.687 ±  32.239   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord         SDK_DELTA  avgt   10  1300.389 ±  21.223   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord          API_ONLY  avgt   10     2.165 ±   0.196   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1321.396 ±  16.329   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord    SDK_CUMULATIVE  avgt   10  1214.101 ±  50.256   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord    SDK_CUMULATIVE  avgt   10    12.611 ±   8.528  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord    SDK_CUMULATIVE  avgt   10     1.925 ±   1.395    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord    SDK_CUMULATIVE  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.time               LongHistogramRecord    SDK_CUMULATIVE  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord         SDK_DELTA  avgt   10  1297.769 ±   5.424   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd          API_ONLY  avgt   10     0.901 ±   0.008   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    17.664 ±   0.143   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd    SDK_CUMULATIVE  avgt   10    29.145 ±   0.822   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd         SDK_DELTA  avgt   10    28.836 ±   0.948   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd          API_ONLY  avgt   10     0.897 ±   0.008   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    18.270 ±   0.159   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    29.956 ±   0.859   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd         SDK_DELTA  avgt   10    29.145 ±   0.829   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord          API_ONLY  avgt   10     2.532 ±   0.097   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.472 ±   0.138   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   118.182 ±   0.482   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord         SDK_DELTA  avgt   10   118.630 ±   0.833   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord         SDK_DELTA  avgt   10   192.914 ±   1.353  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord         SDK_DELTA  avgt   10    24.000 ±   0.001    B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.oneThread:gc.time                                 DoubleHistogramRecord         SDK_DELTA  avgt   10     5.000                ms
MetricsBenchmarks.oneThread                                           LongHistogramRecord          API_ONLY  avgt   10     1.913 ±   0.006   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.737 ±   0.439   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord    SDK_CUMULATIVE  avgt   10   117.504 ±   0.741   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord         SDK_DELTA  avgt   10   118.460 ±   1.293   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord         SDK_DELTA  avgt   10   193.199 ±   2.106  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord         SDK_DELTA  avgt   10    24.000 ±   0.001    B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.oneThread:gc.time                                   LongHistogramRecord         SDK_DELTA  avgt   10     5.000                ms
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd          API_ONLY  avgt   10     4.268 ±   0.029   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    90.090 ±   1.211   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10   338.722 ±   4.522  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    32.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     6.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                       LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     9.000                ms
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd    SDK_CUMULATIVE  avgt   10   141.212 ±   0.982   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd         SDK_DELTA  avgt   10   141.101 ±   0.590   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd          API_ONLY  avgt   10     4.282 ±   0.042   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    92.942 ±   0.717   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10   328.316 ±   2.523  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    32.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     5.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                     DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     8.000                ms
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd    SDK_CUMULATIVE  avgt   10   142.815 ±   1.041   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd         SDK_DELTA  avgt   10   142.041 ±   0.833   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord          API_ONLY  avgt   10    12.945 ±   0.130   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10   177.686 ±   5.963   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   612.016 ±  17.927   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻³              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord         SDK_DELTA  avgt   10   606.133 ±   2.685   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord         SDK_DELTA  avgt   10    ≈ 10⁻³              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord          API_ONLY  avgt   10    10.684 ±   0.073   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10   176.199 ±   2.489   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord    SDK_CUMULATIVE  avgt   10   613.344 ±  23.925   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord    SDK_CUMULATIVE  avgt   10   186.664 ±   6.977  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord    SDK_CUMULATIVE  avgt   10   120.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord    SDK_CUMULATIVE  avgt   10     3.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                  LongHistogramRecord    SDK_CUMULATIVE  avgt   10     5.000                ms
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord         SDK_DELTA  avgt   10   606.087 ±   3.686   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord         SDK_DELTA  avgt   10   188.795 ±   1.145  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord         SDK_DELTA  avgt   10   120.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                  LongHistogramRecord         SDK_DELTA  avgt   10     6.000                ms

There's a lot of noise here, so I've narrowed it down to show the time difference with a single thread and exemplars enabled and disabled:

Benchmark                                                                     (opBuilder)             (sdk)  Mode  Cnt     Score     Error   Units
MetricsBenchmarks.oneThread                                                LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    17.664 ±   0.143   ns/op
MetricsBenchmarks.oneThread                                                LongCounterAdd    SDK_CUMULATIVE  avgt   10    29.145 ±   0.822   ns/op
MetricsBenchmarks.oneThread                                              DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    18.270 ±   0.159   ns/op
MetricsBenchmarks.oneThread                                              DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    29.956 ±   0.859   ns/op
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.472 ±   0.138   ns/op
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   118.182 ±   0.482   ns/op
MetricsBenchmarks.oneThread                                           LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.737 ±   0.439   ns/op
MetricsBenchmarks.oneThread                                           LongHistogramRecord    SDK_CUMULATIVE  avgt   10   117.504 ±   0.741   ns/op

There's no doubt a performance impact to enabling examplars, but to add some context:

  • The record path with exemplars disabled is very optimized. The impact of exemplars may look high as a percentage compared to the base, but isn't much in absolute terms.
  • Small improvements are probably possible for the histogram exemplar reservoir. We use linear search for the bucket index today, and should probably use binary search when buckets are greater than a particular size. Also, quickly glancing at the profile, the majority of the additional time spent is accessing the current nano time from the VM, which seems like it can be optimized.
  • Java has had exemplars enabled by default since we first published a stable metrics API / SDK years ago. We've had nobody complain about performance or the defaults.

@jack-berg
Copy link
Member

Benchmark after optimizing accessing the current nano time from the VM below. Much more in line with what we're seeing from .NET and Go.

Benchmark                                                 (opBuilder)             (sdk)  Mode  Cnt   Score    Error   Units
MetricsBenchmarks.oneThread                            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10  17.323 ±  0.309   ns/op
MetricsBenchmarks.oneThread                            LongCounterAdd    SDK_CUMULATIVE  avgt   10  28.126 ±  1.143   ns/op
MetricsBenchmarks.oneThread                     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  33.842 ±  1.248   ns/op
MetricsBenchmarks.oneThread                     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  55.855 ±  2.418   ns/op

@reyang reyang merged commit e6eccba into open-telemetry:main Apr 20, 2024
7 checks passed
@carlosalberto carlosalberto mentioned this pull request May 6, 2024
carlosalberto added a commit that referenced this pull request May 9, 2024
May 2024 release.

Important changes:

* Mark exemplars as stable (#3870)
* Mark synchronous gauge as stable (#4019)
* Record Links with empty/invalid SpanContext (#3928)
carlosalberto pushed a commit to carlosalberto/opentelemetry-specification that referenced this pull request Oct 31, 2024
…open-telemetry#3943)

Related to open-telemetry#3756. 

See this
[conversation](open-telemetry#3870 (comment))
for context.

---------

Co-authored-by: Reiley Yang <reyang@microsoft.com>
carlosalberto pushed a commit to carlosalberto/opentelemetry-specification that referenced this pull request Oct 31, 2024
carlosalberto added a commit to carlosalberto/opentelemetry-specification that referenced this pull request Oct 31, 2024
May 2024 release.

Important changes:

* Mark exemplars as stable (open-telemetry#3870)
* Mark synchronous gauge as stable (open-telemetry#4019)
* Record Links with empty/invalid SpanContext (open-telemetry#3928)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tracking: Exemplar Specification - Stabilization
7 participants