DataDog Exporter is exporting 0 values for histograms

[emilgelman]

Describe the bug
Using the latest configuration for histograms added in v0.36.0, it seems like the name.bucket metric is always 0 in DataDog.

Steps to reproduce

Full code example:

package main

import (
	"context"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/metric/global"
	controller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
	processor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
	"go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func main() {
	ctx := context.Background()
	client := otlpmetrichttp.NewClient(otlpmetrichttp.WithInsecure(), otlpmetrichttp.WithEndpoint("0.0.0.0:55681"))
	exp, err := otlpmetric.New(ctx, client)
	if err != nil {
		panic("Failed to create the collector exporter")
	}
	defer func() {
		ctx, cancel := context.WithTimeout(ctx, time.Second)
		defer cancel()
		if err := exp.Shutdown(ctx); err != nil {
			otel.Handle(err)
		}
	}()

	pusher := controller.New(
		processor.New(
			simple.NewWithHistogramDistribution(),
			exp,
		),
		controller.WithExporter(exp),
		controller.WithCollectPeriod(time.Millisecond*100),
	)
	global.SetMeterProvider(pusher.MeterProvider())

	if err := pusher.Start(ctx); err != nil {
		panic("could not start metric controoler: %v")
	}
	defer func() {
		ctx, cancel := context.WithTimeout(ctx, time.Second)
		defer cancel()
		// pushes any last exports to the receiver
		if err := pusher.Stop(ctx); err != nil {
			otel.Handle(err)
		}
	}()

	meter := global.Meter("test-meter")

	// Recorder metric example
	recorder := metric.Must(meter).NewInt64Histogram("test.histogram.histogram.distribution.new")
	recorder.Record(ctx, 1000)
	time.Sleep(time.Second * 1)
	recorder.Record(ctx, -20000)
	time.Sleep(time.Second * 1)
	recorder.Record(ctx, 5000)
	time.Sleep(time.Second * 1)

}

What did you expect to see?
I would expect the output in DataDog to match the values I'm seeing in stdout / prometheus exporters.
Output from stdout exporter:

InstrumentationLibraryMetrics #0
InstrumentationLibrary  
Metric #0
Descriptor:
     -> Name: test.histogram.histogram.distribution.new
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints #0
Data point attributes:
     -> HOST_NAME: STRING(emil-gelman)
StartTimestamp: 2021-09-22 08:36:17.710314 +0000 UTC
Timestamp: 2021-09-22 08:36:19.812914 +0000 UTC
Count: 3
Sum: -14000.000000
ExplicitBounds #0: 5000.000000
ExplicitBounds #1: 10000.000000
ExplicitBounds #2: 25000.000000
ExplicitBounds #3: 50000.000000
ExplicitBounds #4: 100000.000000
ExplicitBounds #5: 250000.000000
ExplicitBounds #6: 500000.000000
ExplicitBounds #7: 1000000.000000
ExplicitBounds #8: 2500000.000000
ExplicitBounds #9: 5000000.000000
ExplicitBounds #10: 10000000.000000
Buckets #0, Count: 2
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0

What did you see instead?
Below is the response of the DataDog query API for the reported metric:

{
    "status": "ok",
    "resp_version": 1,
    "series": [
        {
            "end": 1632299999000,
            "attributes": {},
            "metric": "test.histogram.histogram.distribution.new.bucket",
            "interval": 300,
            "tag_set": [
                "lower_bound:-inf",
                "lower_bound:10000.0",
                "lower_bound:100000.0",
                "lower_bound:1e_07.0"
            ],
            "start": 1632294600000,
            "length": 4,
            "query_index": 0,
            "aggr": "avg",
            "scope": "lower_bound:-inf,lower_bound:10000.0,lower_bound:100000.0,lower_bound:1e_07.0",
            "pointlist": [
                [
                    1632294600000.0,
                    0.0
                ],
                [
                    1632294900000.0,
                    0.0
                ],
                [
                    1632295200000.0,
                    0.0
                ],
                [
                    1632299700000.0,
                    0.0
                ]
            ],
            "expression": "avg:test.histogram.histogram.distribution.new.bucket{lower_bound:-inf,lower_bound:10000.0,lower_bound:100000.0,lower_bound:1e_07.0}",
            "unit": null,
            "display_name": "test.histogram.histogram.distribution.new.bucket"
        }
    ],
    "to_date": 1632300222000,
    "query": "avg:test.histogram.histogram.distribution.new.bucket{*} by {lower_bound}",
    "message": "",
    "res_type": "time_series",
    "times": [],
    "from_date": 1632250222000,
    "group_by": [
        "lower_bound"
    ],
    "values": []
}

What version did you use?
Version: v0.36.0

What config did you use?
otel-collector datadog exporter configuration:

exporters:
  datadog:
    use_resource_metadata: false
    env: "local"
    send_metadata: false
    api:
      key: ${DD_API_KEY}
      site: datadoghq.eu
    metrics:
      send_monotonic_counter: false
      histograms:
        mode: counters

Environment
OS: iOS
Compiler(if manually compiled): go 1.16

[mx-psi]

Thanks for reporting this @emilgelman the context is really useful, I really appreciate you took the time to open the issue :) We did a recent refactor and it's possible we are not handling cumulative histograms correctly. I will have a look today.

In the meantime, can you try using v0.35.0 and check if it works correctly for you there? You will have to use the older configuration unfortunately (so report_buckets: true instead of histograms::mode).

[emilgelman]

Thank you @mx-psi for looking into it.
I reverted to v0.35.0 and changed the configuration to

exporters:
  datadog:
    use_resource_metadata: false
    env: "local"
    send_metadata: false
    api:
      key: ${DD_API_KEY}
      site: datadoghq.eu
    metrics:
      send_monotonic_counter: false
      report_buckets: true

Using the same code as before, I now get a name.count_per_bucket metric. However, the values are still 0, and there is only one bucket tag being reported bucket_idx:11

stdout exporter output:

InstrumentationLibraryMetrics #0
InstrumentationLibrary  
Metric #0
Descriptor:
     -> Name: test.histogram.histogram.distribution.035
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints #0
Data point attributes:
     -> HOST_NAME: STRING(emil-gelman)
StartTimestamp: 2021-09-22 09:27:28.454164 +0000 UTC
Timestamp: 2021-09-22 09:27:30.554678 +0000 UTC
Count: 3
Sum: -14000.000000
ExplicitBounds #0: 5000.000000
ExplicitBounds #1: 10000.000000
ExplicitBounds #2: 25000.000000
ExplicitBounds #3: 50000.000000
ExplicitBounds #4: 100000.000000
ExplicitBounds #5: 250000.000000
ExplicitBounds #6: 500000.000000
ExplicitBounds #7: 1000000.000000
ExplicitBounds #8: 2500000.000000
ExplicitBounds #9: 5000000.000000
ExplicitBounds #10: 10000000.000000
Buckets #0, Count: 2
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0

DataDog query API output:

{
    "status": "ok",
    "resp_version": 1,
    "series": [
        {
            "end": 1632302999000,
            "attributes": {},
            "metric": "test.histogram.histogram.distribution.035.count_per_bucket",
            "interval": 300,
            "tag_set": [
                "bucket_idx:11"
            ],
            "start": 1632302700000,
            "length": 1,
            "query_index": 0,
            "aggr": "avg",
            "scope": "bucket_idx:11",
            "pointlist": [
                [
                    1632302700000.0,
                    0.0
                ]
            ],
            "expression": "avg:test.histogram.histogram.distribution.035.count_per_bucket{bucket_idx:11}",
            "unit": null,
            "display_name": "test.histogram.histogram.distribution.035.count_per_bucket"
        }
    ],
    "to_date": 1632303080000,
    "query": "avg:test.histogram.histogram.distribution.035.count_per_bucket{*} by {bucket_idx}",
    "message": "",
    "res_type": "time_series",
    "times": [],
    "from_date": 1632250222000,
    "group_by": [
        "bucket_idx"
    ],
    "values": []
}

[mx-psi]

@emilgelman Thanks for checking with v0.35. I think I understand what is going on. Datadog's backend timeseries storage stores at most one point per second. This means that if multiple points are sent in the same second, the behavior is unspecified (in practice, probably the last datapoint that gets to the backend for a given second wins).

In your example you send an OTLP Histogram every 0.1 seconds, and each histogram has one point (so all buckets are empty except for one). This probably means that most of the time, all bucket metrics would get a 0 value winning, which would explain what you see. Setting the controller.WithCollectPeriod to a duration of 1 second or more should help with this. I think setting a higher collect period is generally good practice, whether you are using Datadog or any other vendor, to reduce the churn of the instrumentation library.

You may also want to set custom histogram bounds if you find that the default bounds (5000, 10000, ...) don't provide a high enough resolution for you (for example, if all or most points end up in the same bucket).

Does that explanation make sense?

---

(edit) Also, a side note: if sub-second resolution for timeseries is a feature you would like Datadog to have, please write to Datadog support to make a feature request so that interest gets tracked

[mx-psi]

I had misunderstood your code so the explanation above is incorrect. I have been able to verify that payloads are sent correctly to Datadog so I am digging further into this. Will keep you posted :)

[mx-psi]

Hi again, using your code we haven't been able to reproduce this, so I have some questions to help troubleshoot this further:

• Were you sending histograms before with v0.35.0 (or a previous version) without having this issue?
• Could you run your test while sending metrics to a different org (e.g. to a personal Datadog account or a different org within your company) and report the results?

[emilgelman]

1. We did not try before 0.35.0
2. Unfortunately we have a single organization, so I can't try it anywhere else. I have opened a support request for DataDog to track this.

I guess we can close this if this is a specific case.

[mx-psi]

Sounds like a good plan, I will make sure to sync with our Solutions Engineer so that they have all the context

[mx-psi]

We may have a fix so I am going to reopen this. It looks like the issue may be caused by an interaction between a bug on the Collector side and undefined behavior on the backend when sending malformed metrics

[ransoor2]

Hi @mx-psi, looks like there are still issues.
I am using sdk v0.24.0 and collector version 0.37.1.

I ran Emil's code example with the same configuration and compared the collector's datadog exporter logs with the collector's logging exporter.

Logging exporter:

InstrumentationLibraryMetrics #0
InstrumentationLibrary
Metric #0
Descriptor:
     -> Name: test.ran.histogram3
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints #0
Data point attributes:
     -> ENVIRONMENT_TYPE: STRING(local)
     -> ENVTYPE: STRING(local)
     -> HOST_NAME: STRING(ran)
     -> env: STRING(local)
     -> mode: STRING(default)
     -> service: STRING(ran-local)
StartTimestamp: 2021-10-24 09:09:58.152098 +0000 UTC
Timestamp: 2021-10-24 09:10:00.255854 +0000 UTC
Count: 3
Sum: -14000.000000
ExplicitBounds #0: 5000.000000
ExplicitBounds #1: 10000.000000
ExplicitBounds #2: 25000.000000
ExplicitBounds #3: 50000.000000
ExplicitBounds #4: 100000.000000
ExplicitBounds #5: 250000.000000
ExplicitBounds #6: 500000.000000
ExplicitBounds #7: 1000000.000000
ExplicitBounds #8: 2500000.000000
ExplicitBounds #9: 5000000.000000
ExplicitBounds #10: 10000000.000000
Buckets #0, Count: 2
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0

Datadog exporter logs:

  "kind": "exporter",
  "name": "datadog",
  "metric": [
    {
      "metric": "test.ran.histogram3.count",
      "points": [
        [
          1635066600,
          1
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.sum",
      "points": [
        [
          1635066600,
          5000
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:-inf",
        "upper_bound:5000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          1
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:5000.0",
        "upper_bound:10000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:10000.0",
        "upper_bound:25000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:25000.0",
        "upper_bound:50000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:50000.0",
        "upper_bound:100000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:100000.0",
        "upper_bound:250000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:250000.0",
        "upper_bound:500000.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:500000.0",
        "upper_bound:1e+06.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:1e+06.0",
        "upper_bound:2.5e+06.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:2.5e+06.0",
        "upper_bound:5e+06.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:5e+06.0",
        "upper_bound:1e+07.0",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "test.ran.histogram3.bucket",
      "points": [
        [
          1635066600,
          0
        ]
      ],
      "type": "count",
      "host": "",
      "tags": [
        "lower_bound:1e+07.0",
        "upper_bound:inf",
        "ENVIRONMENT_TYPE:local",
        "ENVTYPE:local",
        "HOST_NAME:ran",
        "env:local",
        "mode:default",
        "service:ran-local"
      ]
    },
    {
      "metric": "otel.datadog_exporter.metrics.running",
      "points": [
        [
          1635066600,
          1
        ]
      ],
      "type": "gauge",
      "host": "",
      "tags": [
        "version:v0.37.1",
        "command:otelcontribcol"
      ]
    }
  ]
}

It looks like datadog exporter only takes the last point (5000) and ignores the rest