Observing Error posting hostname/tags series in Datadog exporter #26695

narasimharaojm · 2023-09-14T19:30:42Z

Component(s)

exporter/datadog

What happened?

Description

Observing Error posting hostname/tags series in Datadog exporter. While exporting traces in Datadog exporter, it was observed that api endpoint was malformed and resulting in errors posting hostname/tags series.

Steps to Reproduce

Exporter config used,

datadog:
api:
site: https://.api.datadoghq.com
key:
tls:
insecure_skip_verify: true
traces:
endpoint: https://.agent.datadoghq.com
sending_queue:
enabled: true
queue_size: 100000
num_consumers: 100
retry_on_failure:
enabled: false

Expected Result

We should not be seeing any errors due to malformed endpoint while attempting to send hostname/tag series

Actual Result

Sample Error,
2023-09-14T18:48:35.279Z error datadogexporter@v0.83.0/traces_exporter.go:172 Error posting hostname/tags series {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "Post "https://api.https//.api.datadoghq.com/api/v2/series": dial tcp: lookup api.https on 169.254.169.254:53: no such host"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*traceExporter).exportUsageMetrics
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.83.0/traces_exporter.go:172
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*traceExporter).consumeTraces
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.83.0/traces_exporter.go:130
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/traces.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/common.go:197
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:355
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/traces.go:126
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/internal/bounded_memory_queue.go:47

Collector version

v0.83.0

Environment information

Environment

OS: Ubuntu 20.04

OpenTelemetry Collector configuration

extensions:
  health_check:
  pprof:
  memory_ballast:
    size_mib: 36000
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 100
    send_batch_max_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 108000
    spike_limit_mib: 21500
  tail_sampling:
exporters:
  datadog:
    api:
      site: https://<org>.api.datadoghq.com
      key: <key>
    tls:
        insecure_skip_verify: true
    traces:
        endpoint: https://<org>.agent.datadoghq.com
    sending_queue:
      enabled: true
      queue_size: 200000
      num_consumers: 100
    timeout: 200ms
    retry_on_failure:
      enabled: false
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m
  logging:
    verbosity: normal
    sampling_initial: 10
    sampling_thereafter: 50

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [datadog]
  extensions: [health_check, pprof]

Log output

2023-09-14T18:49:39.868Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}
2023-09-14T18:49:39.868Z	info	service/service.go:161	Everything is ready. Begin running and processing data.
2023-09-14T18:49:39.955Z	warn	clientutil/api.go:48	Error while validating API key	{"kind": "exporter", "data_type": "traces", "name": "datadog/tail_sampling", "error": "Get \"https://api.https//paypalpoc.api.datadoghq.com/api/v1/validate\": dial tcp: lookup api.https on 169.254.169.254:53: no such host"}
2023-09-14T18:49:40.378Z	info	clientutil/api.go:44	API key validation successful.	{"kind": "exporter", "data_type": "metrics", "name": "datadog/otel_metrics"}
2023/09/14 18:49:45 WARN: failed to get session token, falling back to IMDSv1: 405 Method Not Allowed: Method Not Allowed
	status code: 405, request id: 
caused by: EC2MetadataError: failed to make EC2Metadata request
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 405 (Method Not Allowed)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>405.</b> <ins>That’s an error.</ins>
  <p>The request method <code>PUT</code> is inappropriate for the URL <code>/latest/api/token</code>.  <ins>That’s all we know.</ins>

	status code: 405, request id: 


2023-09-14T18:48:35.279Z  error datadogexporter@v0.83.0/traces_exporter.go:172  Error posting hostname/tags series  {"kind": "exporter", "data_type": "traces", "name": "datadog", "error": "Post \"https://api.https//<org>.api.datadoghq.com/api/v2/series\": dial tcp: lookup api.https on 169.254.169.254:53: no such host"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*traceExporter).exportUsageMetrics
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.83.0/traces_exporter.go:172
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*traceExporter).consumeTraces
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.83.0/traces_exporter.go:130
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/traces.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/common.go:197
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:355
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/traces.go:126
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
  go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/internal/bounded_memory_queue.go:47

Additional context

No response

github-actions · 2023-09-14T19:31:00Z

Pinging code owners:

exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

songy23 · 2023-09-14T21:00:05Z

Hi @narasimharaojm your config of datadog: api: site is wrong, it should be either datadoghq.com or datadoghq.eu or other US regions like us3.datadoghq.com. Same goes for datadog: traces: endpoint. You can simply remove these 2 lines.

...
exporters:
  datadog:
    api:
      site: https://<org>.api.datadoghq.com  # remove
     ...
    traces:
        endpoint: https://<org>.agent.datadoghq.com  # remove
...

narasimharaojm · 2023-09-15T22:27:18Z

@songy23 The endpoints which I mentioned in exporter config were created and provided to us.

songy23 · 2023-09-16T02:45:15Z

@narasimharaojm in that case you need to set datadog: metrics: endpoint rather than datadog: api: site

...
exporters:
  datadog:
    api:
      site: https://<org>.api.datadoghq.com  # remove
     ...
    metrics:
        endpoint: https://<org>.api.datadoghq.com

narasimharaojm · 2023-09-16T03:57:02Z

@songy23 This is config we have for traces and metrics ingestion.

datadog:
    api:
      site: https://<org>.api.datadoghq.com
      key: <key>
    tls:
        insecure_skip_verify: true
    traces:
        endpoint: https://<org>.agent.datadoghq.com
    metrics:
        endpoint: https://<org>.api.datadoghq.com

Could you pls let me know what config should I use to ingest traces with out getting Error posting hostname/tags series errors

songy23 · 2023-09-18T13:36:21Z

Please remove this line:

site: https://<org>.api.datadoghq.com

Please open a Datadog support ticket if the problem persists after you make the config change, and we'll get the correct team to look into it.

narasimharaojm · 2023-09-18T15:50:28Z

@songy23 Thanks for the looking into and suggestion on config usage. I also observed timeout provided in exporter was too low. Hence, increased its value from 200ms to 1s.

datadog:
    api:
      site: https://<org>.api.datadoghq.com
      key: <key>
    tls:
        insecure_skip_verify: true
    traces:
        endpoint: https://<org>.agent.datadoghq.com
    metrics:
        endpoint: https://<org>.api.datadoghq.com
    timeout: 1s

I see now, key was getting successfully validated and no longer seeing Error posting hostname/tags series

narasimharaojm added bug Something isn't working needs triage New item requiring triage labels Sep 14, 2023

github-actions bot added the exporter/datadog Datadog components label Sep 14, 2023

songy23 removed bug Something isn't working needs triage New item requiring triage labels Sep 14, 2023

mx-psi added the waiting for author label Sep 15, 2023

songy23 closed this as completed Sep 18, 2023

github-actions bot mentioned this issue Sep 19, 2023

Weekly Report: 2023-09-12 - 2023-09-19 kevinslin/opentelemetry-collector-contrib#26

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Observing Error posting hostname/tags series in Datadog exporter #26695

Observing Error posting hostname/tags series in Datadog exporter #26695

narasimharaojm commented Sep 14, 2023

github-actions bot commented Sep 14, 2023

songy23 commented Sep 14, 2023 •

edited

Loading

narasimharaojm commented Sep 15, 2023

songy23 commented Sep 16, 2023

narasimharaojm commented Sep 16, 2023 •

edited

Loading

songy23 commented Sep 18, 2023

narasimharaojm commented Sep 18, 2023

Observing Error posting hostname/tags series in Datadog exporter #26695

Observing Error posting hostname/tags series in Datadog exporter #26695

Comments

narasimharaojm commented Sep 14, 2023

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Sep 14, 2023

songy23 commented Sep 14, 2023 • edited Loading

narasimharaojm commented Sep 15, 2023

songy23 commented Sep 16, 2023

narasimharaojm commented Sep 16, 2023 • edited Loading

songy23 commented Sep 18, 2023

narasimharaojm commented Sep 18, 2023

songy23 commented Sep 14, 2023 •

edited

Loading

narasimharaojm commented Sep 16, 2023 •

edited

Loading