Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[datadogexporter] Use resource attributes for metadata and generated metrics #2023

Merged
merged 12 commits into from
Jan 29, 2021

Conversation

mx-psi
Copy link
Member

@mx-psi mx-psi commented Jan 15, 2021

Description:

  • Set hostname to running metrics from resource attributes if available
  • Use resource metadata to complete host metadata
  • Improve host metadata on cloud environments (AWS and GCP)
  • Add tests for new metadata behavior

Link to tracking Issue: Relates to #1379
Testing: Added unit tests. Tested in e2e environment.

  • Set up the resource detection processor with the environment detector
  • Use it to add a datadog.host.name to something like my-custom-hostname
  • Test that it outputs metrics and metadata with my-custom-hostname

Testing in AWS and GCP should bring a subset of the information that the Datadog Agent provides on its latest version (7.25)

Documentation: Documented new flag

Running metrics can be duplicate in this setup but this is fine since
they are gauges.
If enabled (it is by default) the resource attributes from the first
payload from either metrics or traces will be used to populate host
metadata. If there are fields missing these will be filled in by the
exporter (this preserves behavior when not using
k8s_tagger/resourcedetection).
@codecov
Copy link

codecov bot commented Jan 15, 2021

Codecov Report

Merging #2023 (6b3b21b) into main (8ca2e9b) will increase coverage by 0.15%.
The diff coverage is 89.16%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2023      +/-   ##
==========================================
+ Coverage   90.52%   90.68%   +0.15%     
==========================================
  Files         397      398       +1     
  Lines       19689    19757      +68     
==========================================
+ Hits        17824    17917      +93     
+ Misses       1405     1380      -25     
  Partials      460      460              
Flag Coverage Δ
integration 69.37% <ø> (ø)
unit 89.49% <89.16%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
exporter/datadogexporter/factory.go 82.97% <65.00%> (-0.54%) ⬇️
exporter/datadogexporter/metrics_translator.go 79.25% <66.66%> (+6.39%) ⬆️
exporter/datadogexporter/traces_exporter.go 90.24% <83.33%> (+1.67%) ⬆️
exporter/datadogexporter/metadata/metadata.go 89.70% <97.22%> (+28.92%) ⬆️
exporter/datadogexporter/config/config.go 87.50% <100.00%> (ø)
exporter/datadogexporter/metadata/ec2/ec2.go 47.50% <100.00%> (+17.50%) ⬆️
exporter/datadogexporter/metadata/gcp/gcp.go 100.00% <100.00%> (ø)
exporter/datadogexporter/metadata/host.go 77.14% <100.00%> (+1.38%) ⬆️
exporter/datadogexporter/metrics_exporter.go 100.00% <100.00%> (+37.50%) ⬆️
exporter/datadogexporter/translate_traces.go 89.56% <100.00%> (+1.12%) ⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ca2e9b...6b3b21b. Read the comment docs.

var defaultPrefixes = [3]string{"ip-", "domu", "ec2amaz-"}
var (
defaultPrefixes = [3]string{"ip-", "domu", "ec2amaz-"}
ec2TagPrefix = "ec2.tag."
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to match the prefix defined in #1899, see here

Copy link
Contributor

@KSerrania KSerrania left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change looks good to me overall, I left some questions & notes about tests

exporter/datadogexporter/factory.go Show resolved Hide resolved
exporter/datadogexporter/factory.go Show resolved Hide resolved
exporter/datadogexporter/metadata/gcp/gcp.go Show resolved Hide resolved
exporter/datadogexporter/metadata/metadata.go Outdated Show resolved Hide resolved
metrics := ilm.Metrics()
for k := 0; k < metrics.Len(); k++ {
md := metrics.At(k)
metricsArray := ilm.Metrics()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: these codecov warnings are due to the lack of unit tests for mapMetrics, they'll be fixed in another PR

Base automatically changed from master to main January 28, 2021 00:57
Copy link
Contributor

@KSerrania KSerrania left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on behalf of Datadog.

@mx-psi mx-psi marked this pull request as ready for review January 29, 2021 14:27
@mx-psi mx-psi requested a review from a team January 29, 2021 14:27
@mx-psi
Copy link
Member Author

mx-psi commented Jan 29, 2021

Marking as ready for review; it was approved by someone from Datadog. Note that CodeCov warnings due to lack of tests for mapMetrics will be covered in a future PR.

dyladan referenced this pull request in dynatrace-oss-contrib/opentelemetry-collector-contrib Jan 29, 2021
@bogdandrutu bogdandrutu merged commit 4cc0d2b into open-telemetry:main Jan 29, 2021
@mx-psi mx-psi mentioned this pull request Feb 2, 2021
@mx-psi mx-psi deleted the mx-psi/hostname-on-running branch February 10, 2021 13:24
pmatyjasek-sumo referenced this pull request in pmatyjasek-sumo/opentelemetry-collector-contrib Apr 28, 2021
…metrics (#2023)

* Set hostname to running metrics from resource attributes if available

Running metrics can be duplicate in this setup but this is fine since
they are gauges.

* Use resource metadata to complet host metadata

If enabled (it is by default) the resource attributes from the first
payload from either metrics or traces will be used to populate host
metadata. If there are fields missing these will be filled in by the
exporter (this preserves behavior when not using
k8s_tagger/resourcedetection).

* Add tests for new metadata behavior

* Fix go lint

* Remove unnecessary log messages

* Improve GCP info from attributes

Most logic is taken from here: https://github.com/DataDog/datadog-agent/blob/491c309e374ed5c7f1b385b347d5f9b5ac72c2b6/pkg/util/gce/gce_tags.go#L71-L101

* Use "system" host tags for EC2 tags
This is what the Datadog Agent does https://github.com/DataDog/datadog-agent/blob/491c309e374ed5c7f1b385b347d5f9b5ac72c2b6/pkg/metadata/host/host_tags.go#L97-L139

* Add additional test for GCP hostname from attributes

* Add test for metadata.Pusher function

* Improve test coverage

* Set EC2 tags in OTel field instead of System field
This avoids clashes with the Datadog Agent
ljmsc referenced this pull request in ljmsc/opentelemetry-collector-contrib Feb 21, 2022
…rnal/tools (#2023)

* Bump github.com/golangci/golangci-lint in /internal/tools

Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.40.1 to 1.41.1.
- [Release notes](https://github.com/golangci/golangci-lint/releases)
- [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md)
- [Commits](golangci/golangci-lint@v1.40.1...v1.41.1)

---
updated-dependencies:
- dependency-name: github.com/golangci/golangci-lint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Auto-fix go.sum changes in dependent modules

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aneurysm9 <Aneurysm9@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants