Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/snowflake] add metrics #15296

Merged
merged 548 commits into from
Dec 12, 2022

Conversation

shalper2
Copy link
Contributor

Description:
Defined metrics in metadata.yaml and ran mdatagen to create the metadata package for use in receiver component. Metrics are designed for parity with existing smartagent. Dimensions have been translated into attributes with the same column values associated. The smartagent definition does not include data types so care was taken to apply applicable types to each metric and attribute field.

Link to tracking Issue: 14754

Testing:
simple functional testing

Documentation:
generated using mdatagen

@shalper2 shalper2 requested a review from a team October 19, 2022 20:33
@codeboten codeboten added the receiver/snowflake Snowflake receiver label Oct 20, 2022
receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
Copy link
Member

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay. I was pretty swamped recently. Will be responding quickly going forward

receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
@github-actions
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added Stale and removed Stale labels Nov 26, 2022
@runforesight
Copy link

runforesight bot commented Dec 5, 2022

Foresight Summary

    
Major Impacts

build-and-test-windows duration(3 seconds) has decreased 38 minutes 56 seconds compared to main branch avg(38 minutes 59 seconds).
View More Details

⭕  build-and-test-windows workflow has finished in 3 seconds (38 minutes 56 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
windows-unittest-matrix -     🔗  N/A See Details
windows-unittest -     🔗  N/A See Details

✅  tracegen workflow has finished in 1 minute 2 seconds (2 minutes 19 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
build-dev -     🔗  N/A See Details
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  check-links workflow has finished in 1 minute 20 seconds (1 minute 36 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

✅  changelog workflow has finished in 1 minute 37 seconds (3 minutes 13 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 3 minutes 23 seconds (5 minutes 45 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

❌  build-and-test workflow has finished in 35 minutes 23 seconds (27 minutes 9 seconds less than main branch avg.) and finished at 9th Dec, 2022. 2 jobs failed.


Job Failed Steps Tests
unittest-matrix (1.19, internal) -     🔗  ✅ 592  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, internal) -     🔗  ✅ 592  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, processor) -     🔗  ✅ 1465  ❌ 0  ⏭ 0    🔗 See Details
correctness-metrics -     🔗  ✅ 2  ❌ 0  ⏭ 0    🔗 See Details
correctness-traces -     🔗  ✅ 17  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, extension) -     🔗  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, processor) -     🔗  ✅ 1465  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, extension) -     🔗  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-0) -     🔗  ✅ 2533  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-0) -     🔗  ✅ 2533  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, exporter) -     🔗  ✅ 2417  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, other) -     🔗  ✅ 4205  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-1) -     🔗  ✅ 1854  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, exporter) -     🔗  ✅ 2417  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-1) -     🔗  ✅ 1854  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, other) -     🔗  ✅ 4205  ❌ 0  ⏭ 0    🔗 See Details
integration-tests -     🔗  ✅ 57  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details
checks -     🔗  N/A See Details
check-collector-module-version -     🔗  N/A See Details
check-codeowners -     🔗  N/A See Details
lint-matrix (receiver-0) -     🔗  N/A See Details
lint-matrix (receiver-1) -     🔗  N/A See Details
lint-matrix (processor) -     🔗  N/A See Details
lint-matrix (exporter) -     🔗  N/A See Details
lint-matrix (extension) -     🔗  N/A See Details
lint-matrix (internal) -     🔗  N/A See Details
lint-matrix (other) -     🔗  N/A See Details
build-examples -     🔗  N/A See Details
lint -     🔗  N/A See Details
unittest (1.19) -     🔗  N/A See Details
unittest (1.18) -     🔗  N/A See Details
cross-compile (darwin, amd64) -     🔗  N/A See Details
cross-compile (darwin, arm64) -     🔗  N/A See Details
cross-compile (linux, 386) -     🔗  N/A See Details
cross-compile (linux, amd64) -     🔗  N/A See Details
cross-compile (linux, arm) -     🔗  N/A See Details
cross-compile (linux, arm64) -     🔗  N/A See Details
cross-compile (linux, ppc64le) -     🔗  N/A See Details
cross-compile (windows, 386) -     🔗  N/A See Details
cross-compile (windows, amd64) -     🔗  N/A See Details
build-package (deb) Test deb package     🔗  N/A See Details
build-package (rpm) Test rpm package     🔗  N/A See Details
windows-msi -     🔗  N/A See Details
publish-dev -     🔗  N/A See Details
publish-check -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  load-tests workflow has finished in 8 minutes 27 seconds (7 minutes 32 seconds less than main branch avg.) and finished at 9th Dec, 2022.


Job Failed Steps Tests
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 19  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

Copy link
Member

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase and run make generate. I recently changed the generated documentation

receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
dmitryax and others added 18 commits December 6, 2022 08:09
…g Group (open-telemetry#15678)

* Add changes for Log Retention

* Add comments to the code

* Update README

* Update CHANGELOG

* Add Log Retention config option to awscloudwatchlogsxporter
…estamps (open-telemetry#16062)

I think this issue was introduced in d31a5c3.

I [noticed a memory leak][1] in DataDog exporter but after a lot of
debugging, it turned out that the OpenTelemetry gateway was receiving
invalid timestamps:

```
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
    Trace ID       : 0000000000000000c20a2b82c179228a
    Parent ID      :
    ID             : 7c3415ed370f1777
    Name           : [redacted]
    Kind           : SPAN_KIND_SERVER
    Start time     : 2200-11-16 22:32:41.14035456 +0000 UTC
    End time       : 2200-11-16 22:33:02.68735456 +0000 UTC
```

The year is almost 100 years in the future.

Before we send a span to the gateway, it goes through an OpenTelemetry
collector sidecar. We use Zipkin V1 endpoint to send the spans to the
sidecar.

In the same pod, we're also using Zipkin V2 which did not have any
issues.

The problem itself: zipkin uses microseconds. This has to be multipled
with 1e3 to get nanoseconds, not with 1e6.

[1]: open-telemetry#15720
…ry#16097)

Between v0.64.0 and v0.66.0, the new metrics are not enabled by default.
…pen-telemetry#16142)

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Update the instructions to run the E2E tests to use the Make target.

I chose to use the e2e-test Make target since it will guarantee the local Collector binary is generated and up to date. I also removed the output screenshots since I think they likely don't provide much value.
…ady protected by proc mu (open-telemetry#16103)

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>

Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
)

* Add ContextStatements config

* Add high-level context

* respond to feedback

* Fix merge

* Adjust ParserCollection

* Add example usage temporarily

* apply feedback

* Split into individual files

* Add spanevent and metric contexts

* revert logs processor example

* Fix NewParserCollection

* Fix lint

* Fix checks

* Refactor ParserCollection to have a base

* Update interface to handle context.Context

* cleanup var names

* Fix case statement

* Implement UnmarshalText for ContextID

* Switch to consumer.Signal

* Fix lint

* Update processor/transformprocessor/internal/common/logs.go

Co-authored-by: Evan Bradley <github@evanbradley.org>

* Update processor/transformprocessor/internal/common/metrics.go

Co-authored-by: Evan Bradley <github@evanbradley.org>

* apply feedback

* Add back Options for each parser

Co-authored-by: Evan Bradley <github@evanbradley.org>
…telemetry#16026)

- Make the receiver configurable to be aware of the root filesystem, for Linux only.
- Give the ability to the filesystem scraper to translate mountpoints between host and container
mountpoint attribute emitted without the root path prefix, i.e. from the host's perspective
- No longer requires specification of where to find the mountinfo file through env var in order to avoid errors.
- All scrapers can see RootPath - useful for other scrapers e.g. pagingscraper as discussed in Ability to scrape filesystem host metrics from a container
- Set the environment variables for the user if they are not set.
- Validate that the root_path aligns with the gopsutil env vars (if they are previously set).
…lemetry#16168)

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
* SNMP Receiver added integration tests

Co-authored-by: Stefan Kurek <stefan.kurek@bluemedora.com>
Adds code owners without write permissions as reviewers to a PR when the corresponding component label is added to the PR.
…ry#16177)

migrate off of deprecated google cloud generated client
@shalper2 shalper2 requested a review from mx-psi as a code owner December 6, 2022 14:33
@github-actions github-actions bot added the cmd/configschema configschema command label Dec 6, 2022
receiver/snowflakereceiver/documentation.md Outdated Show resolved Hide resolved
.chloggen/ottl-merge-maps.yaml~merged Outdated Show resolved Hide resolved
@shalper2 shalper2 closed this Dec 7, 2022
@shalper2 shalper2 deleted the snowflake_add_metrics branch December 8, 2022 01:54
@shalper2 shalper2 restored the snowflake_add_metrics branch December 8, 2022 01:58
@shalper2 shalper2 reopened this Dec 8, 2022
cmd/otelcontribcol/go.mod Outdated Show resolved Hide resolved
@dmitryax dmitryax merged commit 6d9669b into open-telemetry:main Dec 12, 2022
@shalper2 shalper2 deleted the snowflake_add_metrics branch December 12, 2022 14:30
@plantfansam plantfansam mentioned this pull request Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.