Add section on how to build an otelcol dist #1234

jpkrohling · 2020-10-09T14:04:19Z

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

Description: Added readme section with links to resources on how to create custom OpenTelemetry Collector distributions.

Link to tracking Issue: open-telemetry/opentelemetry-collector#213

Testing: n/a

Documentation: n/a

README.md

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

owais · 2020-10-12T16:42:18Z

README.md

+## Creating your own OpenTelemetry Collector distribution
+
+A guide on how to create your own distribution is available in this blog post: 
+["Building your own OpenTelemetry Collector distribution"](https://medium.com/p/42337e994b63)


I may be wrong but isn't medium content sometimes locked behind a paywall now? Not sure if a freely available project both as in beer and speech should rely on extended documentation that might not be accessible to everyone.

Nothing against medium or content they host. Just not sure if OpenTelemetry as a product should direct users to content/documentation/guides that they might not be able to read.

Great article BTW @jpkrohling

Oh, I see this is the official OpenTelemetry blog on medium. I suppose this decision has already been taken then by the maintainers and we are fine with publishin there? Please disregard if that's the case.

sometimes locked behind a paywall now

Each author/publication can choose to make money on Medium and have a paywall in front of individual posts, or to allow all users to have free access to the content, without any paywalls. I always publish articles without opting-in for the paywall, and I believe the OpenTelemetry publication also doesn't.

README.md

Updates #1234 Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

… add 'messaging.url' span attribute (#1234) * boto3sqs: Fix various issues * do not use 'otel' prefix for propagation keys to make propagation compatible with other SQS instrumentations like Node.Js Inject propergator.fields keys into the MessageAttributeNames argument for 'receive_message' calls to retreive the corresponding message attributes * add 'messaging.url' span attribute to SQS spans * add boto3sqs instrumentation to tox.ini to run tests in CI * add some basic unit tests * changelog * fix linting issues * unset instrumented flag on uninstrument

#### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes #35606

)  #### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes open-telemetry#35606

…atus (#35648)  #### Description Refactors the `extension/healthcheckv2extension/internal/status` into `pkg/status`. This exposes the aggregator to be used by other extensions to gather component status information using the `extension.StatusWatcher`.  #### Link to tracking issue Closes #34692  #### Testing Being it was a refactor and all the same tests provided coverage, no additional testing was added.  #### Documentation Added a `README.md` to the `pkg/status` to provide information on where this package can be used. --------- Co-authored-by: Matthew Wear <matthew.wear@gmail.com>

#### Description Update OTel mode to implementation to serialize log body into body.* fields  #### Link to tracking issue  #### Testing  #### Documentation

…rning error to avoid upstream retries (#35740)  #### Description Log metrics validation error instead of returning to avoid upstream retries  #### Link to tracking issue  #### Testing  #### Documentation

#### Description  #### Link to tracking issue #35741 Fixes Wrong comparison func implement in translator/prometheusremotewrite.  #### Testing  #### Documentation  Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>

#### Description Follow up from #35703. Draft starting the work on adding support for remote write 2.0 in the translation package. Adding support for translating gauges. This is first iteration and to keep the PR small * we don't handle duplicate metrics * only support gauges * don't handle other labels than metric name * don't handle exemplars * don't handle metadata  #### Link to tracking issue #33661 Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: David Ashpole <dashpole@google.com>

#### Description Follow [the contract](https://github.com/open-telemetry/opentelemetry-collector/blob/df3c9e38a80ccc3b14705462be2e2e51c628a3b3/receiver/doc.go#L10) for cloudflare receiver  #### Link to tracking issue Related #5909  #### Testing Added

#### Description Add a new receiver to handle systemd service state reporting as metrics.  #### Link to tracking issue #33532

… prefix conflict in OTel mapping mode (#35651)  #### Description Metric names should be flattened and exported as is, even when one metric name is a prefix of another. Same for attributes for all logs, metrics and traces.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Version of MySQL older than 8.0.22 use the `show slave status` vs `show replica status` view to retrieve replication metrics. This allows for both the newer & older views to work and retrieve data for the two actual replica metrics available (`mysql.replica.time_behind_source` & `mysql.replica.sql_delay`).  #### Link to tracking issue Fixes #35217  #### Testing Manual testing against MySQL versions 8.0, 8.0.37, & MariaDB Ver 15.1 Distrib 10.5.26-MariaDB  #### Documentation None needed

#### Description Adds initial implementation of ntpreceiver.  #### Link to tracking issue #34375

…only (#35839)  #### Description Make OTel mapping mode use RequireDataStream in docappender, which means it will only send to data streams. This prevents auto creating regular indices in OTel mapping mode due to a race condition in Elasticsearch where otel-data index templates are not ready.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Enable gzip compression by default, at hardcoded level BestSpeed. To disable compression, set `compression` to `none`.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Fixes an issue where the Kafka receiver would block on shutdown. There was an earlier fix for this issue [here](#32720). This does solve the issue, but it was only applied to the traces receiver, not the logs or metrics receiver. The issue is this go routine in the `Start()` functions for logs and metrics: ```go go func() { if err := c.consumeLoop(ctx, metricsConsumerGroup); err != nil { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` The `consumeLoop()` function returns a `context.Canceled` error when `Shutdown()` is called, which is expected. However `componentstatus.ReportStatus()` blocks while attempting to report this error. The reason/bug for this can be found [here](open-telemetry/opentelemetry-collector#9824). The previously mentioned PR fixed this for the traces receiver by checking if the error returned by `consumeLoop()` is `context.Canceled`: ```go go func() { if err := c.consumeLoop(ctx, consumerGroup); !errors.Is(err, context.Canceled) { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` Additionally, this is `consumeLoop()` for the traces receiver, with the logs and metrics versions being identical: ```go func (c *kafkaTracesConsumer) consumeLoop(ctx context.Context, handler sarama.ConsumerGroupHandler) error { for { // `Consume` should be called inside an infinite loop, when a // server-side rebalance happens, the consumer session will need to be // recreated to get the new claims if err := c.consumerGroup.Consume(ctx, c.topics, handler); err != nil { c.settings.Logger.Error("Error from consumer", zap.Error(err)) } // check if context was cancelled, signaling that the consumer should stop if ctx.Err() != nil { c.settings.Logger.Info("Consumer stopped", zap.Error(ctx.Err())) return ctx.Err() } } } ``` This does fix the issue, however the only error that can be returned by `consumeLoop()` is a canceled context. When we create the context and cancel function, we use `context.Background()`: ```go ctx, cancel := context.WithCancel(context.Background()) ``` This context is only used by `consumeLoop()` and the cancel function is only called in `Shutdown()`. Because `consumeLoop()` can only return a `context.Canceled` error, this PR removes this unused code for the logs, metrics, and traces receivers. Instead, `consumeLoop()` still logs the `context.Canceled` error but it does not return any error and the go routine simply just calls `consumeLoop()`. Additional motivation for removing the call to `componentstatus.ReportStatus()` is the underlying function called by it, `componentstatus.Report()` says it does not need to be called during `Shutdown()` or `Start()` as the service already does so for the given component, [comment here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/component/componentstatus/status.go#L21-L25). Even if there wasn't a bug causing this call to block, the component still shouldn't call it since it would only be called during `Shutdown()`.  #### Link to tracking issue Fixes #30789  #### Testing Tested in a build of the collector with these changes scraping logs from a Kafka instance. When the collector is stopped and `Shutdown()` gets called, the receiver did not block and the collector stopped gracefully as expected.

#### Link to tracking issue Fixes #40476 based on open-telemetry/semantic-conventions#2319 --------- Signed-off-by: odubajDT <ondrej.dubaj@dynatrace.com> Co-authored-by: Antoine Toulme <atoulme@splunk.com>

Update US1 + US2 AWS PrivateLink endpoint according to our website: https://coralogix.com/docs/integrations/aws/aws-privatelink/aws-privatelink/  #### Description  #### Link to tracking issue Fixes  #### Testing  #### Documentation

#### Description  #### Link to tracking issue Fixes  #### Testing  #### Documentation  --------- Co-authored-by: Antoine Toulme <atoulme@splunk.com>

…te metric (#42158)  #### Description The PR adds changes to extract metrics pertaining to connections per async state. See the related issue for more details.  #### Link to tracking issue Relates #41886  #### Testing Unit and integration tests for the scraper have been extended with data for new metrics.  #### Documentation Documentation generated by mdatagen has been added.  --------- Co-authored-by: Antoine Toulme <atoulme@splunk.com>

…2299)  #### Description Adds `service.instance.id` resource attribute to all metrics and logs emitted by the SQL Server receiver. The attribute format is `<host>:<port>` which provides a unique identifier for each SQL Server instance. Key implementation details: - Compute service instance ID once during scraper initialization for performance - Handle various connection string formats using github.com/microsoft/go-mssqldb library's msdsn.Parse() - Replace localhost/127.0.0.1 with actual hostname from os.Hostname() - Set default port to 1433 when not specified - Instance name is NOT included in service.instance.id (already captured in sqlserver.instance.name)  #### Link to tracking issue Fixes #41894  #### Testing - Added comprehensive unit tests for service instance ID computation covering all edge cases - Added integration tests verifying the attribute is set correctly for both metrics and logs - Tests cover various configuration scenarios: - Explicit server and port configuration - DataSource connection strings with different formats - Localhost replacement scenarios - Default port handling - Named instances  #### Documentation - Updated metadata.yaml to include the new service.instance.id resource attribute - Added inline documentation for the computeServiceInstanceID function  --------- Co-authored-by: Curtis Robert <crobert@splunk.com>

@atoulme

#### Description When running commands like `make fmt` from a subdirectory, `INTEGRATION_TEST_FILES` may resolve to an empty list. In this case, the `INTEGRATION_TESTS` assignment ends up executing `cat` with no input files, which causes it to hang.  #### Link to tracking issue Improves #42366 @atoulme @pjanotti, please take a look as you have been involved in #42369 Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

#### Description remove flaky test (shouldn't impact code coverage)  #### Link to tracking issue Fixes #42431  #### Testing removes a test, none needed  #### Documentation none needed

#### Description When writing profiling data into Elasticsearch, some errors are expected. So we do not want to flood logging with these errors.  #### Link to tracking issue Fixes #38598  #### Testing  #### Documentation  --------- Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

If profiling data is processed, also populate profiling-hosts with resource attribute information of the host.  #### Description  #### Link to tracking issue Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

#### Description As seen in #42475 (and potentially others), certain dependencies may bump gomod to specific version. setting gomod to go 1.24.0 shouldn't impact anything more than setting go 1.23.0 on previous versions did.  #### Link to tracking issue Prerequisite for #42475  #### Testing existing tests  #### Documentation none needed

…ent Hub SDK (#42034)  #### Description This is an enhancement to the `azureeventhubreceiver` to allow for the optional usage of the new `Azure SDK` via a feature flag. This PR abstracts the logic so that the existing functionality works the same as it currently does with the option for a user to opt in to the using the new SDK. There are multiple new features that would be very beneficial to implement but for this initial update I kept the functionality and existing logic as close as I could to the initial implementation. 2 new configuration options were added specifically for when the feature flag is enabled. This is because the new library polls the event hub for new events compared to streaming. The polling settings are opinionated so they should be able to be configured by the user.  #### Link to tracking issue Fixes #40795  #### Testing Existing tests kept in place to verify no changes, and a couple of implementation specific tests were added to both the legacy library as well as the new `azeventhub` library.  #### Documentation A callout to the feature flag has been added to the readme as well as information about the 2 new configuration options.

#### Description remove references to Datadog/opentelemetry-mapping-go library. This library has been completely replaced by copies inside datadog-agent and will be archived soon.  #### Link to tracking issue Fixes internal issue  #### Testing migrate existing tests  #### Documentation chloggen

#### Description Do not reuse the free port allocation between tests  #### Link to tracking issue Fixes #42472 Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#### Description Small change to avoid sending data twice - might help reduce flakiness.  #### Link to tracking issue Related to #42438 Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…40220)  #### Description This PR adapts the k8scluster receiver to support retrieving resources from multiple namespaces. Due to the way the informer stores are maintained internally the required changes for this feature are more complex than anticipated: previously, as there was only the option to either observe all namespaces, or only one, there was one informer per GVK, whose store was used to regularly collect the metrics for the observed resources. Now, as the informers provided by the k8s library can only observe either one, or all namespaces, we can potentially have multiple informers per namespace for the same GVK. This is now done as a `map[GVK]map[string]informer.Store`, where the string represents the namespace.  #### Link to tracking issue Fixes #40089  #### Testing Added unit tests and a new e2e test  #### Documentation Adapted readme to describe the added option  --------- Signed-off-by: Florian Bacher <florian.bacher@dynatrace.com> Co-authored-by: Christos Markou <chrismarkou92@gmail.com> Co-authored-by: Antoine Toulme <atoulme@splunk.com>

…ted fixes (#42516)  #### Description  Updates #42458 to adapt to stretchr/testify#1427 #### Link to tracking issue Fixes #42486 Fixes #42411  #### Testing  Ran each test 100 times locally and made sure it passed

#### Description  Includes commit to which we have updated. #### Link to tracking issue Updates #42519

…rtity between Sarama and Franz-Go (#42507)  #### Description Add support for configuring Kafka topic auto-creation in the exporter by introducing the `allow_auto_topic_creation` option for producers. Franz-go and Sarama differ in their defaults for topic auto-creation. Sarama enables it by default, while Franz-go requires explicit configuration. This PR aligns behavior across both clients and gives users explicit control to avoid unexpected broker errors when topics are not pre-created.  #### Link to tracking issue Fixes #42468  #### Testing I tested the option using both Sarama and Franz-Go with both values (`true` and `false`) and the behaviour is the expected. --------- Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

…42497)  #### Description The supervisor will sometimes ignore certain Remote Config messages from the server. Specifically if the supervisor does not think the config is different. Instead it should still respond. In the event the supervisor does not think the config is different, it will send an applied status (functionally the config that was sent is being ran, so applied is appropriate here).  #### Link to tracking issue Fixes #42474  #### Testing Unit tests updated  #### Documentation  --------- Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>

#### Description When a request to the EventsAPI exceeds 10 MB, split it into N smaller requests, each no larger than 10 MB. If any of these requests fail, data processing stops and an at-most-once delivery approach is followed: - If at least one request succeeds: return a permanent error to prevent data duplication. - If no request succeeds: return the error from the first request, following the original error retry policy. This change ensures that large requests (≥ 10 MB) do not result in complete data loss. Some data may still be lost because processing halts after the first failure, but this can be improved later by returning the missing data via `consumererror.Trace` and derivatives, allowing the collector to retry (out of the scope for this PR).  #### Link to tracking issue Fixes #41782  #### Testing Tested were added to ensure that the at-most-once delivery approach is followed  #### Documentation

#### Description Introduce Explicit Bucket for Histogram configuration ```yaml histogram: max_size: 170 explicit_buckets: - matcher_pattern: "foo.*" buckets: [1, 10, 100] - matcher_pattern: "bar.*" buckets: [0.1, 0.5, 1] ``` If the metrics name matches one of the matcher pattern (sequential order), it will use the explicit bucket instead of exponential histogram  #### Link to tracking issue Fixes #41203  #### Testing Adding unit tests  #### Documentation Added in the readme  --------- Signed-off-by: clavinjune <24659468+clavinjune@users.noreply.github.com>

#### Description logs now report scope name and version  #### Link to tracking issue Fixes #42426  #### Testing added a new test `TestScopeNameAndVersion` in `receiver/k8seventsreceiver/k8s_event_to_logdata_test.go` Co-authored-by: Christos Markou <chrismarkou92@gmail.com>

#### Description This PR adds support for resource detection in Hetzner Cloud. I have not added the Hetzner cloud into semantic conventions, but I can open a PR for it.  #### Link to tracking issue Fixes #42476  #### Testing Added tests to cover most of the cases, and I personally tested on a Hetzner server. ``` 2025-09-03T17:31:23.666+0100 info service@v0.134.0/service.go:211 Starting otelcontribcol... {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}, "Version": "0.134.0-dev", "NumCPU": 8} 2025-09-03T17:31:23.666+0100 info extensions/extensions.go:41 Starting extensions... {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}} 2025-09-03T17:31:23.666+0100 info internal/resourcedetection.go:137 began detecting resource information {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics", "otelcol.signal": "metrics"} 2025-09-03T17:31:23.672+0100 info internal/resourcedetection.go:188 detected resource information {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics", "otelcol.signal": "metrics", "resource": {"cloud.availability_zone":"nbg1-dc3","cloud.provider":"hetzner","cloud.region":"eu-central","host.id":"30727666","host.name":"vm1"}} 2025-09-03T17:31:23.672+0100 info service@v0.134.0/service.go:234 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}} 2025-09-03T17:31:24.673+0100 info Metrics {"resource": {"service.instance.id": "f054fbb3-ad03-4aef-8c7a-1e38550cfec7", "service.name": "otelcontribcol", "service.version": "0.134.0-dev"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 3, "metrics": 5, "data points": 73} 2025-09-03T17:31:24.673+0100 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 Resource attributes: -> cloud.provider: Str(hetzner) -> host.id: Str(30727666) -> host.name: Str(vm1) -> cloud.region: Str(eu-central) -> cloud.availability_zone: Str(nbg1-dc3) ```  #### Documentation Updated the `README.md` with this new detector.  --------- Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

…DME (#42547)  #### Description It would be nice for the users to see a conversion example directly in README, so they have a better understanding about how the conversion works.  #### Link to tracking issue Fixes #42512 Signed-off-by: odubajDT <ondrej.dubaj@dynatrace.com>

#### Description  Updates #42458 to adapt to stretchr/testify#1427

#### Description pins now-removed component announcement links to the tagged version where the components were announced  #### Link to tracking issue Fixes https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/17561312676/job/49878257899?pr=42564  #### Testing  #### Documentation

#### Description Add `S3PartitionTimezone` config value to be able to change the timezone used for time-based partitioning.  #### Link to tracking issue Resolves #42319.  #### Testing Updated existing tests to include time location.  #### Documentation Added new config value to README.  --------- Co-authored-by: Antoine Toulme <antoine@toulme.name>

Removing deprecated zorkian codepath. Previously added a warning: #42165  #### Link to tracking issue (#17373)  #### Testing  #### Documentation

#### Description By default enables rate limiting in cx exporter  #### Link to tracking issue Fixes #42552  #### Testing  #### Documentation  Co-authored-by: Antoine Toulme <atoulme@splunk.com> Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>

….16.0 to v1.15.0 (#42607)  #### Description [PR bumping dep](#42099) [gosnowflake issue](snowflakedb/gosnowflake#1533)  #### Link to tracking issue Fixes #42605

jpkrohling requested a review from a team October 9, 2020 14:04

bogdandrutu reviewed Oct 12, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

bogdandrutu changed the title ~~Added section on how to build an otelcol dist~~ Add section on how to build an otelcol dist Oct 12, 2020

jpkrohling added 2 commits October 12, 2020 17:31

Added section on how to build an otelcol dist

3166666

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

s/OpenTracing/OpenTelemetry

acf4d51

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

jpkrohling force-pushed the jpkrohling/add-link-to-blog-post-on-building-custom-otelcol branch from 73f9801 to acf4d51 Compare October 12, 2020 15:31

owais reviewed Oct 12, 2020

View reviewed changes

anuraaga reviewed Oct 13, 2020

View reviewed changes

README.md Show resolved Hide resolved

owais approved these changes Oct 13, 2020

View reviewed changes

anuraaga approved these changes Oct 14, 2020

View reviewed changes

bogdandrutu merged commit a987792 into open-telemetry:master Oct 14, 2020

kisieland referenced this pull request in kisieland/opentelemetry-collector-contrib Mar 16, 2021

Use cached tools instead of uploading to the artifacts (#2475)

9b2660f

Updates #1234 Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

ljmsc referenced this pull request in ljmsc/opentelemetry-collector-contrib Feb 21, 2022

Move asyncronous metrics state helper out of apimetric/metrictes (#1234)

f60f51d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 •

edited

Loading

Uh oh!

owais Oct 12, 2020

Uh oh!

jpkrohling Oct 12, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

Conversation

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owais Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

jpkrohling Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

owais Oct 12, 2020 •

edited

Loading

jpkrohling Oct 12, 2020 •

edited

Loading