[exporter/elasticsearch] add missing scope info in span/log attributes #27288

JaredTan95 · 2023-10-01T14:16:59Z

Description:

Link to tracking Issue: #27282

Testing:

Documentation:

Signed-off-by: Jared Tan <jian.tan@daocloud.io>

exporter/elasticsearchexporter/model.go

pengweiqhca · 2023-10-18T00:58:31Z

Logs missing also.

**Description:** Remove the parquet exporter **Link to tracking Issue:** Fixes open-telemetry#27284

…etry#27235) **Description:** Functions to modify matched text during replacement can now be passed as optional arguments to the following Editors: - replace_pattern - replace_all_patterns - replace_match - replace_all_matches **Documentation:** https://github.com/rnishtala-sumo/opentelemetry-collector-contrib/blob/ottl-replace-pattern/pkg/ottl/ottlfuncs/README.md#replace_pattern **Issue:** Resolves open-telemetry#22787

**Description:** Add validation of protocol name and be case insensitive **Link to tracking Issue:** **Testing:** Unit Tests **Documentation:** N/A --------- Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

…ion (open-telemetry#27480)

open-telemetry#27481) Linked to open-telemetry#25190

…r.Factory and pass checkapi (open-telemetry#27077) Related to: open-telemetry#26304

…oudwatchlogs exporter (open-telemetry#26692) Adds support to the to parallelism in the awscloudwatchlogs exporter by leveraging the [exporter helper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md). In this PR, we are adding support to the `num_consumers` configuration in the `sending_queue`. This will allow users to specify the number of consumers that will consume from the sending_queue in parallel. It is possible and straightforward to use this approach because CloudWatch logs [no longer requires that you use a token to control access to the stream that you are writing to](https://aws.amazon.com/about-aws/whats-new/2023/01/amazon-cloudwatch-logs-log-stream-transaction-quota-sequencetoken-requirement/). You can write to the same stream in parallel. To achieve this, this PR does the following: * Create Pusher that is able to push to multiple streams at the same time. * Move lifecycle of the Pusher to the function that is used to consume from the sending queue. This allows you to safely send to multiple streams at the same time without any resource contention since each call to consume logs will not share resources with others that are happening in parallel (one exception is the creation of log streams). Besides that I analyzed the code and removed other limitations: * locks that were not necessary * Limiter that was used to limit the number of requests per stream to 5 per second. [The TPS is much higher now and is per account.](https://aws.amazon.com/about-aws/whats-new/2023/01/amazon-cloudwatch-logs-log-stream-transaction-quota-sequencetoken-requirement/) ** How to review this PR: ** The first 3 commits in this PR were used to refactor the code before making the real changes. Please use the commits to simplify the review process. **Link to tracking Issue:** open-telemetry#26360 **Testing:** - Unit tests were added. - Tested locally sending logs to cloudwatch logs. **Documentation:** Documentation was added describing the new parameters. --------- Signed-off-by: Raphael Silva <rapphil@gmail.com> Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>

…pen-telemetry#23447) Adding a feature: Making `maxBatchByteSize` a configurable parameter. This would allow users to adjust it based on the capabilities of their specific remote storage, offering more flexibility and potentially improving performance. Example: ```yaml exporters: prometheusremotewrite: endpoint: "https://my-cortex:7900/api/v1/push" max_batch_byte_size: 5000000 ``` Fixes open-telemetry#21911 **Testing:** <Describe what testing was performed and which tests were added.> Added `MaxBatchByteSize` to `TestLoadConfig(t *testing.T)` in `config_test.go` **Documentation:** <Describe the documentation added.> Added to `README.md`: - `max_batch_byte_size` (default = `3000000` -> `~2.861 mb`): Maximum size of a batch of samples to be sent to the remote write endpoint. If the batch size is larger than this value, it will be split into multiple batches. --------- Co-authored-by: Alex Boten <aboten@lightstep.com>

…pen-telemetry#27625) **Description:** v0.87.0 `exporterhelper.queue_settings.validate()` verifies the consumer setting for us. We no longer need to duplicate that validation. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#27624

…#27434) Incremental step towards decoupling the Reader struct. This also clarifies and corrects the behavior of the `ValidateOrClose` function.

This addresses https://pkg.go.dev/vuln/GO-2023-2102 Signed-off-by: Alex Boten <aboten@lightstep.com>

open-telemetry#27616) **Description:** This resource attribute was deprecated in prior release ( open-telemetry#27584) so I am removing it. **Link to tracking Issue:** open-telemetry#26487

…elemetry#27569) **Description:** Adds an e2e test for the k8sobjectsreceiver to github workflow  **Link to tracking Issue:** [18395](open-telemetry#18395)

…#27622) **Description:** Remove stale go mod replace for datadog-agent and upgrade to the released version. **Link to tracking Issue:** Fixes open-telemetry#25119.

…try#27634) Remove translation dropping the `opencensus.resourcetype` resource attribute as it's not emitted by k8s receivers anymore. The last one was removed in open-telemetry#27616

**Documentation:** <Describe the documentation added.> Found a typo while reading the transform processor readme.

…open-telemetry#27272) **Description:** <Describe what has changed.>  Added support for Exporter Helper configuration. **Link to tracking Issue:** open-telemetry#24329 **Testing:** <Describe what testing was performed and which tests were added.> Added tests and manually tested with e2e scenarios --------- Co-authored-by: Ramachandran A G <ramacg@microsoft.com> Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com>

open-telemetry#27485) **Description:** <Describe what has changed.> add k8s.pod.qos_class optional resource attriute **Link to tracking Issue:** <Issue number if applicable> open-telemetry#27483 **Testing:** <Describe what testing was performed and which tests were added.> - updated unit tests **Documentation:** <Describe the documentation added.> - generated

fatsheep9146 · 2023-10-19T01:08:52Z

you seemed add a lot of unrelated commits @JaredTan95

JaredTan95 · 2023-10-19T01:10:54Z

you seemed add a lot of unrelated commits @JaredTan95

maybe it was caused by rebase. but don't mind, It will be squashed during the merge.

open-telemetry#27288) **Description:** <Describe what has changed.>  **Link to tracking Issue:** open-telemetry#27282 **Testing:** <Describe what testing was performed and which tests were added.> **Documentation:** <Describe the documentation added.> --------- Signed-off-by: Jared Tan <jian.tan@daocloud.io> Signed-off-by: Dominik Rosiek <drosiek@sumologic.com> Signed-off-by: Raphael Silva <rapphil@gmail.com> Signed-off-by: Alex Boten <aboten@lightstep.com> Signed-off-by: Pavol Loffay <p.loffay@gmail.com> Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com> Co-authored-by: Raj Nishtala <113392743+rnishtala-sumo@users.noreply.github.com> Co-authored-by: Dominik Rosiek <58699848+sumo-drosiek@users.noreply.github.com> Co-authored-by: Povilas Versockas <povilas.versockas@coralogix.com> Co-authored-by: Priyanshu Raj <55045459+rpriyanshu9@users.noreply.github.com> Co-authored-by: sakulali <sakulali@126.com> Co-authored-by: Raphael Philipe Mendes da Silva <rapphil@gmail.com> Co-authored-by: Anthony Mirabella <a9@aneurysm9.com> Co-authored-by: Yotam loewenbach <48534558+yotamloe@users.noreply.github.com> Co-authored-by: Alex Boten <aboten@lightstep.com> Co-authored-by: bryan-aguilar <46550959+bryan-aguilar@users.noreply.github.com> Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com> Co-authored-by: Jina Jain <jjain@splunk.com> Co-authored-by: Yang Song <songy23@users.noreply.github.com> Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com> Co-authored-by: Curtis Robert <92119472+crobert-1@users.noreply.github.com> Co-authored-by: Abhishek Saharn <102726227+asaharn@users.noreply.github.com> Co-authored-by: Ramachandran A G <ramacg@microsoft.com> Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com> Co-authored-by: Faith Chikwekwe <faithchikwekwe01@gmail.com> Co-authored-by: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Co-authored-by: Daniel Kuiper <44123852+kuiperda@users.noreply.github.com> Co-authored-by: Carlos Castro <ccastro@newrelic.com> Co-authored-by: Christian <calvarez@newrelic.com> Co-authored-by: ArchangelSDY <Archangel.SDY@gmail.com> Co-authored-by: Pavol Loffay <p.loffay@gmail.com> Co-authored-by: Paulo Janotti <pjanotti@splunk.com> Co-authored-by: Nathan Burke <n.burke@natbur.com> Co-authored-by: VihasMakwana <121151420+VihasMakwana@users.noreply.github.com> Co-authored-by: shalper2 <99686388+shalper2@users.noreply.github.com> Co-authored-by: OpenTelemetry Bot <107717825+opentelemetrybot@users.noreply.github.com> Co-authored-by: Martin Majlis <122797378+martin-majlis-s1@users.noreply.github.com> Co-authored-by: hovavza <147598197+hovavza@users.noreply.github.com>

add scope

c9c714d

Signed-off-by: Jared Tan <jian.tan@daocloud.io>

JaredTan95 requested review from a team and bogdandrutu October 1, 2023 14:17

github-actions bot assigned Aneurysm9 Oct 1, 2023

github-actions bot added the exporter/elasticsearch label Oct 1, 2023

crobert-1 approved these changes Oct 2, 2023

View reviewed changes

exporter/elasticsearchexporter/model.go Show resolved Hide resolved

JaredTan95 added 2 commits October 7, 2023 10:14

Merge branch 'main' into elasticsearch_exporter_add_scope

141ddad

Merge branch 'main' into elasticsearch_exporter_add_scope

675e3c0

JaredTan95 mentioned this pull request Oct 17, 2023

Add InstrumentationScope #27802

Closed

songy23 approved these changes Oct 18, 2023

View reviewed changes

atoulme and others added 19 commits October 18, 2023 23:11

[exporter/parquet] remove Parquet exporter (open-telemetry#27285)

d72f20f

**Description:** Remove the parquet exporter **Link to tracking Issue:** Fixes open-telemetry#27284

feat(syslogreceiver): validate protocol name (open-telemetry#27581)

407900a

**Description:** Add validation of protocol name and be case insensitive **Link to tracking Issue:** **Testing:** Unit Tests **Documentation:** N/A --------- Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

[chore] [coralogixexporter] Docs: Update docs to reflect new informat…

064af9b

…ion (open-telemetry#27480)

[processor/tailsampling] [chore][docs]: Fix typo in processor's README (

8fc10ab

open-telemetry#27481) Linked to open-telemetry#25190

[receiver/podman] rename struct and function to keep expected receive…

23ed615

…r.Factory and pass checkapi (open-telemetry#27077) Related to: open-telemetry#26304

[chore][pkg/stanza] Unexport reader's file name field (open-telemetry…

73d3955

…#27434) Incremental step towards decoupling the Reader struct. This also clarifies and corrects the behavior of the `ValidateOrClose` function.

[chore] update link to stack overflow (open-telemetry#27626)

c539979

[chore] bump go versions (open-telemetry#27631)

1682365

This addresses https://pkg.go.dev/vuln/GO-2023-2102 Signed-off-by: Alex Boten <aboten@lightstep.com>

[k8sclusterreceiver] remove opencensus.resourcetype resource attribute (

8276cc1

open-telemetry#27616) **Description:** This resource attribute was deprecated in prior release ( open-telemetry#27584) so I am removing it. **Link to tracking Issue:** open-telemetry#26487

[chore] Remove stale go mod replace for datadog-agent (open-telemetry…

0eecc7d

…#27622) **Description:** Remove stale go mod replace for datadog-agent and upgrade to the released version. **Link to tracking Issue:** Fixes open-telemetry#25119.

[chore] [exporter/signalfx] Remove redundant translation (open-teleme…

0981aa3

…try#27634) Remove translation dropping the `opencensus.resourcetype` resource attribute as it's not emitted by k8s receivers anymore. The last one was removed in open-telemetry#27616

[chore] Fix typo in README (open-telemetry#27639)

049dadf

**Documentation:** <Describe the documentation added.> Found a typo while reading the transform processor readme.

Merge branch 'main' into elasticsearch_exporter_add_scope

511c0f0

github-actions bot added the cmd/otelcontribcol otelcontribcol command label Oct 19, 2023

JaredTan95 changed the title ~~[exporter/elasticsearch] add missing scope info in span attributes~~ [exporter/elasticsearch] add missing scope info in span/log attributes Oct 19, 2023

JaredTan95 added 2 commits October 20, 2023 16:36

Merge branch 'main' into elasticsearch_exporter_add_scope

19b3231

Merge branch 'main' into elasticsearch_exporter_add_scope

af9c9c6

mx-psi approved these changes Oct 23, 2023

View reviewed changes

mx-psi merged commit 98da843 into open-telemetry:main Oct 23, 2023
83 checks passed

github-actions bot added this to the next release milestone Oct 23, 2023

JaredTan95 deleted the elasticsearch_exporter_add_scope branch October 23, 2023 15:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/elasticsearch] add missing scope info in span/log attributes #27288

[exporter/elasticsearch] add missing scope info in span/log attributes #27288

JaredTan95 commented Oct 1, 2023

pengweiqhca commented Oct 18, 2023

fatsheep9146 commented Oct 19, 2023

JaredTan95 commented Oct 19, 2023 •

edited

Loading

[exporter/elasticsearch] add missing scope info in span/log attributes #27288

[exporter/elasticsearch] add missing scope info in span/log attributes #27288

Conversation

JaredTan95 commented Oct 1, 2023

pengweiqhca commented Oct 18, 2023

fatsheep9146 commented Oct 19, 2023

JaredTan95 commented Oct 19, 2023 • edited Loading

JaredTan95 commented Oct 19, 2023 •

edited

Loading