[Exporter/OpenSearch] OpenSearch exporter setup #23819

MitchellGale · 2023-06-28T21:16:24Z

Description:
Adding initial set-up for OpenSearch exporter addition. Future PRs will include adding functionality to the exporter, code coverage and e2e tests.

Broken up for easier review.

New component proposal.

Link to tracking Issue:
Start of resolution of #7905.

Testing:
Will come in future PR to 80+% coverage.

Break-up of #23045

…request.yaml, other.yaml

…config.go

OpenSearch Exporter initial PR

codeboten · 2023-06-28T21:33:35Z

Pinging @Aneurysm9 for the initial review

cmd/otelcontribcol/builder-config.yaml

exporter/opensearchexporter/config.go

exporter/opensearchexporter/factory.go

.github/dependabot.yml

exporter/opensearchexporter/README.md

Aneurysm9 · 2023-07-07T14:23:26Z

exporter/opensearchexporter/README.md

+- `flush`: Event bulk buffer flush settings
+    - `bytes` (default=5242880): Write buffer flush limit.
+    - `interval` (default=30s): Write buffer time limit.
+- `retry`: Event retry settings


This should probably use the exporterhelper.RetrySettings type to provide consistency of configuration across exporters.

These values are used to create a backoff function for the OpenSearch client.
We'll check if exporterhelper.RetrySettings can be used here.

The config struct can be changed to use exporterhelper.RetrySettings but the implied behaviour may be different.

exporterhelper.WithRetry retries a batch of trace records, right? OpenSearch supports retrying one failed document from a bulk indexing request.

Would it be more appropriate to log documents that OpenSearch failed to index and move on? Currently they are added back to the bulk indexer queue to be tried later.

You can use the consumererror.Traces type to return an error containing the spans that failed to export. The exporterhelper error handler will unwrap the error, extract the spans, and replace the request with one containing just the failed spans.

exporter/opensearchexporter/README.md

exporter/opensearchexporter/factory.go

…n OpenSearch readme.

…in OpenSearch readme.

…orter-setup

MaxKsyunz · 2023-07-18T01:54:47Z

@Aneurysm9 based on your feedback and how opensearch-go client is set up, I'd like to do the following:

Use HTTPClientSettings in exporter's configuration. Map these settings to opensearch-go client's config.
Limit OpenSearch client to one BulkIndexer worker. A worker will be created and shut down for each batch of trace data.
Use RetrySettings but retry only if the whole batch fails.

This way the exporter will not use internal queue management done by opensearch-go and use existing opentel-collector data structures and code where possible.

What do you think?

Remove exporter-specific settings and use confighttp.HTTPClientSettings and exporterhelper.RetrySettings Update README to reflect the above. Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

Aneurysm9

LGTM.

I've still got some questions around the retry and authN logic, but I think we can deal with those on the implementation PRs when they're more concrete conversations.

Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

# Conflicts: # .github/dependabot.yml

dependabot.yml reached maximum limit so a package is getting dropped but CI will fail without this change. See open-telemetry#19410 Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

MaxKsyunz · 2023-07-21T01:55:12Z

GHAs failed on a known flaky test. See #21094

Is there a way to re-run it?

fullammo · 2023-08-22T13:17:21Z

Hi! can we enable this exporter for logs as well?

steevex35 · 2023-09-28T10:56:08Z

Hi!
This exporter can be used for logs ?
the OpensearchExporter readme.md mentions only trace pipeline.
thank you

MitchellGale and others added 21 commits June 25, 2023 14:00

Added space

31742d8

Added MitchellGale to CODEOWNERS

cb37238

Added OpenSearchExporter builder-config.yaml.

89dca47

Added OpenSearchExporter go.mod otel contrib col.

1928c8f

Added MakeFile for OpenSearchExporter.

4a1672b

Added OpenSearchExporter for versions.yaml

aa474de

Added OpenSearchExporter for .golangci.yml, bug_report.yaml, feature_…

8758cba

…request.yaml, other.yaml

Reverted changes in go.mod

e355bd2

Reverted MakeFile space.

969fc59

Reverted go.mod manual change.

b4aedd5

added OS config.go

b2ae742

added link for OS config.go

3d99174

Added go mod

f9dd5cd

Renamed Makefile

d1a6c93

Switched go mod go version from 1.20 to 1.19

29076e6

Ran make genotelcontribcol

4a35bea

Made with go mod changes and adding factory.go, and adding config to …

62dd060

…config.go

Fixed issues in config and factor go.

e286d18

go sum

8a51d24

Added YangDB and Max Ksyunz to code owners for OpenSearch exporter.

85d206f

Merge pull request #20 from Bit-Quill/mainCopy

fdc9b14

OpenSearch Exporter initial PR

MitchellGale requested review from a team and songy23 June 28, 2023 21:16

github-actions bot assigned codeboten Jun 28, 2023

github-actions bot added the cmd/otelcontribcol otelcontribcol command label Jun 28, 2023

Removed commented out code. Added @Aneurysm9 as code owner.

a078ad6

songy23 requested a review from Aneurysm9 June 28, 2023 23:20

MitchellGale changed the title ~~OpenSearch exporter setup~~ [Exporter/OpenSearch] OpenSearch exporter setup Jun 29, 2023

Aneurysm9 reviewed Jun 30, 2023

View reviewed changes

cmd/otelcontribcol/builder-config.yaml Outdated Show resolved Hide resolved

exporter/opensearchexporter/config.go Show resolved Hide resolved

exporter/opensearchexporter/factory.go Outdated Show resolved Hide resolved

MitchellGale added 4 commits July 5, 2023 14:50

Updated README.md

581ac14

Updated go mod and go sum

d4db8ab

Added change log for opensearch.

bb9972f

Updated dependabot.yml

993ce1a

MitchellGale force-pushed the Integ/OpenSearchExporter-setup branch from 943e2c1 to 993ce1a Compare July 5, 2023 22:48

MitchellGale added 5 commits July 5, 2023 15:49

Reverted changes in otelcontrib.

244fc29

Reverted changes in versions.

9fce291

Added new line at end of new component

d57a6af

Reverted file to golangci.yml

c778313

Added Opensearch exporter to excluded modules in versions.yaml.

c6eaf01

Aneurysm9 reviewed Jul 7, 2023

View reviewed changes

MitchellGale and others added 5 commits July 7, 2023 09:28

Adding defaults for Namespace and Dataset in factory.go as outlined i…

50db30a

…n OpenSearch readme.

swapped defaults for Namespace and Dataset in factory.go as outlined …

e240c46

…in OpenSearch readme.

Merge remote-tracking branch 'upstream/main' into Integ/OpenSearchExp…

e383557

…orter-setup

Update dependabot.

fc8d591

Update README.md with details on default num_workers

9575754

MaxKsyunz deleted the Integ/OpenSearchExporter-setup branch July 14, 2023 03:15

Address review feedback.

4d7d124

Remove exporter-specific settings and use confighttp.HTTPClientSettings and exporterhelper.RetrySettings Update README to reflect the above. Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

Aneurysm9 approved these changes Jul 20, 2023

View reviewed changes

Re-run fmt, mod tidy

a2ed3e8

Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

MaxKsyunz force-pushed the Integ/OpenSearchExporter-setup branch from 53a36a5 to a2ed3e8 Compare July 21, 2023 00:35

MaxKsyunz added 2 commits July 20, 2023 17:37

Merge branch 'main' into integ/OpenSearchExporter-setup

232e82d

# Conflicts: # .github/dependabot.yml

Re-run gendependabot

0f53242

dependabot.yml reached maximum limit so a package is getting dropped but CI will fail without this change. See open-telemetry#19410 Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

Aneurysm9 added the ready to merge Code review completed; ready to merge by maintainers label Jul 21, 2023

codeboten merged commit 675710d into open-telemetry:main Jul 21, 2023

github-actions bot added this to the next release milestone Jul 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Exporter/OpenSearch] OpenSearch exporter setup #23819

[Exporter/OpenSearch] OpenSearch exporter setup #23819

MitchellGale commented Jun 28, 2023 •

edited

Loading

codeboten commented Jun 28, 2023

Aneurysm9 Jul 7, 2023

MaxKsyunz Jul 7, 2023

MaxKsyunz Jul 17, 2023

Aneurysm9 Jul 21, 2023 •

edited

Loading

MaxKsyunz commented Jul 18, 2023

Aneurysm9 left a comment

MaxKsyunz commented Jul 21, 2023

fullammo commented Aug 22, 2023

steevex35 commented Sep 28, 2023

[Exporter/OpenSearch] OpenSearch exporter setup #23819

[Exporter/OpenSearch] OpenSearch exporter setup #23819

Conversation

MitchellGale commented Jun 28, 2023 • edited Loading

codeboten commented Jun 28, 2023

Aneurysm9 Jul 7, 2023

Choose a reason for hiding this comment

MaxKsyunz Jul 7, 2023

Choose a reason for hiding this comment

MaxKsyunz Jul 17, 2023

Choose a reason for hiding this comment

Aneurysm9 Jul 21, 2023 • edited Loading

Choose a reason for hiding this comment

MaxKsyunz commented Jul 18, 2023

Aneurysm9 left a comment

Choose a reason for hiding this comment

MaxKsyunz commented Jul 21, 2023

fullammo commented Aug 22, 2023

steevex35 commented Sep 28, 2023

MitchellGale commented Jun 28, 2023 •

edited

Loading

Aneurysm9 Jul 21, 2023 •

edited

Loading