Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[exporter/elasticsearch] Replace go-elasticsearch BulkIndexer with go-docappender #32359

Merged
merged 50 commits into from
May 9, 2024
Merged

[exporter/elasticsearch] Replace go-elasticsearch BulkIndexer with go-docappender #32359

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
1ecda8e
Use go-docappender bulk indexer
carsonip Apr 15, 2024
4889835
Remove go mod replace directive
carsonip Apr 17, 2024
d4caa07
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 17, 2024
e2c4373
Use runtime.NumCPU() as default numWorkers
carsonip Apr 17, 2024
f518aa5
Add changelog
carsonip Apr 17, 2024
aed3960
Describe defaults in readme
carsonip Apr 18, 2024
accd2fb
worker flush should log, not return
carsonip Apr 18, 2024
0784b18
Workers use receive only channel
carsonip Apr 18, 2024
40d8987
Add godoc
carsonip Apr 18, 2024
e4accdc
Make bulkIndexerPool private
carsonip Apr 18, 2024
3ceb029
Grammar
carsonip Apr 18, 2024
0fcddd2
Update changelog
carsonip Apr 18, 2024
dde9f6c
Use go-docappender v2, accept pipeline config
carsonip Apr 18, 2024
3ed5349
Use wg instead of errgroup
carsonip Apr 18, 2024
03a094f
Log error returned by Add and configure client
carsonip Apr 18, 2024
e256d82
Use go-docappender v2.0.1
carsonip Apr 18, 2024
7748722
Fix test loop variable
carsonip Apr 18, 2024
926d817
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 18, 2024
c6ca007
go mod tidy all
carsonip Apr 18, 2024
43a2d53
Add back missing return
carsonip Apr 18, 2024
ad790d7
Check for closed channel without zero check
carsonip Apr 18, 2024
b6642b2
Fix channel closed check
carsonip Apr 18, 2024
274b862
Fix import
carsonip Apr 18, 2024
0ae62df
Use new go-docappender
carsonip Apr 22, 2024
d1e4f4f
Use 429 instead of 500 in integration test
carsonip Apr 22, 2024
cf342f2
Use go-docappender fix to pass integrationtest
carsonip Apr 23, 2024
324efdc
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 23, 2024
057c673
Revert integration test change and wait for rebase
carsonip Apr 23, 2024
7519621
Use upcoming retry.retry_on_status config
carsonip Apr 23, 2024
533331c
Bump go-docappender, remove go mod replace
carsonip Apr 23, 2024
0861fae
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 23, 2024
13cc9f9
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 24, 2024
1b2a018
Log document level failures
carsonip Apr 24, 2024
13d35e3
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 3, 2024
6e163a7
Add back missing import
carsonip May 3, 2024
cbf5977
Bump go-docappender
carsonip May 3, 2024
ea7e5b0
go mod tidy
carsonip May 3, 2024
92ee227
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 6, 2024
d9886c1
Update changelog
carsonip May 6, 2024
fd85db1
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 6, 2024
69f8b59
Fix conflict in go.sum
carsonip May 6, 2024
3225676
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
e63d855
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
8bad25c
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
8106bfa
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
336b3c5
Update exporter/elasticsearchexporter/elasticsearch_bulk.go
carsonip May 9, 2024
fd1b4c7
Update exporter/elasticsearchexporter/elasticsearch_bulk.go
carsonip May 9, 2024
43ed76d
Use gzip.NoCompression for readability
carsonip May 9, 2024
2293b53
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 9, 2024
6402cd4
Remove compression level config
carsonip May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into go-docappender-bulkindexer
  • Loading branch information
@carsonip
carsonip committed May 3, 2024
commit 13d35e39950f38aa85551dcd13efeb06532f8d93
2 changes: 1 addition & 1 deletion cmd/configschema/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ require (
github.com/ardielle/ardielle-go v1.5.2 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.51.27 // indirect
github.com/aws/aws-sdk-go v1.51.32 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.11 // indirect
Expand Down
1 change: 1 addition & 0 deletions cmd/configschema/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion cmd/otelcontribcol/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,7 @@ require (
github.com/ardielle/ardielle-go v1.5.2 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.51.27 // indirect
github.com/aws/aws-sdk-go v1.51.32 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.11 // indirect
Expand Down
1 change: 1 addition & 0 deletions cmd/otelcontribcol/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion exporter/elasticsearchexporter/elasticsearch_bulk.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
"time"

"github.com/cenkalti/backoff/v4"
"github.com/elastic/go-docappender/v2"
elasticsearch7 "github.com/elastic/go-elasticsearch/v7"
"go.uber.org/zap"

Expand All @@ -30,7 +29,7 @@

type esBulkIndexerCurrent = bulkIndexerPool

type esBulkIndexerItem = docappender.BulkIndexerItem

Check failure on line 32 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (exporter-1) 

undefined: docappender

Check failure on line 32 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (cmd-0) 

undefined: docappender

// clientLogger implements the estransport.Logger interface
// that is required by the Elasticsearch client for logging.
Expand Down Expand Up @@ -197,7 +196,7 @@
pool.wg.Add(numWorkers)

for i := 0; i < numWorkers; i++ {
bi, err := docappender.NewBulkIndexer(docappender.BulkIndexerConfig{

Check failure on line 199 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (exporter-1) 

undefined: docappender

Check failure on line 199 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (exporter-1) 

undefined: docappender

Check failure on line 199 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (cmd-0) 

undefined: docappender

Check failure on line 199 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (cmd-0) 

undefined: docappender
Client: client,
MaxDocumentRetries: maxDocRetry,
CompressionLevel: 0,
carsonip marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -267,7 +266,7 @@
}

type worker struct {
indexer *docappender.BulkIndexer

Check failure on line 269 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (exporter-1) 

undefined: docappender

Check failure on line 269 in exporter/elasticsearchexporter/elasticsearch_bulk.go

View workflow job for this annotation

GitHub Actions / govulncheck (cmd-0) 

undefined: docappender
items <-chan esBulkIndexerItem
flushInterval time.Duration
flushTimeout time.Duration
Expand Down
24 changes: 12 additions & 12 deletions exporter/elasticsearchexporter/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,23 +53,23 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.19.0 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.52.3 // indirect
github.com/prometheus/common v0.53.0 // indirect
github.com/prometheus/procfs v0.13.0 // indirect
go.elastic.co/apm/module/apmzap/v2 v2.6.0 // indirect
go.elastic.co/apm/v2 v2.6.0 // indirect
go.elastic.co/fastjson v1.3.0 // indirect
go.opentelemetry.io/collector v0.99.0 // indirect
go.opentelemetry.io/collector/config/configretry v0.99.0 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.99.0 // indirect
go.opentelemetry.io/collector/consumer v0.99.0 // indirect
go.opentelemetry.io/collector/extension v0.99.0 // indirect
go.opentelemetry.io/collector/receiver v0.99.0 // indirect
go.opentelemetry.io/otel v1.25.0 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.47.0 // indirect
go.opentelemetry.io/otel/sdk v1.25.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.25.0 // indirect
go.opentelemetry.io/collector v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/collector/config/configretry v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/collector/consumer v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/collector/extension v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/collector/receiver v0.99.1-0.20240502202854-2875844e3c35 // indirect
go.opentelemetry.io/otel v1.26.0 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.48.0 // indirect
go.opentelemetry.io/otel/sdk v1.26.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.26.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/net v0.23.0 // indirect
golang.org/x/net v0.24.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.19.0 // indirect
golang.org/x/text v0.14.0 // indirect
Expand Down
52 changes: 26 additions & 26 deletions exporter/elasticsearchexporter/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions exporter/elasticsearchexporter/integrationtest/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 0 additions & 2 deletions exporter/elasticsearchexporter/logs_exporter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ type elasticsearchLogsExporter struct {
model mappingModel
}

var retryOnStatus = []int{500, 502, 503, 504, 429}

func newLogsExporter(logger *zap.Logger, cfg *Config) (*elasticsearchLogsExporter, error) {
if err := cfg.Validate(); err != nil {
return nil, err
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ require (
github.com/ardielle/ardielle-go v1.5.2 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.51.27 // indirect
github.com/aws/aws-sdk-go v1.51.32 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.11 // indirect
Expand Down
1 change: 1 addition & 0 deletions go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.