[exporterqueue] Bare minimum frame of queue batcher + unit test. #11532
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-sili
changed the title
sfc-gh-sili
force-pushed
the
sili-queue-tiny
branch
from
7365e27 to
7be491d
Compare
sfc-gh-sili
changed the title
sfc-gh-sili
force-pushed
the
sili-queue-tiny
branch
from
7be491d to
a5458ed
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #11532 +/- ##
==========================================
- Coverage 91.46% 91.45% -0.01%
==========================================
Files 435 438 +3
Lines 23757 23827 +70
==========================================
+ Hits 21729 21791 +62
- Misses 1650 1658 +8
Partials 378 378 ☔ View full report in Codecov by Sentry. |
dmitryax
reviewed
Oct 24, 2024
bogdandrutu
reviewed
Oct 24, 2024
exporter/internal/queue/batcher.go
Outdated
| queue Queue[internal.Request] | ||
| maxWorkers int | ||
|
|
||
| exportFunc func(context.Context, internal.Request) error |
There was a problem hiding this comment.
Is this not Request.export?
There was a problem hiding this comment.
Good point.
This was modeled after consumeFunc used in consumer but it seems having req.Export() here is enough for the use case.
exporter/internal/queue/batcher.go
Outdated
| idxList []uint64 | ||
| } | ||
|
|
||
| type Batcher struct { |
There was a problem hiding this comment.
Comment on the struct please.
exporter/internal/queue/batcher.go
Outdated
| } | ||
| } | ||
|
|
||
| // If preconditions pass, flush() take an item from the head of batch list and exports it. |
There was a problem hiding this comment.
In golang, comments start with func name. flush
exporter/internal/queue/batcher.go
Outdated
Comment on lines
63
to
68
| go func() { | ||
| qb.flush(batchToFlush) | ||
| qb.stopWG.Done() | ||
| }() |
There was a problem hiding this comment.
This is a best practice to ensure function gets called on all possible "return paths". Even though now is a simple func may get more complicated.
Suggested change
| go func() { | |
| qb.flush(batchToFlush) | |
| qb.stopWG.Done() | |
| }() | |
| go func() { | |
| defer qb.stopWG.Done() | |
| qb.flush(batchToFlush) | |
| }() |
There was a problem hiding this comment.
Thanks for the advice!
exporter/internal/queue/batcher.go
Outdated
| return err | ||
| } | ||
|
|
||
| if qb.batchCfg.Enabled { |
There was a problem hiding this comment.
I think would be simpler if Batcher is an interface, and the "not enabled" and "enabled" are 2 different implementations.
There was a problem hiding this comment.
That's a good idea.
Implemented a special-case batcher called DisabledBatcher to handle the "not enabled" case. Let me know what you think of it.
sfc-gh-sili
force-pushed
the
sili-queue-tiny
branch
2 times, most recently
from
20a8bb1 to
fc658a0
Compare
sfc-gh-sili
force-pushed
the
sili-queue-tiny
branch
from
654fb15 to
e42ea72
Compare
bogdandrutu
approved these changes
Oct 25, 2024
exporter/internal/queue/batcher.go
Outdated
| stopWG sync.WaitGroup | ||
| } | ||
|
|
||
| func NewBatcher(batchCfg exporterbatcher.Config, queue Queue[internal.Request], maxWorkers int) Batcher { |
There was a problem hiding this comment.
Make it return error instead of panic.
dmitryax
approved these changes
Oct 25, 2024
dmitryax
pushed a commit
that referenced
this pull request
Oct 29, 2024
#### Description This PR follows #11532 and implements support for limited worker pool for queue batcher. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue #8122 #10368
djaglowski
pushed a commit
to djaglowski/opentelemetry-collector
that referenced
this pull request
Nov 21, 2024
…n-telemetry#11532) #### Description This PR is a bare minimum implementation of a component called queue batcher. On completion, this component will replace `consumers` in `queue_sender`, and thus moving queue-batch from a pulling model instead of pushing model. Limitations of the current code * This implements only the case where batching is disabled, which means no merge of splitting of requests + no timeout flushing. * This implementation does not enforce an upper bound on concurrency All these code paths are marked as panic currently, and they will be replaced with actual implementation in coming PRs. This PR is split from open-telemetry#11507 for easier review. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue open-telemetry#8122 open-telemetry#10368
djaglowski
pushed a commit
to djaglowski/opentelemetry-collector
that referenced
this pull request
Nov 21, 2024
…elemetry#11540) #### Description This PR follows open-telemetry#11532 and implements support for limited worker pool for queue batcher. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue open-telemetry#8122 open-telemetry#10368
HongChenTW
pushed a commit
to HongChenTW/opentelemetry-collector
that referenced
this pull request
Dec 19, 2024
…n-telemetry#11532) #### Description This PR is a bare minimum implementation of a component called queue batcher. On completion, this component will replace `consumers` in `queue_sender`, and thus moving queue-batch from a pulling model instead of pushing model. Limitations of the current code * This implements only the case where batching is disabled, which means no merge of splitting of requests + no timeout flushing. * This implementation does not enforce an upper bound on concurrency All these code paths are marked as panic currently, and they will be replaced with actual implementation in coming PRs. This PR is split from open-telemetry#11507 for easier review. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue open-telemetry#8122 open-telemetry#10368
HongChenTW
pushed a commit
to HongChenTW/opentelemetry-collector
that referenced
this pull request
Dec 19, 2024
…elemetry#11540) #### Description This PR follows open-telemetry#11532 and implements support for limited worker pool for queue batcher. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue open-telemetry#8122 open-telemetry#10368
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is a bare minimum implementation of a component called queue batcher. On completion, this component will replace
consumersinqueue_sender, and thus moving queue-batch from a pulling model instead of pushing model.Limitations of the current code
All these code paths are marked as panic currently, and they will be replaced with actual implementation in coming PRs. This PR is split from #11507 for easier review.
Design doc:
https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing
Link to tracking issue
#8122
#10368
Testing
Documentation