-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: configurable record batches in flight #759
feat: configurable record batches in flight #759
Conversation
Codecov Report
@@ Coverage Diff @@
## main #759 +/- ##
==========================================
- Coverage 68.20% 68.19% -0.01%
==========================================
Files 296 296
Lines 46057 46066 +9
==========================================
+ Hits 31411 31415 +4
- Misses 14646 14651 +5
... and 4 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* feat: make max_record_batches_in_flight configurable * chore: make scan_batch_size works only for iterators * chore: use separate scan options for compaction * chore: fix clippy warnings * chore: refactor implement stream for batch of streams * chore: stop poll after input stream is exhausted in sst writer
Which issue does this PR close?
Closes #735
Rationale for this change
Currently, streaming reading sst has already been supported. However, the max record batches in flight is not configurable, which is hard-coded as 1024. This pr makes it configurable, especially for normal query and compaction reading.
What changes are included in this PR?
scan_max_record_batches_in_flight
configurable;Are there any user-facing changes?
A new config options is provided called `scan_max_record_batches_in_flight.
How does this change test
Existing tests.