[improve][broker] Don't rollover empty ledgers based on inactivity #21893
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhotari
requested review from
merlimat,
rdhabalia,
hangc0276,
AnonHxy,
codelipenghui,
zymap and
poorbarcode
|
@lifepuzzlefun Please review this since you worked on related change #20276 in this area. #21508 is also related. |
coderzc
approved these changes
Jan 12, 2024
@coderzc The empty ledgers that get created will get deleted before this change. The user doesn't see them currently and nothing changes from the user's perspective. Documentation might not be needed in the configuration docs. |
Technoboy-
approved these changes
Jan 13, 2024
lifepuzzlefun
approved these changes
Jan 14, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #21893 +/- ##
============================================
- Coverage 73.72% 73.57% -0.16%
+ Complexity 32405 32333 -72
============================================
Files 1859 1859
Lines 138273 138273
Branches 15158 15158
============================================
- Hits 101939 101731 -208
- Misses 28488 28657 +169
- Partials 7846 7885 +39
Flags with carried forward coverage won't be shown. Click here to find out more.
|
poorbarcode
approved these changes
Jan 15, 2024
|
well, maybe we can consider merging this logic into |
we can raise another new patch for this, I think. |
Technoboy-
pushed a commit
that referenced
this pull request
Jan 15, 2024
…21893) ### Motivation When `managedLedgerInactiveLedgerRolloverTimeSeconds` is set, let's say to `300` (5 minutes), the ledger will also get rolled in the case when no new entries (messages) were added to the ledger. This doesn't make sense. Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies, and metadata stores (zookeeper). ### Modifications Skip rolling the ledger if it is empty.
lhotari
added a commit
that referenced
this pull request
Jan 15, 2024
…21893) ### Motivation When `managedLedgerInactiveLedgerRolloverTimeSeconds` is set, let's say to `300` (5 minutes), the ledger will also get rolled in the case when no new entries (messages) were added to the ledger. This doesn't make sense. Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies, and metadata stores (zookeeper). ### Modifications Skip rolling the ledger if it is empty. (cherry picked from commit 49edc3d)
lhotari
added a commit
that referenced
this pull request
Jan 15, 2024
…21893) ### Motivation When `managedLedgerInactiveLedgerRolloverTimeSeconds` is set, let's say to `300` (5 minutes), the ledger will also get rolled in the case when no new entries (messages) were added to the ledger. This doesn't make sense. Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies, and metadata stores (zookeeper). ### Modifications Skip rolling the ledger if it is empty. (cherry picked from commit 49edc3d)
15 tasks
mukesh-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 1, 2024
…pache#21893) ### Motivation When `managedLedgerInactiveLedgerRolloverTimeSeconds` is set, let's say to `300` (5 minutes), the ledger will also get rolled in the case when no new entries (messages) were added to the ledger. This doesn't make sense. Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies, and metadata stores (zookeeper). ### Modifications Skip rolling the ledger if it is empty. (cherry picked from commit 49edc3d) (cherry picked from commit 84a4885)
mukesh-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 6, 2024
…pache#21893) ### Motivation When `managedLedgerInactiveLedgerRolloverTimeSeconds` is set, let's say to `300` (5 minutes), the ledger will also get rolled in the case when no new entries (messages) were added to the ledger. This doesn't make sense. Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies, and metadata stores (zookeeper). ### Modifications Skip rolling the ledger if it is empty. (cherry picked from commit 49edc3d) (cherry picked from commit 84a4885)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When
managedLedgerInactiveLedgerRolloverTimeSecondsis set, let's say to300(5 minutes), the ledger will get rolled also in the case when no new entries (messages) were added to the ledger. This doesn't make sense.Empty ledgers are deleted, but having this extra churn is causing extra load on brokers, bookies and metadata store (zookeeper).
Modifications
Skip rolling the ledger if it is empty.
Documentation
docdoc-requireddoc-not-neededdoc-complete