-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf #24391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf #24391
Conversation
…Type settings to broker.conf
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #24391 +/- ##
============================================
+ Coverage 73.57% 74.28% +0.71%
- Complexity 32624 32683 +59
============================================
Files 1877 1867 -10
Lines 139502 145276 +5774
Branches 15299 16612 +1313
============================================
+ Hits 102638 107924 +5286
+ Misses 28908 28818 -90
- Partials 7956 8534 +578
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit b55a3f7)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit 5a7ad7e)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit b55a3f7)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit b55a3f7)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit 5a7ad7e)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit b55a3f7)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402) (cherry picked from commit 5a7ad7e)
…s to broker.conf (apache#24391) (cherry picked from commit 1cc2402)
Motivation
To reduce the size of ZNodes, there's ManagedCursorInfo compression which was introduced in PIP-146. ManagedLedgerInfo compression was introduced earlier, in PR #11490.
ManagedCursorInfo compression can enable using a higher
managedLedgerMaxUnackedRangesToPersist
andmanagedLedgerMaxUnackedRangesToPersistInMetadataStore
settings.ManagedLedgerInfo compression is required for a very large amount of BookKeeper or tiered storage ledgers for a single topic. Both settings reduce the size of metadata storage which will reduce overall ZooKeeper memory usage. After enabling compression, existing entries won't be compressed until they are modified.
Modifications
broker.conf
andstandalone.conf
so that it's easier to enable the settings. Without settings inbroker.conf
, it's necessary to usePULSAR_PREFIX_
syntax with Pulsar's Helm chart and docker image when using theapply-config-from-env.py
solution that is used to apply environment variables tobroker.conf
.Documentation
doc
doc-required
doc-not-needed
doc-complete