Skip to content

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 6, 2025

Motivation

To reduce the size of ZNodes, there's ManagedCursorInfo compression which was introduced in PIP-146. ManagedLedgerInfo compression was introduced earlier, in PR #11490.
ManagedCursorInfo compression can enable using a higher managedLedgerMaxUnackedRangesToPersist and managedLedgerMaxUnackedRangesToPersistInMetadataStore settings.
ManagedLedgerInfo compression is required for a very large amount of BookKeeper or tiered storage ledgers for a single topic. Both settings reduce the size of metadata storage which will reduce overall ZooKeeper memory usage. After enabling compression, existing entries won't be compressed until they are modified.

Modifications

  • add the settings to broker.conf and standalone.conf so that it's easier to enable the settings. Without settings in broker.conf, it's necessary to use PULSAR_PREFIX_ syntax with Pulsar's Helm chart and docker image when using the apply-config-from-env.py solution that is used to apply environment variables to broker.conf.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 4.1.0 milestone Jun 6, 2025
@lhotari lhotari self-assigned this Jun 6, 2025
@lhotari lhotari changed the title [improve][broker] Add managedCursor/LedgerInfoCompressionType setting to broker.conf [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf Jun 6, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 6, 2025
@codecov-commenter
Copy link

codecov-commenter commented Jun 6, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.28%. Comparing base (bbc6224) to head (3d91679).
⚠️ Report is 1311 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24391      +/-   ##
============================================
+ Coverage     73.57%   74.28%   +0.71%     
- Complexity    32624    32683      +59     
============================================
  Files          1877     1867      -10     
  Lines        139502   145276    +5774     
  Branches      15299    16612    +1313     
============================================
+ Hits         102638   107924    +5286     
+ Misses        28908    28818      -90     
- Partials       7956     8534     +578     
Flag Coverage Δ
inttests 26.71% <ø> (+2.12%) ⬆️
systests 23.28% <ø> (-1.04%) ⬇️
unittests 73.76% <ø> (+0.92%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1082 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lhotari lhotari requested a review from merlimat June 6, 2025 16:39
Copy link
Member

@dao-jun dao-jun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari lhotari merged commit 1cc2402 into apache:master Jun 9, 2025
56 of 59 checks passed
lhotari added a commit that referenced this pull request Jun 9, 2025
lhotari added a commit that referenced this pull request Jun 9, 2025
lhotari added a commit that referenced this pull request Jun 9, 2025
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 12, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit b55a3f7)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 12, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit 5a7ad7e)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 12, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit b55a3f7)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 13, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit b55a3f7)
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 13, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit 5a7ad7e)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 16, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit b55a3f7)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 16, 2025
…s to broker.conf (apache#24391)

(cherry picked from commit 1cc2402)
(cherry picked from commit 5a7ad7e)
nodece pushed a commit to nodece/pulsar that referenced this pull request Jun 18, 2025
KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025
walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants