[fix] [cli] Fix Broker crashed by too much memory usage of pulsar tools #20031
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
poorbarcode
changed the title
coderzc
reviewed
Apr 7, 2023
conf/pulsar_tools_env.sh
Outdated
| # Discard parameter "-Xms" of $PULSAR_MEM, which tends to be the Broker's minimum memory, to avoid using too much | ||
| # memory by tools. | ||
| if [ -n "$PULSAR_MEM" ]; then | ||
| PULSAR_MEM_REWRITE="-Xmx128m -XX:MaxDirectMemorySize=128m" |
Member
There was a problem hiding this comment.
This shell seems to set PULSAR_MEM to -Xmx128m -XX:MaxDirectMemorySize=128m?
Contributor
Author
There was a problem hiding this comment.
If PULSAR_MEM exists, -Xmx and -XX:MaxDirectMemorySize will be overridden to 128m, and other options in PULSAR_MEM will not be changed.
Member
There was a problem hiding this comment.
We can just remove the -Xms, -Xmx and -XX:MaxDirectMemorySize don't need overriding to 128m
gaoran10
approved these changes
Apr 7, 2023
poorbarcode
force-pushed
the
fix/tools_use_a_log_mem
branch
from
f95c66a to
eb60a33
Compare
poorbarcode
force-pushed
the
fix/tools_use_a_log_mem
branch
from
eb60a33 to
20715ec
Compare
Contributor
Author
|
rebase master |
poorbarcode
force-pushed
the
fix/tools_use_a_log_mem
branch
from
f57f10e to
20715ec
Compare
poorbarcode
added a commit
that referenced
this pull request
Apr 11, 2023
…ls (#20031) ### Motivation After #15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd)
poorbarcode
added a commit
that referenced
this pull request
Apr 11, 2023
…ls (#20031) ### Motivation After #15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 11, 2023
…ls (apache#20031) ### Motivation After apache#15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd) (cherry picked from commit 1fe05d5)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
After #15868, we allow
PULSAR_MEM&PULSAR_GCto be overridden inpulsar_tool_env.sh.Many users set
-Xmsto2Gor larger inPULSAR_MEM, this will make the tools(such aspulsar-admin) cost a lot of memory, and if users executepulsar-adminor another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash.Modifications
When
PULSAR_MEMis overridden inpulsar_tool_env.sh, delete parameter-XmsDocumentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: