FEAT-#4501: Add RSS Memory Profiling to Modin Logging #4502
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
Codecov Report
@@ Coverage Diff @@
## master #4502 +/- ##
==========================================
+ Coverage 86.64% 89.37% +2.73%
==========================================
Files 226 229 +3
Lines 18298 21229 +2931
==========================================
+ Hits 15854 18974 +3120
+ Misses 2444 2255 -189
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
jeffreykennethli
previously approved these changes
May 31, 2022
Signed-off-by: Naren Krishna <naren@ponder.io>
naren-ponder
changed the title
Signed-off-by: Naren Krishna <naren@ponder.io>
mvashishtha
reviewed
May 31, 2022
naren-ponder
changed the title
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
mvashishtha
reviewed
Jun 1, 2022
There was a problem hiding this comment.
For logging memory specifically, I think we should probably use this: https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
Signed-off-by: Naren Krishna <naren@ponder.io>
RehanSD
reviewed
Jun 1, 2022
modin/logging/config.py
Outdated
| logfile = RotatingFileHandler( | ||
| filename=log_filename, | ||
| mode="a", | ||
| maxBytes=100000000, |
There was a problem hiding this comment.
Why do we have maxBytes set so high?
There was a problem hiding this comment.
I set it to 10MB for each file (since it is being stored locally I think this is reasonable), and logging a max of 10GB.
There was a problem hiding this comment.
https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler -- because backupCount is set to 100. At maximum you get 100 files of 10MB each. If people are running logs at that level anyhow and need support, they would likely reach out to us somehow.
There was a problem hiding this comment.
This looks like 100 MB (10^8)? Is that what you meant to put? I think something like int(1e8) would be more readable.
There was a problem hiding this comment.
I think it might be better to make this configurable instead of hard coded.
I think setting a default max of 10GB of logs is way too high. This is 10GB per Modin job, not total. I prefer to cap it at 100MB per job by default. If more is needed, they can set a configuration.
There was a problem hiding this comment.
So 10MB per file, and a max of 10 files, for a total of 100MB per job? Was looking through Google and the max MB per file for logs people recommend for a local filesystem is somewhere in the 10-50MB range.
devin-petersohn
requested changes
Jun 2, 2022
modin/logging/config.py
Outdated
| logfile = RotatingFileHandler( | ||
| filename=log_filename, | ||
| mode="a", | ||
| maxBytes=100000000, |
There was a problem hiding this comment.
I think it might be better to make this configurable instead of hard coded.
I think setting a default max of 10GB of logs is way too high. This is 10GB per Modin job, not total. I prefer to cap it at 100MB per job by default. If more is needed, they can set a configuration.
Signed-off-by: Naren Krishna <naren@ponder.io>
devin-petersohn
approved these changes
Jun 3, 2022
There was a problem hiding this comment.
This LGTM, @naren-ponder thanks!
I want to separate the different kinds of logs in a separate PR. Will open an issue for that.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Naren Krishna naren@ponder.io
What do these changes do?
Add RSS Memory Profiling at
LogMemoryIntervalinterval to Modin logging. This helps with out-of-core measurements.flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date