Add memory and cache stats to PeriodicStatsReporter #9723
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
from
1ec0204
to
37549d5
Compare
tanjialiang
changed the title
xiaoxmeng
reviewed
May 6, 2024
| @@ -39,13 +41,22 @@ class PeriodicStatsReporter { | |||
| struct Options { | |||
| Options() {} | |||
|
|
|||
| const memory::MemoryArbitrator* arbitrator{nullptr}; | |||
| const velox::memory::MemoryAllocator* const allocator{nullptr}; | |||
There was a problem hiding this comment.
const velox::memory::MemoryAllocator* allocator{nullptr};
const velox::cache::AsyncDataCache* cache{nullptr};
| if (auto* mmapAllocator = | ||
| dynamic_cast<const velox::memory::MmapAllocator*>(allocator_)) { | ||
| RECORD_METRIC_VALUE( | ||
| kMetricMmapRawAllocBytesSmall, (mmapAllocator->numMallocBytes())); |
There was a problem hiding this comment.
s/kMetricMmapRawAllocBytesSmall/kMetricMmapDelegatedAllocBytes/?
| const auto cacheStats = cache_->refreshStats(); | ||
|
|
||
| // Snapshots. | ||
| RECORD_METRIC_VALUE(kMetricMemoryCacheNumEntries, cacheStats.numEntries); |
There was a problem hiding this comment.
Since this is a velox change and you update SSD cache production with these non-average stats? Or you first report average stats and does the non-average stats in followup? Thanks!
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
2 times, most recently
from
2d391c5
to
0fb06c7
Compare
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
3 times, most recently
from
a340965
to
cf8ebda
Compare
xiaoxmeng
reviewed
May 8, 2024
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
2 times, most recently
from
46da2b2
to
05e9cd4
Compare
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
xiaoxmeng
approved these changes
May 8, 2024
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
zacw7
approved these changes
May 8, 2024
velox/common/base/Counters.cpp
Outdated
| // Total number of bytes written to SSD. | ||
| DEFINE_METRIC(kMetricSsdCacheWrittenBytes, facebook::velox::StatType::SUM); | ||
|
|
||
| // Cumulative number of SsdCache entries that are aged out and evicted given |
There was a problem hiding this comment.
Cumulative -> Total to be consistent with other comments?
velox/common/base/Counters.cpp
Outdated
| // configured TTL. | ||
| DEFINE_METRIC(kMetricSsdCacheAgedOutEntries, facebook::velox::StatType::SUM); | ||
|
|
||
| // Cumulative number of SsdCache regions that are aged out and evicted given |
tanjialiang
added a commit
to tanjialiang/velox-1
that referenced
this pull request
May 8, 2024
…r#9723) Summary: Added memory allocator and cache stats to PeriodicStatsReporter. Added documentations to monitoring doc. Reviewed By: xiaoxmeng, zacw7 Differential Revision: D57085696 Pulled By: tanjialiang
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
from
05e9cd4
to
06c7837
Compare
|
This pull request was exported from Phabricator. Differential Revision: D57085696 |
…r#9723) Summary: Added memory allocator and cache stats to PeriodicStatsReporter. Added documentations to monitoring doc. Reviewed By: xiaoxmeng, zacw7 Differential Revision: D57085696 Pulled By: tanjialiang
tanjialiang
force-pushed
the
add_alloc_mem_stats
branch
from
06c7837
to
72a371a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D57085696 |
|
@tanjialiang merged this pull request in b8a26ce. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
Jun 7, 2024
…r#9723) Summary: Added memory allocator and cache stats to PeriodicStatsReporter. Added documentations to monitoring doc. Pull Request resolved: facebookincubator#9723 Reviewed By: xiaoxmeng, zacw7 Differential Revision: D57085696 Pulled By: tanjialiang fbshipit-source-id: 7dbc3791d314212e0c7658d0d868dac074f34fcf
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
Jun 7, 2024
…r#9723) Summary: Added memory allocator and cache stats to PeriodicStatsReporter. Added documentations to monitoring doc. Pull Request resolved: facebookincubator#9723 Reviewed By: xiaoxmeng, zacw7 Differential Revision: D57085696 Pulled By: tanjialiang fbshipit-source-id: 7dbc3791d314212e0c7658d0d868dac074f34fcf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added memory allocator and cache stats to PeriodicStatsReporter.
Added documentations to monitoring doc.