[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

silverbullet233 · 2024-05-28T08:39:16Z

Why I'm doing:

What I'm doing:

Major changes:

add new metrics to query profile to observe the memory occupied by partial runtime bloom filter.
add new be system metrics to observe the number of rf events in the queue and the bloom filter memory consumption in the queue.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

be/src/runtime/runtime_filter_worker.h

be/src/runtime/runtime_filter_worker.cpp

be/src/util/system_metrics.cpp

Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions · 2024-05-29T08:16:29Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-05-29T08:25:39Z

[BE Incremental Coverage Report]

✅ pass : 72 / 76 (94.74%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/runtime/runtime_filter_worker.h	18	21	85.71%	[141, 142, 144]
🔵	be/src/util/system_metrics.cpp	23	24	95.83%	[688]
🔵	be/src/exec/pipeline/hashjoin/hash_join_build_operator.cpp	7	7	100.00%	[]
🔵	be/src/runtime/runtime_filter_worker.cpp	18	18	100.00%	[]
🔵	be/src/exec/hash_joiner.cpp	1	1	100.00%	[]
🔵	be/src/exprs/runtime_filter.h	5	5	100.00%	[]

kevincai · 2024-05-29T08:37:46Z

be/src/util/system_metrics.cpp

+    for (int i = 0; i < EventType::MAX_COUNT; i++) {
+        auto* metrics = new RuntimeFilterMetrics();
+        const auto& type = EventTypeToString((EventType)i);
+#define REGISTER_RUNTIME_FILTER_METRIC(name) \


maybe better move this macro out of the function body, favor for code readability.

kevincai · 2024-05-29T08:38:51Z

be/src/util/system_metrics.cpp

@@ -135,6 +143,9 @@ SystemMetrics::~SystemMetrics() {
    if (_line_ptr != nullptr) {
        free(_line_ptr);
    }
+    for (auto& it : _runtime_filter_metrics) {
+        delete it.second;


may consider using std::unique_ptr to manage the pointers instead of using the raw one. so the resource deallocation will be as easy as _runtime_filter_metrics.clear()

silverbullet233 · 2024-05-29T08:53:44Z

@mergify backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

mergify · 2024-05-29T08:54:23Z

backport branch-2.5 branch-3.0 branch-3.1 branch-3.2 branch-3.3

✅ Backports have been created

#46406 [Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) has been created for branch branch-2.5 but encountered conflicts
#46407 [Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) has been created for branch branch-3.0 but encountered conflicts
#46408 [Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) has been created for branch branch-3.1 but encountered conflicts
#46409 [Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) has been created for branch branch-3.2 but encountered conflicts
#46410 [Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) has been created for branch branch-3.3 but encountered conflicts

…ilter size (#46360) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com> (cherry picked from commit 16ec9c1) # Conflicts: # be/src/exec/pipeline/hashjoin/hash_join_build_operator.cpp # be/src/runtime/runtime_filter_worker.cpp # be/src/runtime/runtime_filter_worker.h

…ilter size (#46360) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com> (cherry picked from commit 16ec9c1) # Conflicts: # be/src/runtime/runtime_filter_worker.cpp # be/src/runtime/runtime_filter_worker.h

…ilter size (StarRocks#46360) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

…ilter size(backport #46360) (#46438) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

…ilter(backport #46360) (#46441) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

…ilter size (backport #46360) (#46445) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

…ilter size(backport #46360) (#46453) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

…ilter size (backport #46360) (#46430) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

more profile and metrics to measure runtiem bloom filter size

cecfad4

Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

silverbullet233 requested review from a team as code owners May 28, 2024 08:39

mergify bot assigned silverbullet233 May 28, 2024

starrocks-cr bot reviewed May 28, 2024

View reviewed changes

be/src/runtime/runtime_filter_worker.h Show resolved Hide resolved

starrocks-cr bot reviewed May 28, 2024

View reviewed changes

be/src/runtime/runtime_filter_worker.cpp Show resolved Hide resolved

starrocks-cr bot reviewed May 28, 2024

View reviewed changes

be/src/util/system_metrics.cpp Show resolved Hide resolved

fix

e74dc5d

Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

satanson previously approved these changes May 28, 2024

View reviewed changes

stdpain previously approved these changes May 28, 2024

View reviewed changes

ZiheLiu previously approved these changes May 28, 2024

View reviewed changes

fix format

8f9e5f9

Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

silverbullet233 dismissed stale reviews from ZiheLiu, stdpain, and satanson via 8f9e5f9 May 29, 2024 06:22

ZiheLiu approved these changes May 29, 2024

View reviewed changes

satanson approved these changes May 29, 2024

View reviewed changes

silverbullet233 enabled auto-merge (squash) May 29, 2024 08:34

kevincai approved these changes May 29, 2024

View reviewed changes

silverbullet233 merged commit 16ec9c1 into StarRocks:main May 29, 2024
46 checks passed

silverbullet233 deleted the rf_metrics branch May 29, 2024 08:44

github-actions bot added the 3.3 label May 29, 2024

github-actions bot added 3.2 3.1 labels May 29, 2024

github-actions bot added 3.0 2.5 labels May 29, 2024

This was referenced May 29, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46407

Closed

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46408

Closed

This was referenced May 29, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46409

Closed

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46410

Closed

silverbullet233 mentioned this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46430

Merged

24 tasks

silverbullet233 mentioned this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size(backport #46360) #46438

Merged

24 tasks

silverbullet233 mentioned this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter(backport #46360) #46441

Merged

24 tasks

silverbullet233 mentioned this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size (backport #46360) #46445

Merged

24 tasks

silverbullet233 mentioned this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom filter size(backport #46360) #46453

Merged

24 tasks

silverbullet233 added a commit that referenced this pull request May 30, 2024

[Enhancement] add more profile and metrics to measure runtime bloom f…

9e6ac31

…ilter size(backport #46360) (#46438) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions bot added the 3.2-merged label May 30, 2024

silverbullet233 added a commit that referenced this pull request May 31, 2024

[Enhancement] add more profile and metrics to measure runtime bloom f…

bc107d6

…ilter(backport #46360) (#46441) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions bot added the 3.1-merged label May 31, 2024

silverbullet233 added a commit that referenced this pull request May 31, 2024

[Enhancement] add more profile and metrics to measure runtime bloom f…

1e86d11

…ilter size (backport #46360) (#46445) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions bot added the 3.0-merged label May 31, 2024

silverbullet233 added a commit that referenced this pull request May 31, 2024

[Enhancement] add more profile and metrics to measure runtime bloom f…

f35a5ec

…ilter size(backport #46360) (#46453) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions bot added the 2.5-merged label May 31, 2024

silverbullet233 added a commit that referenced this pull request Jun 7, 2024

[Enhancement] add more profile and metrics to measure runtime bloom f…

d696eb8

…ilter size (backport #46360) (#46430) Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>

github-actions bot added the 3.3-merged label Jun 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

silverbullet233 commented May 28, 2024 •

edited

Loading

github-actions bot commented May 29, 2024

github-actions bot commented May 29, 2024

kevincai May 29, 2024

kevincai May 29, 2024

silverbullet233 commented May 29, 2024

mergify bot commented May 29, 2024 •

edited

Loading

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

[Enhancement] add more profile and metrics to measure runtime bloom filter size #46360

Conversation

silverbullet233 commented May 28, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

github-actions bot commented May 29, 2024

[FE Incremental Coverage Report]

github-actions bot commented May 29, 2024

[BE Incremental Coverage Report]

file detail

kevincai May 29, 2024

Choose a reason for hiding this comment

kevincai May 29, 2024

Choose a reason for hiding this comment

silverbullet233 commented May 29, 2024

mergify bot commented May 29, 2024 • edited Loading

✅ Backports have been created

silverbullet233 commented May 28, 2024 •

edited

Loading

mergify bot commented May 29, 2024 •

edited

Loading