[Refactor] reduce sending audit stats in exchange #37083

fzhedu · 2023-12-14T13:23:20Z

Why I'm doing:

sending audit stats for each chunk request takes 27% of time in high concurrent workloads.

What I'm doing:

reduce sending audit stats every _sent_audit_stats_frequency.
_sent_audit_stats_frequency = max(64, RoundUpToPowerOfTwo(fragment_ctx->num_drivers() * 4))), it is scale up with drivers' num. It help to know large query' audit stats even missing the last chunk.

the last chunk of exchange should sent audit stats.

results:
reduce 4% cost to null

Fixes #37082

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

be/src/exec/pipeline/exchange/sink_buffer.cpp

This reverts commit b859217. Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

liuyehcf · 2023-12-15T08:45:44Z

be/src/exec/pipeline/exchange/sink_buffer.cpp

-                request.params->clear_query_statistics();
-                _eos_query_stats[instance_id.lo]->to_pb(request.params->mutable_query_statistics());
-                _eos_query_stats[instance_id.lo]->clear();
+                if (auto final_stats = _fragment_ctx->runtime_state()->query_ctx()->intermediate_query_statistic()) {


what if there is a short circuit, the receiver side won't wait the last eos packet.

it is up to the sent stats by _sent_audit_stats_frequency

murphyatwork · 2023-12-15T08:46:35Z

be/src/exec/pipeline/exchange/sink_buffer.cpp

+                }
+            }
+        } else {
+            if ((_request_sent & _sent_audit_stats_frequency) == _sent_audit_stats_frequency) {


What if total _request_sent is less than 127 ?

that will be up the eos's last request.

silverbullet233 · 2023-12-15T10:09:51Z

be/src/exec/pipeline/exchange/sink_buffer.h

@@ -192,6 +189,8 @@ class SinkBuffer {
    int64_t _first_send_time = -1;
    int64_t _last_receive_time = -1;
    int64_t _rpc_http_min_size = 0;
+
+    int64_t _sent_audit_stats_frequency = 127;


better to define a static const variable for 127?

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

github-actions · 2023-12-18T05:04:30Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2023-12-18T05:05:56Z

[BE Incremental Coverage Report]

❌ fail : 0 / 11 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/query_context.cpp	0	1	00.00%	[168]
🔵	src/exec/pipeline/exchange/sink_buffer.cpp	0	10	00.00%	[33, 35, 95, 96, 97, 98, 100, 101, 325, 326]

fzhedu · 2023-12-18T08:38:00Z

@mergify backport branch-2.5

mergify · 2023-12-18T08:38:04Z

backport branch-2.5

✅ Backports have been created

#37234 [Refactor] reduce sending audit stats in exchange (backport #37083) has been created for branch branch-2.5 but encountered conflicts

fzhedu · 2023-12-18T08:38:19Z

@mergify backport branch-3.0

mergify · 2023-12-18T08:38:24Z

backport branch-3.0

✅ Backports have been created

#37235 [Refactor] reduce sending audit stats in exchange (backport #37083) has been created for branch branch-3.0 but encountered conflicts

fzhedu · 2023-12-18T08:38:27Z

@mergify backport branch-3.1

mergify · 2023-12-18T08:38:31Z

backport branch-3.1

✅ Backports have been created

#37236 [Refactor] reduce sending audit stats in exchange (backport #37083) has been created for branch branch-3.1 but encountered conflicts

fzhedu · 2023-12-18T08:38:35Z

@mergify backport branch-3.2

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> (cherry picked from commit cd07b2d) # Conflicts: # be/src/exec/pipeline/exchange/sink_buffer.cpp # be/src/runtime/runtime_state.cpp # be/src/runtime/runtime_state.h

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> (cherry picked from commit cd07b2d) # Conflicts: # be/src/runtime/runtime_state.cpp # be/src/runtime/runtime_state.h

mergify · 2023-12-18T08:39:20Z

backport branch-3.2

✅ Backports have been created

#37237 [Refactor] reduce sending audit stats in exchange (backport #37083) has been created for branch branch-3.2

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> (cherry picked from commit cd07b2d)

…37236) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

…37234) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

…37235) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

…37237) Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Signed-off-by: 张敢 <zhanggan@deepexi.com>

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

fzhedu requested review from liuyehcf, murphyatwork and silverbullet233 and removed request for liuyehcf and murphyatwork December 14, 2023 13:23

starrocks-cr bot reviewed Dec 14, 2023

View reviewed changes

be/src/exec/pipeline/exchange/sink_buffer.cpp Show resolved Hide resolved

mergify bot assigned fzhedu Dec 14, 2023

fzhedu force-pushed the auditStats branch from 697dae0 to c1815c9 Compare December 14, 2023 13:26

fzhedu added 3 commits December 15, 2023 12:03

Revert "[BugFix] Fix negative audit (StarRocks#30646)"

66827e5

This reverts commit b859217. Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

[Refacotr] reduce sending audit query stats

3e43e50

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

fix clang format

6587427

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

liuyehcf reviewed Dec 15, 2023

View reviewed changes

murphyatwork reviewed Dec 15, 2023

View reviewed changes

murphyatwork previously approved these changes Dec 15, 2023

View reviewed changes

silverbullet233 reviewed Dec 15, 2023

View reviewed changes

update frequency

8dc66fc

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

fzhedu dismissed murphyatwork’s stale review via 8dc66fc December 15, 2023 10:33

fzhedu force-pushed the auditStats branch from 0413a8a to 8dc66fc Compare December 15, 2023 10:33

move to work threads insetread brpc thread

830ac9d

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

silverbullet233 approved these changes Dec 18, 2023

View reviewed changes

murphyatwork approved these changes Dec 18, 2023

View reviewed changes

fzhedu merged commit cd07b2d into StarRocks:main Dec 18, 2023
44 checks passed

This was referenced Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) #37234

Merged

[Refactor] reduce sending audit stats in exchange (backport #37083) #37235

Merged

[Refactor] reduce sending audit stats in exchange (backport #37083) #37236

Merged

mergify bot pushed a commit that referenced this pull request Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (#37083)

00d092c

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> (cherry picked from commit cd07b2d)

mergify bot mentioned this pull request Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) #37237

Merged

18 tasks

wanpengfei-git pushed a commit that referenced this pull request Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) (#…

09c0ee9

…37236) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

wanpengfei-git pushed a commit that referenced this pull request Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) (#…

2925bb5

…37234) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

wanpengfei-git pushed a commit that referenced this pull request Dec 18, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) (#…

e56ebda

…37235) Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

wanpengfei-git pushed a commit that referenced this pull request Dec 21, 2023

[Refactor] reduce sending audit stats in exchange (backport #37083) (#…

0c7af46

…37237) Co-authored-by: Zhuhe Fang <fzhedu@gmail.com>

Zhangg7723 pushed a commit to Zhangg7723/starrocks that referenced this pull request Dec 26, 2023

[Refactor] reduce sending audit stats in exchange (StarRocks#37083)

1770392

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com> Signed-off-by: 张敢 <zhanggan@deepexi.com>

wangruin added the backport_ok label Jan 9, 2024

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Jan 10, 2024

[Refactor] reduce sending audit stats in exchange (StarRocks#37083)

d66a22a

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Jan 11, 2024

[Refactor] reduce sending audit stats in exchange (StarRocks#37083)

a02a24b

Signed-off-by: Zhuhe Fang <fzhedu@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] reduce sending audit stats in exchange #37083

[Refactor] reduce sending audit stats in exchange #37083

fzhedu commented Dec 14, 2023 •

edited

Loading

liuyehcf Dec 15, 2023

fzhedu Dec 15, 2023

murphyatwork Dec 15, 2023

fzhedu Dec 15, 2023

silverbullet233 Dec 15, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 18, 2023

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 •

edited

Loading

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 •

edited

Loading

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 •

edited

Loading

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 •

edited

Loading

[Refactor] reduce sending audit stats in exchange #37083

[Refactor] reduce sending audit stats in exchange #37083

Conversation

fzhedu commented Dec 14, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

liuyehcf Dec 15, 2023

Choose a reason for hiding this comment

fzhedu Dec 15, 2023

Choose a reason for hiding this comment

murphyatwork Dec 15, 2023

Choose a reason for hiding this comment

fzhedu Dec 15, 2023

Choose a reason for hiding this comment

silverbullet233 Dec 15, 2023

Choose a reason for hiding this comment

github-actions bot commented Dec 18, 2023

[FE Incremental Coverage Report]

github-actions bot commented Dec 18, 2023

[BE Incremental Coverage Report]

file detail

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 • edited Loading

✅ Backports have been created

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 • edited Loading

✅ Backports have been created

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 • edited Loading

✅ Backports have been created

fzhedu commented Dec 18, 2023

mergify bot commented Dec 18, 2023 • edited Loading

✅ Backports have been created

fzhedu commented Dec 14, 2023 •

edited

Loading

mergify bot commented Dec 18, 2023 •

edited

Loading

mergify bot commented Dec 18, 2023 •

edited

Loading

mergify bot commented Dec 18, 2023 •

edited

Loading

mergify bot commented Dec 18, 2023 •

edited

Loading