[Enhancement] Support audic statistics for insert statement #29901

liuyehcf · 2023-08-24T11:49:11Z

Fixes #issue
To add some statistics of CPU, Memory, Scanbytes, Scanrows for INSERT INTO statements.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

Signed-off-by: liuyehcf <1559500551@qq.com>

test/sql/test_audit/R/test_audit_log

Signed-off-by: liuyehcf <1559500551@qq.com>

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

gensrc/thrift/FrontendService.thrift

Signed-off-by: liuyehcf <1559500551@qq.com>

sonarqubecloud · 2023-08-28T03:14:46Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
38 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

wanpengfei-git · 2023-08-28T08:08:43Z

[FE Incremental Coverage Report]

😞 fail : 21 / 43 (48.84%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/qe/QeProcessorImpl.java	8	22	36.36%	[180, 181, 182, 187, 188, 189, 190, 191, 195, 196, 197, 199, 200, 201]
🔵	com/starrocks/qe/DefaultCoordinator.java	10	18	55.56%	[893, 894, 895, 896, 897, 898, 899, 900]
🔵	com/starrocks/service/FrontendServiceImpl.java	1	1	100.00%	[]
🔵	com/starrocks/qe/StmtExecutor.java	2	2	100.00%	[]

wanpengfei-git · 2023-08-28T08:09:44Z

[BE Incremental Coverage Report]

😞 fail : 0 / 68 (00.00%)

file detail

	path	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/sink/export_sink_operator.cpp	1	00.00%	[195]
🔵	src/exec/pipeline/sink/iceberg_table_sink_operator.cpp	1	00.00%	[64]
🔵	src/exec/pipeline/pipeline_driver_executor.cpp	19	00.00%	[335, 336, 338, 339, 340, 341, 343, 344, 347, 350, 351, 353, 354, 355, 356, 357, 359, 360, 363]
🔵	src/exec/pipeline/olap_table_sink_operator.cpp	1	00.00%	[141]
🔵	src/exec/pipeline/audit_statistics_reporter.cpp	29	00.00%	[33, 35, 36, 37, 38, 39, 42, 43, 47, 48, 49, 50, 52, 54, 55, 57, 59, 60, 61, 62, 63, 67, 68, 69, 70, 71, 72, 73, 75]
🔵	src/runtime/query_statistics.cpp	14	00.00%	[58, 59, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 71, 72]
🔵	src/exec/pipeline/exec_state_reporter.cpp	2	00.00%	[165, 175]
🔵	src/exec/pipeline/sink/memory_scratch_sink_operator.cpp	1	00.00%	[49]

liuyehcf · 2023-08-30T13:28:28Z

@Mergifyio backport branch-3.1

mergify · 2023-08-30T13:28:32Z

backport branch-3.1

✅ Backports have been created

#30173 [Enhancement] Support audic statistics for insert statement (backport #29901) has been created for branch branch-3.1 but encountered conflicts

liuyehcf · 2023-08-30T13:28:46Z

@Mergifyio backport branch-3.0

liuyehcf · 2023-08-30T13:28:52Z

@Mergifyio backport branch-2.5

* [Enhancement] Support audic statistics for insert statement Signed-off-by: liuyehcf <1559500551@qq.com> * update test Signed-off-by: liuyehcf <1559500551@qq.com> * update 2 Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java

mergify · 2023-08-30T13:29:19Z

backport branch-3.0

✅ Backports have been created

#30174 [Enhancement] Support audic statistics for insert statement (backport #29901) has been created for branch branch-3.0 but encountered conflicts

mergify · 2023-08-30T13:29:23Z

backport branch-2.5

✅ Backports have been created

#30175 [Enhancement] Support audic statistics for insert statement (backport #29901) has been created for branch branch-2.5 but encountered conflicts

* [Enhancement] Support audic statistics for insert statement Signed-off-by: liuyehcf <1559500551@qq.com> * update test Signed-off-by: liuyehcf <1559500551@qq.com> * update 2 Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # be/src/exec/pipeline/pipeline_driver_executor.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java

* [Enhancement] Support audic statistics for insert statement Signed-off-by: liuyehcf <1559500551@qq.com> * update test Signed-off-by: liuyehcf <1559500551@qq.com> * update 2 Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # be/src/exec/pipeline/olap_table_sink_operator.cpp # be/src/exec/pipeline/pipeline_driver_executor.cpp # be/src/exec/pipeline/pipeline_driver_executor.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java

…29901) (#30175) * [Enhancement] Support audic statistics for insert statement (#29901) --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # be/src/exec/pipeline/olap_table_sink_operator.cpp # be/src/exec/pipeline/pipeline_driver_executor.cpp # be/src/exec/pipeline/pipeline_driver_executor.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java * solve conflict Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…29901) (#30174) * [Enhancement] Support audic statistics for insert statement (#29901) --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # be/src/exec/pipeline/pipeline_driver_executor.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java * solve conflict Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…29901) (#30173) * [Enhancement] Support audic statistics for insert statement (#29901) --------- Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 728573e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/service/FrontendServiceImpl.java * solfe conflict Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…s#29901) * [Enhancement] Support audic statistics for insert statement Signed-off-by: liuyehcf <1559500551@qq.com> * update test Signed-off-by: liuyehcf <1559500551@qq.com> * update 2 Signed-off-by: liuyehcf <1559500551@qq.com> --------- Signed-off-by: liuyehcf <1559500551@qq.com>

[Enhancement] Support audic statistics for insert statement

dd38d91

Signed-off-by: liuyehcf <1559500551@qq.com>

mergify bot assigned liuyehcf Aug 24, 2023

kevincai reviewed Aug 24, 2023

View reviewed changes

test/sql/test_audit/R/test_audit_log Outdated Show resolved Hide resolved

update test

529893a

Signed-off-by: liuyehcf <1559500551@qq.com>

Seaven previously approved these changes Aug 25, 2023

View reviewed changes

kangkaisen reviewed Aug 28, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java Show resolved Hide resolved

kangkaisen reviewed Aug 28, 2023

View reviewed changes

gensrc/thrift/FrontendService.thrift Outdated Show resolved Hide resolved

update 2

f64dc1e

Signed-off-by: liuyehcf <1559500551@qq.com>

liuyehcf dismissed Seaven’s stale review via f64dc1e August 28, 2023 02:54

kangkaisen approved these changes Aug 28, 2023

View reviewed changes

Seaven approved these changes Aug 28, 2023

View reviewed changes

kangkaisen approved these changes Aug 28, 2023

View reviewed changes

kangkaisen merged commit 728573e into StarRocks:main Aug 28, 2023

mergify bot mentioned this pull request Aug 30, 2023

[Enhancement] Support audic statistics for insert statement (backport #29901) #30173

Merged

This was referenced Aug 30, 2023

[Enhancement] Support audic statistics for insert statement (backport #29901) #30174

Merged

[Enhancement] Support audic statistics for insert statement (backport #29901) #30175

Merged

liuyehcf mentioned this pull request Aug 31, 2023

[BugFix] Fix wrong state of 'is_result_sink' of query context #30098

Merged

22 tasks

liuyehcf mentioned this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follower #30663

Merged

22 tasks

jaogoy requested a review from yingtingdong September 14, 2023 08:51

jaogoy added the behavior_changed label Sep 14, 2023

ZiheLiu mentioned this pull request Sep 19, 2023

增强sql审计功能 #30262

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Support audic statistics for insert statement #29901

[Enhancement] Support audic statistics for insert statement #29901

liuyehcf commented Aug 24, 2023 •

edited by jaogoy

Loading

sonarqubecloud bot commented Aug 28, 2023

wanpengfei-git commented Aug 28, 2023

wanpengfei-git commented Aug 28, 2023

liuyehcf commented Aug 30, 2023

mergify bot commented Aug 30, 2023 •

edited

Loading

liuyehcf commented Aug 30, 2023

liuyehcf commented Aug 30, 2023

mergify bot commented Aug 30, 2023 •

edited

Loading

mergify bot commented Aug 30, 2023 •

edited

Loading

[Enhancement] Support audic statistics for insert statement #29901

[Enhancement] Support audic statistics for insert statement #29901

Conversation

liuyehcf commented Aug 24, 2023 • edited by jaogoy Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarqubecloud bot commented Aug 28, 2023

wanpengfei-git commented Aug 28, 2023

[FE Incremental Coverage Report]

file detail

wanpengfei-git commented Aug 28, 2023

[BE Incremental Coverage Report]

file detail

liuyehcf commented Aug 30, 2023

mergify bot commented Aug 30, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Aug 30, 2023

liuyehcf commented Aug 30, 2023

mergify bot commented Aug 30, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Aug 30, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Aug 24, 2023 •

edited by jaogoy

Loading

mergify bot commented Aug 30, 2023 •

edited

Loading

mergify bot commented Aug 30, 2023 •

edited

Loading

mergify bot commented Aug 30, 2023 •

edited

Loading