[BugFix] Support audit for insert into statement executing from follower #30663

liuyehcf · 2023-09-08T11:16:05Z

This is an continuation work of #29901, and it cannot work well when there are multi frontend instances.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

kangkaisen · 2023-09-11T08:57:05Z

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

@@ -480,6 +480,10 @@ public void execute() throws Exception {
                // Write the resource group information to audit log.
                prepareResourceGroup(context, ResourceGroupClassifier.QueryType.fromStatement(parsedStmt));

+                Preconditions.checkState(!(parsedStmt instanceof InsertStmt && parsedStmt.isExplain()


why not forward to leader FE?

kangkaisen · 2023-09-11T08:58:09Z

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

@@ -1368,6 +1372,9 @@ private void handleShow() throws IOException, AnalysisException, DdlException {
    }

    private void handleExplainStmt(String explainString) throws IOException {
+        if (explainString == null) {


Why explainString could be null?

fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java

Signed-off-by: liuyehcf <1559500551@qq.com>

sonarqubecloud · 2023-09-13T05:58:33Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

0.0% Coverage
0.0% Duplication

wanpengfei-git · 2023-09-13T10:56:59Z

[FE Incremental Coverage Report]

😞 fail : 62 / 117 (52.99%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/common/util/AuditStatisticsUtil.java	46	95	48.42%	[27, 30, 40, 41, 42, 43, 44, 45, 46, 47, 54, 58, 64, 70, 76, 82, 88, 93, 94, 96, 97, 98, 99, 100, 102, 103, 104, 105, 106, 108, 110, 111, 112, 114, 117, 118, 120, 126, 136, 148, 149, 150, 151, 153, 154, 156, 157, 159, 160]
🔵	com/starrocks/qe/StmtExecutor.java	5	10	50.00%	[1409, 1410, 1411, 1544, 1545]
🔵	com/starrocks/qe/LeaderOpExecutor.java	3	4	75.00%	[122]
🔵	com/starrocks/sql/ast/InsertStmt.java	1	1	100.00%	[]
🔵	com/starrocks/qe/ConnectProcessor.java	3	3	100.00%	[]
🔵	com/starrocks/qe/DefaultCoordinator.java	4	4	100.00%	[]

wanpengfei-git · 2023-09-13T10:57:19Z

[BE Incremental Coverage Report]

😞 fail : 0 / 38 (00.00%)

file detail

	path	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/result_sink_operator.h	3	00.00%	[114, 117, 125]
🔵	src/exec/pipeline/sink/export_sink_operator.cpp	3	00.00%	[195, 196, 225]
🔵	src/exec/pipeline/sink/memory_scratch_sink_operator.h	5	00.00%	[44, 45, 89, 91, 99]
🔵	src/exec/pipeline/sink/iceberg_table_sink_operator.cpp	2	00.00%	[64, 65]
🔵	src/exec/pipeline/olap_table_sink_operator.h	1	00.00%	[106]
🔵	src/exec/pipeline/pipeline_driver_executor.cpp	2	00.00%	[348, 349]
🔵	src/exec/pipeline/result_sink_operator.cpp	1	00.00%	[68]
🔵	src/exec/pipeline/sink/export_sink_operator.h	6	00.00%	[39, 40, 82, 83, 84, 92]
🔵	src/exec/pipeline/olap_table_sink_operator.cpp	5	00.00%	[141, 142, 200, 203, 206]
🔵	src/exec/pipeline/sink/iceberg_table_sink_operator.h	7	00.00%	[51, 108, 109, 110, 111, 112, 120]
🔵	src/runtime/query_statistics.cpp	1	00.00%	[58]
🔵	src/exec/pipeline/sink/memory_scratch_sink_operator.cpp	2	00.00%	[49, 50]

fe/fe-core/src/main/java/com/starrocks/sql/ast/InsertStmt.java

wanpengfei-git · 2023-09-14T02:26:46Z

@Mergifyio backport branch-3.0

wanpengfei-git · 2023-09-14T02:26:48Z

@Mergifyio backport branch-2.5

mergify · 2023-09-14T02:26:48Z

backport branch-3.1

✅ Backports have been created

#31043 [BugFix] Support audit for insert into statement executing from follower (backport #30663) has been created for branch branch-3.1 but encountered conflicts

mergify · 2023-09-14T02:26:50Z

backport branch-3.0

✅ Backports have been created

#31044 [BugFix] Support audit for insert into statement executing from follower (backport #30663) has been created for branch branch-3.0 but encountered conflicts

mergify · 2023-09-14T02:26:53Z

backport branch-2.5

✅ Backports have been created

#31045 [BugFix] Support audit for insert into statement executing from follower (backport #30663) has been created for branch branch-2.5 but encountered conflicts

…wer (#30663) Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 4a6b0c9) # Conflicts: # be/src/exec/pipeline/result_sink_operator.h # fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

…wer (#30663) Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 4a6b0c9) # Conflicts: # be/src/exec/pipeline/olap_table_sink_operator.cpp # be/src/exec/pipeline/olap_table_sink_operator.h # be/src/exec/pipeline/result_sink_operator.h # be/src/exec/pipeline/sink/export_sink_operator.cpp # be/src/exec/pipeline/sink/export_sink_operator.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # be/src/exec/pipeline/sink/iceberg_table_sink_operator.h # be/src/exec/pipeline/sink/memory_scratch_sink_operator.cpp # be/src/exec/pipeline/sink/memory_scratch_sink_operator.h # fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

…wer (#30663) Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 4a6b0c9) # Conflicts: # be/src/exec/pipeline/olap_table_sink_operator.cpp # be/src/exec/pipeline/olap_table_sink_operator.h # be/src/exec/pipeline/result_sink_operator.h # be/src/exec/pipeline/sink/export_sink_operator.cpp # be/src/exec/pipeline/sink/export_sink_operator.h # be/src/exec/pipeline/sink/iceberg_table_sink_operator.cpp # be/src/exec/pipeline/sink/iceberg_table_sink_operator.h # be/src/exec/pipeline/sink/memory_scratch_sink_operator.cpp # be/src/exec/pipeline/sink/memory_scratch_sink_operator.h # fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/LeaderOpExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # gensrc/thrift/FrontendService.thrift

liuyehcf · 2023-09-14T02:29:10Z

@Mergifyio backport branch-3.2

mergify · 2023-09-14T02:29:13Z

backport branch-3.2

✅ Backports have been created

#31046 [BugFix] Support audit for insert into statement executing from follower (backport #30663) has been created for branch branch-3.2 but encountered conflicts

…wer (#30663) Signed-off-by: liuyehcf <1559500551@qq.com> (cherry picked from commit 4a6b0c9) # Conflicts: # be/src/exec/pipeline/result_sink_operator.h # fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java # fe/fe-core/src/main/java/com/starrocks/qe/LeaderOpExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # gensrc/thrift/FrontendService.thrift

…wer (backport #30663) (#31045) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…om follower (backport #30663) (#31045)" This reverts commit 4b43dd7.

…wer(2) (backport StarRocks#30663)

…wer (backport #30663) (#31044) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…wer (backport #30663) (#31043) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

…wer (backport #30663) (#31046) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

github-actions bot added the 3.1 label Sep 8, 2023

mergify bot assigned liuyehcf Sep 8, 2023

github-actions bot added 3.0 2.5 labels Sep 8, 2023

liuyehcf force-pushed the audit_for_insert branch 2 times, most recently from 590bc28 to 46b9bdf Compare September 11, 2023 01:45

kangkaisen reviewed Sep 11, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java Show resolved Hide resolved

liuyehcf force-pushed the audit_for_insert branch 4 times, most recently from faa4edb to 4bafe8a Compare September 13, 2023 02:19

liuyehcf changed the title ~~[BugFix] Fix audit update process for insert into statements~~ [BugFix] Support audit for insert into statement executing from follower Sep 13, 2023

liuyehcf force-pushed the audit_for_insert branch 4 times, most recently from 4405320 to 8953f42 Compare September 13, 2023 02:59

ZiheLiu previously approved these changes Sep 13, 2023

View reviewed changes

[BugFix] Support audit for insert into statement executing from follower

8f28f9a

Signed-off-by: liuyehcf <1559500551@qq.com>

liuyehcf dismissed ZiheLiu’s stale review via 8f28f9a September 13, 2023 05:11

liuyehcf force-pushed the audit_for_insert branch from 8953f42 to 8f28f9a Compare September 13, 2023 05:11

fix ut

f01bfb7

Signed-off-by: liuyehcf <1559500551@qq.com>

ZiheLiu approved these changes Sep 14, 2023

View reviewed changes

kangkaisen reviewed Sep 14, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/sql/ast/InsertStmt.java Show resolved Hide resolved

kangkaisen approved these changes Sep 14, 2023

View reviewed changes

kangkaisen merged commit 4a6b0c9 into StarRocks:main Sep 14, 2023

github-actions bot removed 3.1 3.0 labels Sep 14, 2023

github-actions bot removed the 2.5 label Sep 14, 2023

mergify bot mentioned this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follower (backport #30663) #31043

Merged

mergify bot mentioned this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follower (backport #30663) #31044

Merged

mergify bot mentioned this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follower (backport #30663) #31045

Merged

mergify bot mentioned this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follower (backport #30663) #31046

Merged

github-actions bot added 3.1 3.0 2.5 labels Sep 14, 2023

kangkaisen pushed a commit that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

4b43dd7

…wer (backport #30663) (#31045) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

liuyehcf added a commit that referenced this pull request Sep 14, 2023

Revert "[BugFix] Support audit for insert into statement executing fr…

a61a282

…om follower (backport #30663) (#31045)" This reverts commit 4b43dd7.

liuyehcf added a commit to liuyehcf/starrocks that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

4918ceb

…wer(2) (backport StarRocks#30663)

liuyehcf added a commit to liuyehcf/starrocks that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

5a7af9e

…wer(2) (backport StarRocks#30663)

liuyehcf added a commit that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

069b104

…wer (backport #30663) (#31044) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

liuyehcf added a commit that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

432215d

…wer (backport #30663) (#31043) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

liuyehcf added a commit that referenced this pull request Sep 14, 2023

[BugFix] Support audit for insert into statement executing from follo…

cc9c31d

…wer (backport #30663) (#31046) Signed-off-by: liuyehcf <1559500551@qq.com> Co-authored-by: liuyehcf <1559500551@qq.com>

ZiheLiu mentioned this pull request Sep 19, 2023

增强sql审计功能 #30262

Closed

wanpengfei-git added the backport_ok label Dec 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] Support audit for insert into statement executing from follower #30663

[BugFix] Support audit for insert into statement executing from follower #30663

liuyehcf commented Sep 8, 2023 •

edited

Loading

kangkaisen Sep 11, 2023

kangkaisen Sep 11, 2023

liuyehcf Sep 13, 2023

sonarqubecloud bot commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

wanpengfei-git commented Sep 14, 2023

wanpengfei-git commented Sep 14, 2023

mergify bot commented Sep 14, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading

liuyehcf commented Sep 14, 2023

mergify bot commented Sep 14, 2023 •

edited

Loading

[BugFix] Support audit for insert into statement executing from follower #30663

[BugFix] Support audit for insert into statement executing from follower #30663

Conversation

liuyehcf commented Sep 8, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

kangkaisen Sep 11, 2023

Choose a reason for hiding this comment

kangkaisen Sep 11, 2023

Choose a reason for hiding this comment

liuyehcf Sep 13, 2023

Choose a reason for hiding this comment

sonarqubecloud bot commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

[FE Incremental Coverage Report]

file detail

wanpengfei-git commented Sep 13, 2023

[BE Incremental Coverage Report]

file detail

wanpengfei-git commented Sep 14, 2023

wanpengfei-git commented Sep 14, 2023

mergify bot commented Sep 14, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Sep 14, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Sep 14, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Sep 14, 2023

mergify bot commented Sep 14, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Sep 8, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading

mergify bot commented Sep 14, 2023 •

edited

Loading