[Feature] support prepare statement #27840

Jay-ju · 2023-07-24T12:07:33Z

Fixes #issue

support prepare statement

What type of PR is this:

Feature will improve query efficiency when queries are same in pattern.
it also can prevent sql injection， usage is same as other db， like mysql, oracle

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

Checklist:

I have added test cases for my bug fix or my new feature
This pr will affect users' behaviors
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

imay · 2023-07-24T12:57:13Z

fe/fe-core/src/main/java/com/starrocks/analysis/Parameter.java

@@ -0,0 +1,107 @@
+// Copyright 2021-present StarRocks, Inc. All rights reserved.


need use starrocks license header.

liuyehcf · 2023-07-25T08:26:30Z

Please explain the purpose and function of this pr.

wanpengfei-git · 2023-07-26T09:29:42Z

[FE PR Coverage Check]

😞 fail : 26 / 294 (08.84%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/ast/AstVisitor.java	0	4	00.00%	[85, 89, 93, 1174]
🔵	com/starrocks/analysis/DecimalLiteral.java	0	11	00.00%	[524, 525, 527, 528, 529, 530, 531, 532, 533, 534, 535]
🔵	com/starrocks/analysis/LiteralExpr.java	0	36	00.00%	[258, 262, 265, 268, 271, 274, 277, 280, 285, 288, 291, 298, 302, 305, 307, 313, 314, 315, 318, 319, 320, 321, 324, 325, 326, 327, 330, 331, 339, 340, 342, 343, 344, 345, 346, 348]
🔵	com/starrocks/sql/ast/ExecuteStmt.java	0	10	00.00%	[30, 31, 32, 33, 36, 40, 45, 46, 51, 56]
🔵	com/starrocks/analysis/FloatLiteral.java	0	4	00.00%	[247, 248, 249, 250]
🔵	com/starrocks/analysis/DateLiteral.java	0	31	00.00%	[450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 461, 463, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 476, 477, 478, 479, 481, 482, 484, 487, 490]
🔵	com/starrocks/analysis/IntLiteral.java	0	11	00.00%	[403, 405, 406, 408, 409, 411, 412, 414, 415, 417, 419]
🔵	com/starrocks/sql/optimizer/transformer/SqlToScalarOperatorTranslator.java	0	3	00.00%	[293, 294, 296]
🔵	com/starrocks/sql/ast/DeallocateStmt.java	0	7	00.00%	[23, 24, 25, 28, 33, 38, 43]
🔵	com/starrocks/sql/ast/StatementBase.java	0	3	00.00%	[111, 115, 116]
🔵	com/starrocks/sql/analyzer/ExpressionAnalyzer.java	0	1	00.00%	[1603]
🔵	com/starrocks/qe/StmtExecutor.java	1	46	02.17%	[448, 449, 450, 451, 452, 453, 455, 456, 457, 458, 655, 656, 657, 658, 1448, 1449, 1450, 1451, 1453, 1454, 1455, 1458, 1459, 1461, 1462, 1464, 1465, 1466, 1468, 1473, 1475, 1477, 1479, 1480, 1482, 1483, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493]
🔵	com/starrocks/qe/ConnectProcessor.java	4	51	07.84%	[463, 465, 467, 469, 470, 471, 472, 473, 474, 475, 477, 479, 480, 482, 484, 486, 487, 488, 489, 490, 492, 494, 496, 497, 498, 499, 500, 502, 503, 504, 506, 508, 509, 510, 511, 512, 513, 514, 516, 517, 519, 522, 523, 524, 525, 528, 571]
🔵	com/starrocks/analysis/Parameter.java	3	24	12.50%	[51, 52, 53, 54, 55, 58, 62, 66, 70, 71, 74, 75, 79, 84, 89, 90, 92, 97, 98, 100, 105]
🔵	com/starrocks/sql/ast/PrepareStmt.java	7	23	30.43%	[44, 49, 51, 54, 55, 56, 58, 62, 63, 64, 65, 67, 69, 70, 75, 85]
🔵	com/starrocks/qe/ConnectContext.java	2	6	33.33%	[263, 264, 267, 271]
🔵	com/starrocks/sql/analyzer/PrepareAnalyzer.java	6	17	35.29%	[37, 42, 43, 44, 45, 47, 48, 49, 50, 53, 54]
🔵	com/starrocks/sql/analyzer/AnalyzerVisitor.java	2	4	50.00%	[884, 885]
🔵	com/starrocks/qe/SessionVariable.java	1	2	50.00%	[2307]

fe/fe-core/src/main/java/com/starrocks/analysis/LiteralExpr.java

fe/fe-core/src/main/java/com/starrocks/analysis/Parameter.java

fe/fe-core/src/test/java/com/starrocks/analysis/PreparedStmtTest.java

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

Jay-ju · 2023-08-03T07:09:08Z

Parameter.java

PR intention has been mentioned
"1. Feature will improve query efficiency when queries are same in pattern, this will skip some analyzer, only fill params .
2. it also can prevent sql injection， usage is same as other db， like mysql, oracle"

Please explain the purpose and function of this pr.

sonarqubecloud · 2023-08-04T05:42:55Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
42 Code Smells

0.0% Coverage
0.6% Duplication

The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

packy92 · 2023-08-04T06:41:20Z

fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java

@@ -471,6 +552,7 @@ private void dispatch() throws IOException {
                handleQuit();
                break;
            case COM_QUERY:
+            case COM_STMT_PREPARE:
                handleQuery();


Is this PR support JDBC useServerPrepStmts=true option? I mean there are some differents between using sql text and using JDBC prepareStament.

yes, using sql text will bring some problems like sql injection, in addition, prepare statement here will skip parser. In future we also can use it skip parser or planner, but the work will be more. so util now we only offer 2 benefits

need reply

fe/fe-core/src/main/java/com/starrocks/analysis/LiteralExpr.java

Seaven · 2023-08-25T06:34:34Z

fe/fe-core/src/main/java/com/starrocks/qe/ConnectProcessor.java

+            // audit will affect performance
+            boolean enableAudit = ctx.getSessionVariable().isAuditExecuteStmt();
+            String originStmt = enableAudit ? executeStmt.toSql() : "/* omit */";
+            executeStmt.setOrigStmt(new OriginStatement(originStmt, 0));


I suggest to print prepareStmt and parameters separately, or print the prepareStmt's name and parameters, it's better than not print

currently， print contents are ExecuteStmt$toSql， only contain prepareStmt's name and parameters. we have run some tests, found that will influence performance. so here add a config iterm

OK, get it, and I think it's another question? we can split PR to reslove print log performance? because other statements may need to handle too

or can you commit a issuse to describe the performance question and your code, it's can help us to optimize it late

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

fe/fe-core/src/main/java/com/starrocks/sql/ast/StatementBase.java

...ore/src/main/java/com/starrocks/sql/optimizer/transformer/SqlToScalarOperatorTranslator.java

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

Seaven · 2023-09-08T03:07:13Z

fe/fe-core/src/main/java/com/starrocks/sql/analyzer/ExpressionAnalyzer.java

+
+        @Override
+        public Void visitParameterExpr(Parameter node, Scope context) {
+            return null;


should analyze actual expr here

but other expression will depend on child's class type(like lambda/date func), I think it's can't work good, and will throw some expcetion

Signed-off-by: jukejian <jukejian@bytedance.com>

sonarqubecloud · 2023-09-13T02:32:34Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
45 Code Smells

0.0% Coverage
0.5% Duplication

wanpengfei-git · 2023-09-13T04:48:05Z

[FE Incremental Coverage Report]

😞 fail : 50 / 314 (15.92%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/ast/AstVisitor.java	0	4	00.00%	[86, 90, 94, 1209]
🔵	com/starrocks/analysis/DecimalLiteral.java	0	10	00.00%	[529, 532, 533, 534, 535, 536, 537, 538, 539, 540]
🔵	com/starrocks/analysis/StringLiteral.java	0	7	00.00%	[274, 275, 276, 278, 279, 280, 281]
🔵	com/starrocks/sql/ast/ExecuteStmt.java	0	10	00.00%	[29, 30, 31, 32, 35, 39, 44, 45, 50, 55]
🔵	com/starrocks/analysis/FloatLiteral.java	0	5	00.00%	[246, 247, 248, 249, 251]
🔵	com/starrocks/qe/PrepareStmtContext.java	0	8	00.00%	[25, 26, 27, 28, 29, 32, 36, 40]
🔵	com/starrocks/analysis/DateLiteral.java	0	31	00.00%	[459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 470, 472, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 485, 486, 487, 488, 490, 491, 493, 496, 499]
🔵	com/starrocks/analysis/IntLiteral.java	0	11	00.00%	[406, 408, 409, 411, 412, 414, 415, 417, 418, 420, 422]
🔵	com/starrocks/sql/optimizer/transformer/SqlToScalarOperatorTranslator.java	0	3	00.00%	[295, 296, 298]
🔵	com/starrocks/sql/ast/DeallocateStmt.java	0	7	00.00%	[23, 24, 25, 28, 33, 38, 43]
🔵	com/starrocks/sql/analyzer/ExpressionAnalyzer.java	0	1	00.00%	[1814]
🔵	com/starrocks/qe/StmtExecutor.java	1	47	02.13%	[459, 460, 461, 462, 463, 464, 466, 467, 468, 469, 675, 676, 677, 678, 1487, 1488, 1489, 1490, 1492, 1493, 1494, 1497, 1498, 1500, 1501, 1502, 1503, 1505, 1509, 1511, 1513, 1515, 1516, 1518, 1519, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 1532, 1533]
🔵	com/starrocks/qe/ConnectProcessor.java	1	42	02.38%	[361, 484, 486, 488, 489, 491, 492, 493, 494, 496, 498, 499, 501, 502, 505, 507, 510, 511, 512, 513, 514, 516, 517, 518, 520, 522, 523, 524, 526, 527, 528, 530, 531, 533, 536, 537, 538, 539, 542, 584, 585]
🔵	com/starrocks/analysis/Parameter.java	3	20	15.00%	[29, 30, 31, 32, 35, 39, 43, 44, 48, 53, 58, 59, 61, 66, 67, 69, 74]
🔵	com/starrocks/qe/ConnectContext.java	1	6	16.67%	[261, 262, 265, 269, 270]
🔵	com/starrocks/sql/ast/PrepareStmt.java	8	24	33.33%	[46, 50, 51, 58, 59, 62, 63, 64, 66, 70, 71, 72, 73, 74, 79, 89]
🔵	com/starrocks/sql/analyzer/PrepareAnalyzer.java	6	17	35.29%	[37, 42, 43, 44, 45, 47, 48, 49, 50, 53, 54]
🔵	com/starrocks/analysis/LiteralExpr.java	27	55	49.09%	[258, 262, 263, 264, 266, 295, 296, 319, 320, 321, 324, 325, 326, 327, 330, 331, 332, 333, 336, 337, 345, 346, 348, 349, 350, 351, 352, 354]
🔵	com/starrocks/sql/analyzer/AnalyzerVisitor.java	2	4	50.00%	[893, 894]
🔵	com/starrocks/qe/SessionVariable.java	1	2	50.00%	[2544]

wanpengfei-git · 2023-09-13T04:48:22Z

[BE Incremental Coverage Report]

😞 fail : 3 / 62 (04.84%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/result_sink_operator.h	0	4	00.00%	[97, 115, 116, 117]
🔵	src/exec/pipeline/fragment_executor.cpp	0	3	00.00%	[769, 770, 771]
🔵	src/runtime/result_sink.h	0	1	00.00%	[82]
🔵	src/runtime/mysql_result_writer.cpp	0	7	00.00%	[50, 55, 67, 141, 142, 186, 187]
🔵	src/util/mysql_row_buffer.h	0	1	00.00%	[48]
🔵	src/runtime/result_sink.cpp	0	3	00.00%	[73, 103, 104]
🔵	src/exec/pipeline/result_sink_operator.cpp	0	2	00.00%	[39, 40]
🔵	src/util/mysql_row_buffer.cpp	3	41	07.32%	[83, 84, 86, 87, 88, 89, 101, 102, 104, 105, 106, 108, 109, 110, 112, 113, 114, 116, 117, 118, 120, 121, 122, 124, 125, 126, 128, 129, 140, 188, 213, 214, 314, 315, 316, 317, 318, 319]

decster · 2023-09-13T07:00:23Z

@mergify backport branch-3.2

mergify · 2023-09-13T07:00:33Z

backport branch-3.2

✅ Backports have been created

#30963 [Feature] support prepare statement (backport #27840) has been created for branch branch-3.2

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org> (cherry picked from commit 264b6dc)

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org> (cherry picked from commit 264b6dc) Signed-off-by: Binglin Chang <decstery@gmail.com>

Signed-off-by: jukejian <jukejian@bytedance.com> Signed-off-by: Binglin Chang <decstery@gmail.com> Co-authored-by: jay <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

Jay-ju requested a review from a team as a code owner July 24, 2023 12:07

mergify bot assigned Jay-ju Jul 24, 2023

Jay-ju changed the title ~~[Feature] add prepare statement~~ [Feature] support prepare statement Jul 24, 2023

imay reviewed Jul 24, 2023

View reviewed changes

Jay-ju force-pushed the prepare_statement branch 3 times, most recently from 44eed3c to 04de190 Compare July 26, 2023 09:28

decster reviewed Jul 28, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/analysis/LiteralExpr.java Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/analysis/Parameter.java Outdated Show resolved Hide resolved

fe/fe-core/src/test/java/com/starrocks/analysis/PreparedStmtTest.java Show resolved Hide resolved

decster reviewed Jul 28, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java Show resolved Hide resolved

Jay-ju force-pushed the prepare_statement branch 3 times, most recently from c82c7a9 to b407652 Compare August 4, 2023 05:22

packy92 reviewed Aug 4, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java Show resolved Hide resolved

packy92 previously approved these changes Aug 4, 2023

View reviewed changes

Seaven reviewed Aug 25, 2023

View reviewed changes

Jay-ju force-pushed the prepare_statement branch from be64ecd to 9f4464d Compare September 7, 2023 04:57

wanpengfei-git added the documentation Improvements or additions to documentation label Sep 7, 2023

github-actions bot added the behavior_changed label Sep 7, 2023

Jay-ju force-pushed the prepare_statement branch 3 times, most recently from e4d620f to 830776c Compare September 7, 2023 09:25

Seaven reviewed Sep 8, 2023

View reviewed changes

Jay-ju force-pushed the prepare_statement branch 2 times, most recently from 14d9532 to 192ea6c Compare September 11, 2023 12:59

packy92 previously approved these changes Sep 12, 2023

View reviewed changes

Jay-ju dismissed packy92’s stale review via 5d96437 September 13, 2023 02:04

Jay-ju and others added 8 commits September 13, 2023 10:05

fix main conflict

5338e4b

Signed-off-by: jukejian <jukejian@bytedance.com>

fix mysql type mapping

6f6d7ff

Signed-off-by: jukejian <jukejian@bytedance.com>

fix thread unsafe LiteralExpr

418c8ab

Signed-off-by: jukejian <jukejian@bytedance.com>

fix mysql result for jdbc request

4faaa7c

Signed-off-by: jukejian <jukejian@bytedance.com>

add use case about order by

1cd1f52

Signed-off-by: jukejian <jukejian@bytedance.com>

for jdbc request

e0b3af5

Signed-off-by: jukejian <jukejian@bytedance.com>

fix code style

3cd39f6

Signed-off-by: jukejian <jukejian@bytedance.com>

fix byte check

6ca9be4

Signed-off-by: jukejian <jukejian@bytedance.com>

Jay-ju force-pushed the prepare_statement branch from 5d96437 to 6ca9be4 Compare September 13, 2023 02:05

Seaven approved these changes Sep 13, 2023

View reviewed changes

Seaven enabled auto-merge (squash) September 13, 2023 03:51

packy92 approved these changes Sep 13, 2023

View reviewed changes

github-actions bot removed the behavior_changed label Sep 13, 2023

Seaven merged commit 264b6dc into StarRocks:main Sep 13, 2023

github-actions bot added behavior_changed and removed behavior_changed labels Sep 13, 2023

mergify bot pushed a commit that referenced this pull request Sep 13, 2023

[Feature] support prepare statement (#27840)

556797c

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org> (cherry picked from commit 264b6dc)

mergify bot mentioned this pull request Sep 13, 2023

[Feature] support prepare statement (backport #27840) #30963

Merged

caneGuy pushed a commit to caneGuy/starrocks that referenced this pull request Nov 8, 2023

[BackPort][Feature] support prepare statement (StarRocks#27840)

c9db1a7

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

caneGuy pushed a commit to caneGuy/starrocks that referenced this pull request Nov 8, 2023

[BackPort][Feature] support prepare statement (StarRocks#27840)

e8809e4

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

caneGuy pushed a commit to caneGuy/starrocks that referenced this pull request Nov 9, 2023

[BackPort][Feature] support prepare statement (StarRocks#27840)

ca5af62

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

caneGuy pushed a commit to caneGuy/starrocks that referenced this pull request Mar 22, 2024

[BackPort][Feature] support prepare statement (StarRocks#27840)

2cb2f80

Signed-off-by: jukejian <jukejian@bytedance.com> Co-authored-by: root <root@n37-042-050.byted.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] support prepare statement #27840

[Feature] support prepare statement #27840

Jay-ju commented Jul 24, 2023 •

edited by decster

Loading

imay Jul 24, 2023

liuyehcf commented Jul 25, 2023

wanpengfei-git commented Jul 26, 2023

Jay-ju commented Aug 3, 2023 •

edited

Loading

sonarqubecloud bot commented Aug 4, 2023

packy92 Aug 4, 2023 •

edited

Loading

Jay-ju Aug 10, 2023

Seaven Aug 25, 2023 •

edited

Loading

Jay-ju Sep 6, 2023

Seaven Sep 8, 2023

Seaven Sep 8, 2023

Seaven Sep 8, 2023 •

edited

Loading

Seaven Sep 8, 2023 •

edited

Loading

sonarqubecloud bot commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

decster commented Sep 13, 2023

mergify bot commented Sep 13, 2023 •

edited

Loading

		@@ -0,0 +1,107 @@
		// Copyright 2021-present StarRocks, Inc. All rights reserved.

[Feature] support prepare statement #27840

[Feature] support prepare statement #27840

Conversation

Jay-ju commented Jul 24, 2023 • edited by decster Loading

What type of PR is this:

Does this PR entail a change in behavior?

Checklist:

Bugfix cherry-pick branch check:

imay Jul 24, 2023

Choose a reason for hiding this comment

liuyehcf commented Jul 25, 2023

wanpengfei-git commented Jul 26, 2023

[FE PR Coverage Check]

file detail

Jay-ju commented Aug 3, 2023 • edited Loading

sonarqubecloud bot commented Aug 4, 2023

packy92 Aug 4, 2023 • edited Loading

Choose a reason for hiding this comment

Jay-ju Aug 10, 2023

Choose a reason for hiding this comment

Seaven Aug 25, 2023 • edited Loading

Choose a reason for hiding this comment

Jay-ju Sep 6, 2023

Choose a reason for hiding this comment

Seaven Sep 8, 2023

Choose a reason for hiding this comment

Seaven Sep 8, 2023

Choose a reason for hiding this comment

Seaven Sep 8, 2023 • edited Loading

Choose a reason for hiding this comment

Seaven Sep 8, 2023 • edited Loading

Choose a reason for hiding this comment

sonarqubecloud bot commented Sep 13, 2023

wanpengfei-git commented Sep 13, 2023

[FE Incremental Coverage Report]

file detail

wanpengfei-git commented Sep 13, 2023

[BE Incremental Coverage Report]

file detail

decster commented Sep 13, 2023

mergify bot commented Sep 13, 2023 • edited Loading

✅ Backports have been created

Jay-ju commented Jul 24, 2023 •

edited by decster

Loading

Jay-ju commented Aug 3, 2023 •

edited

Loading

packy92 Aug 4, 2023 •

edited

Loading

Seaven Aug 25, 2023 •

edited

Loading

Seaven Sep 8, 2023 •

edited

Loading

Seaven Sep 8, 2023 •

edited

Loading

mergify bot commented Sep 13, 2023 •

edited

Loading