Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](common) If the properties in DDL is not provided, the mysql client will lost connection. #30256

Merged
merged 1 commit into from
Jan 29, 2024
Merged

[fix](common) If the properties in DDL is not provided, the mysql client will lost connection. #30256

merged 1 commit into from
Jan 29, 2024

Conversation

allenhooo
Copy link
Contributor

fix If the properties in DDL is not provided, the mysql client will lost connection, and no error message is output on the client side.

Case

If the properties in DDL is not provided (wrong DDL), the mysql client will lost connection:

mysql> CREATE EXTERNAL RESOURCE 'mysql_v';
ERROR 2013 (HY000): Lost connection to MySQL server during query

The correct error message should appear here.

Here is the stack trace:
’‘’
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Resource properties can't be null
at org.apache.doris.analysis.CreateResourceStmt.analyze(CreateResourceStmt.java:84) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:1080) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:728) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:509) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:462) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2024-01-23 09:31:51,757 WARN (mysql-nio-pool-1|316) [ConnectProcessor.handleQueryException():329] Process one query failed because unknown reason:
java.lang.NullPointerException: Cannot invoke "java.util.Map.entrySet()" because "this.map" is null
at org.apache.doris.common.util.PrintableMap.showEntries(PrintableMap.java:108) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.PrintableMap.toString(PrintableMap.java:97) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.lang.String.valueOf(Unknown Source) ~[?:?]
at java.lang.StringBuilder.append(Unknown Source) ~[?:?]
at org.apache.doris.analysis.CreateResourceStmt.toSql(CreateResourceStmt.java:111) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.AuditLogHelper.logAuditLog(AuditLogHelper.java:101) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.auditAfterExec(ConnectProcessor.java:171) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:267) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
‘’‘
It's because PrintableMap.toString() does not correctly handle the situation where map is null.

The test results are as follows:

'''sql
mysql> CREATE EXTERNAL RESOURCE 'mysql_v';
ERROR 1105 (HY000): errCode = 2, detailMessage = Resource properties can't be null
'''

Further comments

The modification is small, please evaluate the side effects of this modification

@allenhooo
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38709 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9aa2bf7c4c7921c1923f9369d78cbd13a60cf145, data reload: false

------ Round 1 ----------------------------------
q1	17621	5349	5337	5337
q2	2040	139	133	133
q3	10643	1134	1153	1134
q4	10223	758	805	758
q5	7720	3177	3107	3107
q6	201	120	123	120
q7	855	492	481	481
q8	9227	1929	1926	1926
q9	7211	6375	6327	6327
q10	8220	3051	3089	3051
q11	420	222	213	213
q12	348	188	186	186
q13	17982	3328	3339	3328
q14	247	216	214	214
q15	538	507	507	507
q16	470	366	369	366
q17	944	519	486	486
q18	7334	6768	6897	6768
q19	3240	1325	1372	1325
q20	613	317	299	299
q21	2764	2336	2379	2336
q22	348	307	310	307
Total cold run time: 109209 ms
Total hot run time: 38709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5523	5315	5263	5263
q2	319	221	209	209
q3	3335	3253	3246	3246
q4	2070	2088	2057	2057
q5	5990	5896	5943	5896
q6	195	119	117	117
q7	2306	1886	1904	1886
q8	3204	3354	3385	3354
q9	8938	8910	8806	8806
q10	3915	3839	3848	3839
q11	563	458	445	445
q12	814	613	621	613
q13	16942	3143	3137	3137
q14	283	255	287	255
q15	540	508	502	502
q16	507	446	466	446
q17	1865	1857	1892	1857
q18	9571	18455	9513	9513
q19	25351	1628	1517	1517
q20	4595	1931	1939	1931
q21	14380	5408	5375	5375
q22	997	503	572	503
Total cold run time: 112203 ms
Total hot run time: 60767 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9aa2bf7c4c7921c1923f9369d78cbd13a60cf145, data reload: false

query1	968	340	329	329
query2	6556	1986	1977	1977
query3	6706	201	197	197
query4	32014	22200	22140	22140
query5	4452	446	360	360
query6	243	153	164	153
query7	4600	271	260	260
query8	226	169	176	169
query9	8347	2529	2531	2529
query10	411	234	232	232
query11	16926	15572	15530	15530
query12	120	71	65	65
query13	1692	386	395	386
query14	10585	6922	7034	6922
query15	215	176	179	176
query16	5799	257	251	251
query17	945	476	469	469
query18	1791	269	251	251
query19	171	138	132	132
query20	70	71	68	68
query21	186	127	128	127
query22	5142	4907	4936	4907
query23	31368	30887	30813	30813
query24	7970	2816	2809	2809
query25	511	315	314	314
query26	883	145	144	144
query27	2653	293	286	286
query28	3883	1848	1849	1848
query29	909	618	599	599
query30	267	140	141	140
query31	936	750	760	750
query32	73	51	49	49
query33	427	212	209	209
query34	822	453	466	453
query35	887	742	777	742
query36	1304	1157	1250	1157
query37	93	59	62	59
query38	3406	3252	3225	3225
query39	1317	1254	1262	1254
query40	191	88	82	82
query41	37	37	34	34
query42	88	83	77	77
query43	519	499	488	488
query44	1045	690	696	690
query45	191	175	172	172
query46	1063	653	657	653
query47	1686	1517	1515	1515
query48	394	304	309	304
query49	937	280	285	280
query50	691	311	311	311
query51	5258	5156	5119	5119
query52	87	84	74	74
query53	322	253	264	253
query54	205	186	185	185
query55	83	72	73	72
query56	176	174	167	167
query57	1023	943	888	888
query58	186	159	154	154
query59	3060	2760	2683	2683
query60	213	190	197	190
query61	85	81	81	81
query62	499	354	359	354
query63	279	249	253	249
query64	3677	1765	1734	1734
query65	3377	3270	3236	3236
query66	1216	315	328	315
query67	15828	15102	14973	14973
query68	12821	542	513	513
query69	591	311	295	295
query70	1637	1454	1478	1454
query71	10435	10203	10202	10202
query72	5037	2841	2822	2822
query73	2984	306	308	306
query74	7139	6354	6422	6354
query75	5191	2288	2255	2255
query76	6435	1097	1028	1028
query77	670	227	233	227
query78	9646	9039	8728	8728
query79	994	500	501	500
query80	587	331	319	319
query81	475	208	200	200
query82	207	79	78	78
query83	144	118	117	117
query84	280	70	71	70
query85	1101	329	328	328
query86	386	336	377	336
query87	3473	3358	3319	3319
query88	2731	2203	2180	2180
query89	451	374	354	354
query90	2119	187	184	184
query91	161	129	124	124
query92	54	46	44	44
query93	1491	459	449	449
query94	1354	158	160	158
query95	509	477	449	449
query96	624	313	315	313
query97	4246	4125	4150	4125
query98	211	192	189	189
query99	975	730	715	715
Total cold run time: 293449 ms
Total hot run time: 186549 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9aa2bf7c4c7921c1923f9369d78cbd13a60cf145, data reload: false

query1	0.04	0.02	0.02
query2	0.05	0.02	0.03
query3	0.22	0.05	0.05
query4	1.71	0.08	0.07
query5	0.53	0.52	0.52
query6	1.27	0.62	0.63
query7	0.02	0.01	0.01
query8	0.05	0.03	0.02
query9	0.55	0.51	0.50
query10	0.56	0.58	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.60
query14	0.78	0.82	0.82
query15	0.79	0.76	0.76
query16	0.39	0.37	0.39
query17	1.01	1.04	1.03
query18	0.25	0.25	0.25
query19	1.83	1.83	1.80
query20	0.02	0.01	0.01
query21	15.40	0.57	0.57
query22	2.80	2.40	1.63
query23	17.37	0.79	0.89
query24	2.32	1.65	0.84
query25	0.42	0.27	0.24
query26	0.52	0.14	0.14
query27	0.06	0.05	0.06
query28	11.14	0.78	0.77
query29	12.52	3.25	3.25
query30	0.60	0.50	0.49
query31	2.77	0.36	0.36
query32	3.35	0.48	0.50
query33	3.29	3.24	3.24
query34	15.85	4.24	4.31
query35	4.35	4.32	4.28
query36	1.11	1.09	1.08
query37	0.07	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.18 s
Total hot run time: 30.91 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9aa2bf7c4c7921c1923f9369d78cbd13a60cf145 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@LemonLiTree LemonLiTree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 9b8e3ed into apache:master Jan 29, 2024
27 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5 dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants