Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](avro) avoid BE crash if avro scanner's dependency jars is mssing #33031

Merged
merged 3 commits into from
Apr 1, 2024
Merged

[fix](avro) avoid BE crash if avro scanner's dependency jars is mssing #33031

merged 3 commits into from
Apr 1, 2024

Conversation

morningman
Copy link
Contributor

Proposed changes

  1. Check the return value of avro reader's init_fetch_table_schema_reader()
  2. Also fix a bug but the parse exception of Nereids may suppress the real exception from old planner
    It will result unable to see the real error msg.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

morrySnow
morrySnow previously approved these changes Mar 29, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38669 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edb937ff0b2ca377d8c57e1f803bc0863b78f11f, data reload: false

------ Round 1 ----------------------------------
q1	18003	4432	4223	4223
q2	2108	195	180	180
q3	10540	1111	1169	1111
q4	10638	788	761	761
q5	8212	2854	2811	2811
q6	218	135	133	133
q7	1039	648	613	613
q8	9416	2124	2096	2096
q9	7694	6761	6680	6680
q10	9153	3535	3522	3522
q11	479	241	247	241
q12	453	216	225	216
q13	17788	2966	3031	2966
q14	268	232	235	232
q15	532	471	471	471
q16	531	394	377	377
q17	960	562	616	562
q18	7486	6756	6692	6692
q19	5297	1472	1430	1430
q20	660	310	295	295
q21	3445	2744	2802	2744
q22	360	313	329	313
Total cold run time: 115280 ms
Total hot run time: 38669 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4210	4151	4151
q2	363	275	272	272
q3	2963	2708	2764	2708
q4	1876	1583	1531	1531
q5	5295	5250	5254	5250
q6	208	122	125	122
q7	2235	1865	1827	1827
q8	3200	3323	3311	3311
q9	8613	8587	8586	8586
q10	3900	3677	3675	3675
q11	571	471	484	471
q12	761	599	569	569
q13	17784	2919	2952	2919
q14	317	260	268	260
q15	502	461	466	461
q16	473	430	434	430
q17	1781	1455	1442	1442
q18	7591	7309	7412	7309
q19	1617	1482	1512	1482
q20	1955	1764	1734	1734
q21	5023	4663	4838	4663
q22	518	438	448	438
Total cold run time: 71856 ms
Total hot run time: 53611 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edb937ff0b2ca377d8c57e1f803bc0863b78f11f, data reload: false

query1	906	372	1119	372
query2	7335	1936	1903	1903
query3	6664	220	228	220
query4	24646	21294	21336	21294
query5	4265	415	388	388
query6	260	197	169	169
query7	4596	301	298	298
query8	224	172	170	170
query9	8418	2255	2239	2239
query10	527	230	244	230
query11	14903	14221	14250	14221
query12	144	92	89	89
query13	1636	369	374	369
query14	10802	6711	6861	6711
query15	197	189	181	181
query16	6523	277	270	270
query17	1392	585	560	560
query18	1316	291	288	288
query19	209	161	156	156
query20	103	92	88	88
query21	203	127	133	127
query22	4992	4846	4820	4820
query23	33424	33189	32899	32899
query24	12428	2846	2816	2816
query25	639	376	391	376
query26	1501	155	156	155
query27	3363	322	315	315
query28	7620	1860	1832	1832
query29	888	631	622	622
query30	297	1073	149	149
query31	982	719	720	719
query32	85	56	57	56
query33	692	257	255	255
query34	1128	479	499	479
query35	850	679	692	679
query36	1085	905	896	896
query37	218	73	69	69
query38	3572	3468	3431	3431
query39	1609	1567	1539	1539
query40	281	128	126	126
query41	51	47	45	45
query42	115	99	99	99
query43	489	465	455	455
query44	1134	736	719	719
query45	288	264	272	264
query46	1078	675	690	675
query47	1973	1922	1857	1857
query48	387	295	307	295
query49	1153	360	368	360
query50	792	387	390	387
query51	6892	6820	6710	6710
query52	104	97	91	91
query53	349	281	278	278
query54	297	244	241	241
query55	85	83	82	82
query56	248	235	228	228
query57	1209	1118	1156	1118
query58	232	206	212	206
query59	2797	2726	2634	2634
query60	263	245	255	245
query61	115	115	114	114
query62	671	445	431	431
query63	304	280	276	276
query64	6563	4066	4112	4066
query65	3127	3040	3017	3017
query66	1474	350	340	340
query67	15839	14826	14943	14826
query68	8898	539	541	539
query69	576	306	330	306
query70	1289	1192	1184	1184
query71	511	276	267	267
query72	6569	2617	2439	2439
query73	780	331	337	331
query74	6985	6396	6587	6396
query75	3952	2253	2292	2253
query76	5206	926	1002	926
query77	615	259	260	259
query78	11005	10136	10059	10059
query79	11998	526	531	526
query80	1979	411	409	409
query81	487	221	214	214
query82	219	90	85	85
query83	219	168	170	168
query84	258	80	83	80
query85	1063	263	254	254
query86	353	299	299	299
query87	3713	3500	3509	3500
query88	5961	2427	2434	2427
query89	492	368	370	368
query90	2081	170	177	170
query91	126	98	95	95
query92	63	46	47	46
query93	6763	520	505	505
query94	1424	178	179	178
query95	418	305	305	305
query96	603	267	274	267
query97	2674	2472	2514	2472
query98	240	218	212	212
query99	1031	874	830	830
Total cold run time: 307503 ms
Total hot run time: 181004 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8838/24861)
Line Coverage: 27.28% (72457/265567)
Region Coverage: 26.49% (37493/141540)
Branch Coverage: 23.30% (19118/82048)
Coverage Report: http://coverage.selectdb-in.cc/coverage/edb937ff0b2ca377d8c57e1f803bc0863b78f11f_edb937ff0b2ca377d8c57e1f803bc0863b78f11f/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 29.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edb937ff0b2ca377d8c57e1f803bc0863b78f11f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.08	0.10
query5	0.50	0.52	0.51
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.51
query10	0.54	0.53	0.52
query11	0.15	0.10	0.11
query12	0.13	0.11	0.11
query13	0.58	0.58	0.58
query14	0.76	0.77	0.78
query15	0.82	0.80	0.80
query16	0.35	0.38	0.37
query17	0.92	1.00	1.04
query18	0.22	0.24	0.24
query19	1.74	1.73	1.72
query20	0.01	0.02	0.01
query21	15.56	0.64	0.64
query22	3.11	2.53	1.86
query23	17.25	0.93	0.87
query24	1.18	0.23	0.23
query25	0.09	0.09	0.07
query26	0.24	0.17	0.17
query27	0.08	0.08	0.08
query28	13.96	0.94	0.93
query29	12.56	3.24	3.20
query30	0.27	0.08	0.08
query31	2.83	0.38	0.37
query32	3.29	0.46	0.44
query33	2.82	2.85	2.78
query34	16.88	4.43	4.39
query35	4.44	4.47	4.44
query36	0.58	0.47	0.46
query37	0.17	0.15	0.15
query38	0.15	0.13	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.16
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 106.34 s
Total hot run time: 29.72 s

@xiaokang xiaokang added p0_c and removed dev/2.0.x labels Mar 29, 2024
@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit edb937ff0b2ca377d8c57e1f803bc0863b78f11f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

yiguolei
yiguolei previously approved these changes Mar 30, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@morningman morningman dismissed stale reviews from morrySnow and yiguolei via 20b45f0 March 30, 2024 13:51
@morningman
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8839/24864)
Line Coverage: 27.29% (72484/265611)
Region Coverage: 26.50% (37522/141568)
Branch Coverage: 23.31% (19132/82062)
Coverage Report: http://coverage.selectdb-in.cc/coverage/20b45f0750afb9afb1200b71297a5b855247ee4b_20b45f0750afb9afb1200b71297a5b855247ee4b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38592 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20b45f0750afb9afb1200b71297a5b855247ee4b, data reload: false

------ Round 1 ----------------------------------
q1	17645	4254	4110	4110
q2	2101	192	182	182
q3	10483	1306	1372	1306
q4	10192	843	1072	843
q5	7474	2928	2903	2903
q6	216	132	138	132
q7	1104	647	610	610
q8	9416	1939	2027	1939
q9	6720	6198	6165	6165
q10	8447	3511	3458	3458
q11	419	253	240	240
q12	390	221	216	216
q13	17802	2908	2935	2908
q14	270	242	247	242
q15	532	476	484	476
q16	496	382	377	377
q17	949	921	900	900
q18	7189	6466	6349	6349
q19	1605	1535	1531	1531
q20	621	326	308	308
q21	3500	3094	3112	3094
q22	354	303	321	303
Total cold run time: 107925 ms
Total hot run time: 38592 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4053	4062	4053	4053
q2	345	228	225	225
q3	2937	2952	2929	2929
q4	1880	1853	1838	1838
q5	5210	5234	5216	5216
q6	212	126	127	126
q7	2250	1805	1786	1786
q8	3229	3286	3266	3266
q9	8459	8485	8456	8456
q10	3725	3933	4007	3933
q11	549	474	454	454
q12	766	594	610	594
q13	16437	3126	3051	3051
q14	326	269	273	269
q15	538	508	490	490
q16	526	478	458	458
q17	1795	1756	1749	1749
q18	8133	7606	7552	7552
q19	1689	1665	1661	1661
q20	2031	1806	1825	1806
q21	5296	4932	4951	4932
q22	515	446	450	446
Total cold run time: 70901 ms
Total hot run time: 55290 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182052 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20b45f0750afb9afb1200b71297a5b855247ee4b, data reload: false

query1	1216	1115	1113	1113
query2	6361	1962	1883	1883
query3	6653	216	209	209
query4	24996	21721	21320	21320
query5	4221	419	411	411
query6	277	186	177	177
query7	4616	302	300	300
query8	243	175	185	175
query9	8468	2201	2202	2201
query10	446	260	272	260
query11	15032	14451	14604	14451
query12	152	99	97	97
query13	1637	387	392	387
query14	8421	7084	7045	7045
query15	217	175	176	175
query16	6843	286	290	286
query17	956	622	601	601
query18	1860	292	285	285
query19	212	165	169	165
query20	102	93	96	93
query21	204	135	126	126
query22	4915	4750	4792	4750
query23	33157	32475	32958	32475
query24	11690	3223	3201	3201
query25	674	447	434	434
query26	807	171	171	171
query27	3129	392	386	386
query28	6410	1880	1861	1861
query29	1104	631	614	614
query30	329	161	167	161
query31	1037	784	759	759
query32	102	74	71	71
query33	739	261	262	261
query34	1265	514	529	514
query35	883	743	752	743
query36	1041	839	877	839
query37	130	88	82	82
query38	3754	3579	3624	3579
query39	1648	1557	1623	1557
query40	198	150	150	150
query41	50	49	52	49
query42	123	114	116	114
query43	444	411	408	408
query44	1169	762	739	739
query45	279	262	273	262
query46	1130	848	813	813
query47	1993	1909	1888	1888
query48	410	335	318	318
query49	845	370	382	370
query50	845	427	419	419
query51	6918	6853	6853	6853
query52	117	110	102	102
query53	385	306	317	306
query54	298	250	255	250
query55	93	83	82	82
query56	263	246	242	242
query57	1282	1214	1166	1166
query58	256	245	238	238
query59	2591	2520	2286	2286
query60	267	235	239	235
query61	95	89	93	89
query62	665	459	464	459
query63	311	284	286	284
query64	4685	3424	3085	3085
query65	3073	3027	3028	3027
query66	849	337	330	330
query67	15442	14928	14773	14773
query68	8076	569	576	569
query69	581	337	339	337
query70	1197	1084	1108	1084
query71	510	284	284	284
query72	6493	2600	2434	2434
query73	794	329	326	326
query74	6593	6344	6203	6203
query75	3565	2338	2319	2319
query76	4998	1128	1223	1128
query77	603	253	255	253
query78	11044	10129	10114	10114
query79	9897	540	540	540
query80	1438	449	428	428
query81	503	1077	226	226
query82	761	100	104	100
query83	224	166	163	163
query84	267	89	91	89
query85	1205	294	286	286
query86	412	282	290	282
query87	3710	3470	3464	3464
query88	4167	2338	2321	2321
query89	560	367	361	361
query90	1909	181	182	181
query91	138	118	108	108
query92	65	49	54	49
query93	6946	524	527	524
query94	997	201	199	199
query95	445	333	336	333
query96	609	274	270	270
query97	2647	2530	2487	2487
query98	233	218	222	218
query99	1324	817	828	817
Total cold run time: 293934 ms
Total hot run time: 182052 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20b45f0750afb9afb1200b71297a5b855247ee4b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.06
query4	1.66	0.07	0.08
query5	0.48	0.48	0.48
query6	1.13	0.65	0.65
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.57	0.51	0.52
query10	0.58	0.57	0.56
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.79	0.78
query15	0.87	0.83	0.84
query16	0.36	0.36	0.37
query17	0.97	0.98	1.03
query18	0.25	0.26	0.26
query19	1.84	1.76	1.73
query20	0.02	0.01	0.01
query21	15.53	0.80	0.68
query22	3.71	5.15	1.98
query23	17.56	1.40	1.10
query24	1.90	0.22	0.23
query25	0.13	0.09	0.09
query26	0.29	0.17	0.18
query27	0.08	0.09	0.07
query28	13.48	0.96	0.93
query29	12.55	3.40	3.50
query30	0.28	0.09	0.07
query31	2.83	0.40	0.39
query32	3.25	0.47	0.47
query33	2.80	2.88	2.85
query34	15.50	4.31	4.37
query35	4.34	4.37	4.38
query36	0.69	0.48	0.47
query37	0.20	0.17	0.17
query38	0.17	0.16	0.16
query39	0.05	0.04	0.05
query40	0.18	0.16	0.15
query41	0.09	0.04	0.06
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.53 s
Total hot run time: 30.44 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 20b45f0750afb9afb1200b71297a5b855247ee4b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 3e25370 into apache:master Apr 1, 2024
29 of 32 checks passed
@morningman morningman mentioned this pull request Apr 1, 2024
Merged
morningman added a commit that referenced this pull request Apr 1, 2024
@morningman
[fix](avro) avoid BE crash if avro scanner's dependency jars is mssing …
fee37ff 
…#33031 (#33090)

bp #33031
morningman added a commit to morningman/doris that referenced this pull request Apr 7, 2024
@morningman
[fix](avro) avoid BE crash if avro scanner's dependency jars is mssing (
95da52b 
apache#33031)

1. Check the return value of avro reader's init_fetch_table_schema_reader()
2. Also fix a bug but the parse exception of Nereids may suppress the real exception from old planner
    It will result unable to see the real error msg.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[fix](avro) avoid BE crash if avro scanner's dependency jars is mssing …
3aa8ff5 
…apache#33031 (apache#33090)

bp apache#33031
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DongLiang-0 DongLiang-0 DongLiang-0 approved these changes

@morrySnow morrySnow morrySnow approved these changes

@yiguolei yiguolei yiguolei left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.8-merged dev/2.1.2-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morningman @doris-robot @yiguolei @DongLiang-0 @morrySnow @xiaokang