Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](json) fix be crash while load json data #34283

Merged
merged 1 commit into from
Apr 30, 2024

Conversation

cambyzju
Copy link
Contributor

Proposed changes

Issue Number: close #34280

BE crash while load json data with fuzzy_parse and strip_outer_array opened, and simdjson disabled.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@cambyzju
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1da7f1903ae6545456de5889661a4f8c5050dd8, data reload: false

------ Round 1 ----------------------------------
q1	17610	4322	4228	4228
q2	2006	192	188	188
q3	10451	1243	1182	1182
q4	10204	757	858	757
q5	7546	2790	2787	2787
q6	215	127	133	127
q7	1058	643	618	618
q8	9214	2149	2154	2149
q9	9465	6879	6849	6849
q10	9586	3887	3902	3887
q11	457	260	238	238
q12	498	225	217	217
q13	18205	3167	3061	3061
q14	260	228	229	228
q15	533	471	484	471
q16	531	406	384	384
q17	999	723	755	723
q18	8391	7858	7725	7725
q19	5748	1592	1545	1545
q20	637	327	309	309
q21	5233	3447	4097	3447
q22	341	280	283	280
Total cold run time: 119188 ms
Total hot run time: 41400 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4537	4402	4419	4402
q2	359	279	275	275
q3	3189	2866	2997	2866
q4	1899	1611	1597	1597
q5	5485	5544	5516	5516
q6	214	123	124	123
q7	2373	1967	1965	1965
q8	3265	3451	3409	3409
q9	8880	8910	8953	8910
q10	4004	3728	3869	3728
q11	577	499	495	495
q12	796	630	630	630
q13	16009	3105	3089	3089
q14	310	298	281	281
q15	523	503	485	485
q16	499	452	428	428
q17	1808	1493	1521	1493
q18	7685	7629	7277	7277
q19	1635	1523	1545	1523
q20	1975	1773	1756	1756
q21	9980	4795	4636	4636
q22	543	507	487	487
Total cold run time: 76545 ms
Total hot run time: 55371 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.60% (8932/25092)
Line Coverage: 27.22% (73657/270593)
Region Coverage: 26.41% (38047/144081)
Branch Coverage: 23.19% (19392/83640)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b1da7f1903ae6545456de5889661a4f8c5050dd8_b1da7f1903ae6545456de5889661a4f8c5050dd8/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 184967 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1da7f1903ae6545456de5889661a4f8c5050dd8, data reload: false

query1	909	364	355	355
query2	6447	2344	2202	2202
query3	6657	217	223	217
query4	24200	21280	21341	21280
query5	4167	420	428	420
query6	266	183	188	183
query7	4590	286	286	286
query8	249	194	193	193
query9	8656	2342	2317	2317
query10	457	242	249	242
query11	14666	14205	14416	14205
query12	145	92	88	88
query13	1659	368	370	368
query14	10046	8476	6812	6812
query15	261	194	169	169
query16	8106	259	257	257
query17	1780	567	547	547
query18	1997	284	262	262
query19	340	144	145	144
query20	96	84	83	83
query21	189	129	127	127
query22	5014	4829	4777	4777
query23	34054	33182	33237	33182
query24	10603	2967	2922	2922
query25	558	368	381	368
query26	687	151	143	143
query27	2128	315	312	312
query28	6020	2015	1993	1993
query29	845	599	592	592
query30	247	156	151	151
query31	942	724	744	724
query32	99	52	54	52
query33	697	243	244	243
query34	921	463	470	463
query35	813	660	661	660
query36	1065	887	873	873
query37	108	65	75	65
query38	3207	3010	2993	2993
query39	1564	1548	1567	1548
query40	208	125	129	125
query41	43	38	39	38
query42	107	100	98	98
query43	559	543	515	515
query44	1185	736	744	736
query45	293	257	251	251
query46	1065	706	712	706
query47	1938	1845	1835	1835
query48	371	295	284	284
query49	849	385	402	385
query50	767	368	380	368
query51	6962	6730	6762	6730
query52	98	91	92	91
query53	352	282	283	282
query54	307	241	251	241
query55	81	78	72	72
query56	246	214	217	214
query57	1202	1135	1111	1111
query58	220	199	196	196
query59	3357	3127	3203	3127
query60	276	232	233	232
query61	107	88	103	88
query62	625	440	433	433
query63	301	278	280	278
query64	8231	7106	7091	7091
query65	3116	3028	3107	3028
query66	784	335	346	335
query67	15221	14950	14846	14846
query68	5188	520	523	520
query69	473	298	299	298
query70	1178	1137	1104	1104
query71	407	272	265	265
query72	7460	2787	2412	2412
query73	703	316	322	316
query74	6492	6138	6034	6034
query75	3397	2705	2681	2681
query76	3318	1026	928	928
query77	386	269	271	269
query78	10886	10239	10196	10196
query79	5192	519	509	509
query80	2161	450	425	425
query81	537	223	212	212
query82	938	92	95	92
query83	302	168	170	168
query84	269	85	90	85
query85	1903	281	262	262
query86	468	314	315	314
query87	3253	3079	3070	3070
query88	5184	2323	2325	2323
query89	500	378	378	378
query90	2035	177	179	177
query91	128	96	94	94
query92	61	49	47	47
query93	5382	510	492	492
query94	1278	180	178	178
query95	401	299	307	299
query96	584	272	258	258
query97	3094	2977	2951	2951
query98	241	228	214	214
query99	1217	867	842	842
Total cold run time: 289342 ms
Total hot run time: 184967 ms

@xiaokang xiaokang added the p0_w label Apr 29, 2024
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit c23e422 into apache:master Apr 30, 2024
26 of 29 checks passed
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Apr 30, 2024
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Apr 30, 2024
@xiaokang
Copy link
Contributor

@cambyzju pls add a testcase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.10-merged dev/2.1.3-merged p0_w reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] BE crash while load json data
7 participants