Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](cloud) dump error file when footer parse failure #40943

Merged
merged 4 commits into from
Sep 29, 2024
Merged

[enhancement](cloud) dump error file when footer parse failure #40943

merged 4 commits into from
Sep 29, 2024

Conversation

freemandealer
Copy link
Contributor

dump error file to local disk in cloud mode for better debug. limit the error file directory to avoid disk capacity problem.

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@freemandealer
Copy link
Contributor Author

run buildall

gavinchou
gavinchou requested changes Sep 18, 2024
View reviewed changes
be/src/olap/rowset/segment_v2/segment.cpp Outdated
return Status::OK();
}

std::string file_path = dir_path + std::to_string(cur_time) + "_" + ss.str();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use this pattern to get rid of duplicated dumps (check existence before read and write)

    std::string file_path = dir_path + this->_rowset_id.to_string() + "_" + segment_id + "_" + offset + "_" + size;

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41728 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 872b7b39c70d2fc518194e82df859f41361c4240, data reload: false

------ Round 1 ----------------------------------
q1	17780	7447	7353	7353
q2	2068	165	163	163
q3	10782	1089	1145	1089
q4	10350	796	757	757
q5	7741	3124	3090	3090
q6	243	149	146	146
q7	1015	618	603	603
q8	9430	2062	2026	2026
q9	6870	6452	6481	6452
q10	7048	2264	2352	2264
q11	434	259	245	245
q12	413	215	216	215
q13	17774	2989	2983	2983
q14	246	215	222	215
q15	579	514	537	514
q16	691	641	604	604
q17	999	844	815	815
q18	7543	6691	6645	6645
q19	1390	1036	1091	1036
q20	575	292	275	275
q21	4010	3238	3328	3238
q22	1107	1024	1000	1000
Total cold run time: 109088 ms
Total hot run time: 41728 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7291	7250	7269	7250
q2	323	241	241	241
q3	3111	3036	2934	2934
q4	2072	1857	1834	1834
q5	5631	5617	5665	5617
q6	237	148	147	147
q7	2231	1788	1795	1788
q8	3324	3430	3446	3430
q9	8830	9061	8866	8866
q10	3453	3523	3464	3464
q11	582	473	488	473
q12	833	632	601	601
q13	10276	3162	3136	3136
q14	304	294	289	289
q15	563	532	544	532
q16	730	671	667	667
q17	1827	1599	1583	1583
q18	8367	7788	7923	7788
q19	1739	1693	1603	1603
q20	2125	1907	1903	1903
q21	5675	5444	5315	5315
q22	1129	1072	1056	1056
Total cold run time: 70653 ms
Total hot run time: 60517 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196374 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 872b7b39c70d2fc518194e82df859f41361c4240, data reload: false

query1	1282	878	877	877
query2	6322	2107	2105	2105
query3	10807	3975	4057	3975
query4	63890	27243	23649	23649
query5	5113	467	467	467
query6	423	175	169	169
query7	5446	307	302	302
query8	327	225	217	217
query9	8838	2622	2615	2615
query10	467	299	283	283
query11	17667	15309	15869	15309
query12	156	102	103	102
query13	1485	410	439	410
query14	11078	7651	7675	7651
query15	228	181	177	177
query16	6486	436	472	436
query17	1174	625	631	625
query18	1500	309	323	309
query19	216	146	145	145
query20	125	111	113	111
query21	214	105	104	104
query22	4729	4600	4857	4600
query23	34602	34096	33944	33944
query24	6002	2946	2838	2838
query25	510	397	407	397
query26	619	158	157	157
query27	1592	282	297	282
query28	4198	2444	2424	2424
query29	670	420	430	420
query30	239	157	156	156
query31	987	762	831	762
query32	73	54	55	54
query33	421	309	295	295
query34	894	489	487	487
query35	830	729	743	729
query36	1072	958	959	958
query37	148	92	92	92
query38	3991	3916	3950	3916
query39	1484	1433	1425	1425
query40	200	96	98	96
query41	52	48	49	48
query42	118	96	96	96
query43	532	473	478	473
query44	1149	807	795	795
query45	197	167	167	167
query46	1131	777	784	777
query47	1902	1821	1857	1821
query48	478	360	358	358
query49	678	387	390	387
query50	826	393	395	393
query51	7034	6977	6925	6925
query52	96	84	83	83
query53	245	181	177	177
query54	566	447	459	447
query55	77	80	78	78
query56	263	264	257	257
query57	1240	1117	1097	1097
query58	217	250	245	245
query59	3263	3105	3087	3087
query60	285	258	265	258
query61	107	105	101	101
query62	770	678	663	663
query63	216	190	186	186
query64	1287	644	621	621
query65	3255	3195	3195	3195
query66	664	290	303	290
query67	15969	15359	15558	15359
query68	3965	584	585	584
query69	702	398	321	321
query70	1165	1096	1097	1096
query71	482	273	276	273
query72	7062	3989	3996	3989
query73	800	324	324	324
query74	9766	9074	9117	9074
query75	3484	2691	2668	2668
query76	3096	851	985	851
query77	678	300	287	287
query78	10033	9189	9206	9189
query79	1140	549	533	533
query80	861	466	438	438
query81	539	241	238	238
query82	239	139	143	139
query83	276	136	141	136
query84	294	82	78	78
query85	916	302	278	278
query86	359	304	303	303
query87	4388	4415	4288	4288
query88	3293	2323	2325	2323
query89	397	280	283	280
query90	2251	195	186	186
query91	179	135	136	135
query92	59	51	49	49
query93	2208	530	542	530
query94	828	299	266	266
query95	352	255	248	248
query96	607	275	275	275
query97	3284	3134	3069	3069
query98	213	199	203	199
query99	1621	1273	1300	1273
Total cold run time: 314659 ms
Total hot run time: 196374 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 872b7b39c70d2fc518194e82df859f41361c4240, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.51	0.52
query6	1.15	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.48
query10	0.58	0.55	0.54
query11	0.14	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.58	0.58
query14	3.08	2.96	3.05
query15	0.90	0.80	0.80
query16	0.37	0.39	0.39
query17	1.06	1.05	1.04
query18	0.19	0.20	0.19
query19	1.89	1.89	1.98
query20	0.01	0.01	0.00
query21	15.35	0.59	0.56
query22	2.66	2.29	1.52
query23	17.45	0.86	0.84
query24	2.42	1.58	0.67
query25	0.30	0.14	0.04
query26	0.41	0.14	0.14
query27	0.04	0.03	0.04
query28	11.10	1.08	1.07
query29	12.52	3.27	3.27
query30	0.24	0.06	0.05
query31	2.87	0.37	0.37
query32	3.31	0.45	0.47
query33	2.96	2.98	3.07
query34	17.03	4.40	4.39
query35	4.50	4.42	4.42
query36	0.68	0.50	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.54 s
Total hot run time: 32.16 s

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41749 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd4483cdd60743f4a23e0b657409f7fcaa58f3b0, data reload: false

------ Round 1 ----------------------------------
q1	17596	7408	7251	7251
q2	2051	175	152	152
q3	10752	1099	1156	1099
q4	10523	808	807	807
q5	7747	3046	3098	3046
q6	234	161	160	160
q7	1018	654	620	620
q8	9513	2058	2058	2058
q9	6852	6404	6426	6404
q10	7047	2248	2314	2248
q11	447	245	260	245
q12	410	220	220	220
q13	17776	2975	3002	2975
q14	251	215	215	215
q15	569	532	540	532
q16	698	619	617	617
q17	987	826	826	826
q18	7269	6689	6765	6689
q19	1396	1062	1043	1043
q20	571	293	281	281
q21	4042	3277	3273	3273
q22	1125	1013	988	988
Total cold run time: 108874 ms
Total hot run time: 41749 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7265	7291	7271	7271
q2	344	228	235	228
q3	3086	2976	2954	2954
q4	2039	1872	1830	1830
q5	5547	5600	5679	5600
q6	240	148	154	148
q7	2194	1826	1843	1826
q8	3341	3447	3423	3423
q9	8852	8949	8820	8820
q10	3451	3427	3497	3427
q11	589	485	484	484
q12	806	596	589	589
q13	10460	3147	3190	3147
q14	313	292	262	262
q15	591	527	519	519
q16	711	683	681	681
q17	1813	1643	1596	1596
q18	8197	7771	7695	7695
q19	1711	1604	1577	1577
q20	2124	1884	1872	1872
q21	5571	5359	5457	5359
q22	1163	1052	1066	1052
Total cold run time: 70408 ms
Total hot run time: 60360 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195793 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd4483cdd60743f4a23e0b657409f7fcaa58f3b0, data reload: false

query1	1267	869	912	869
query2	6410	2144	2084	2084
query3	10785	3932	3879	3879
query4	64975	27008	23538	23538
query5	5242	457	456	456
query6	412	174	164	164
query7	5478	303	296	296
query8	311	233	223	223
query9	8800	2628	2627	2627
query10	500	287	286	286
query11	18009	15270	15769	15270
query12	156	106	108	106
query13	1521	414	413	413
query14	10924	7629	7643	7629
query15	208	187	178	178
query16	6834	422	458	422
query17	1214	628	613	613
query18	1519	312	322	312
query19	227	154	148	148
query20	140	118	116	116
query21	205	103	107	103
query22	4611	4270	4486	4270
query23	34750	33951	34314	33951
query24	6061	2946	2908	2908
query25	504	418	403	403
query26	623	158	159	158
query27	1597	284	290	284
query28	4262	2456	2425	2425
query29	665	434	438	434
query30	239	149	159	149
query31	987	769	772	769
query32	68	54	56	54
query33	439	297	296	296
query34	912	488	490	488
query35	861	746	731	731
query36	1107	948	921	921
query37	150	88	85	85
query38	4028	3951	3906	3906
query39	1483	1460	1426	1426
query40	209	96	96	96
query41	50	49	49	49
query42	119	101	100	100
query43	512	482	492	482
query44	1159	828	795	795
query45	194	165	166	165
query46	1134	764	740	740
query47	1921	1819	1834	1819
query48	461	362	361	361
query49	704	414	419	414
query50	857	403	425	403
query51	7054	6856	6944	6856
query52	100	90	85	85
query53	246	182	174	174
query54	566	450	449	449
query55	78	82	74	74
query56	295	250	256	250
query57	1184	1115	1100	1100
query58	228	233	231	231
query59	3276	3173	3032	3032
query60	288	267	269	267
query61	110	109	109	109
query62	772	663	655	655
query63	209	187	180	180
query64	1326	665	672	665
query65	3273	3194	3167	3167
query66	665	302	307	302
query67	15954	15599	15523	15523
query68	4614	591	569	569
query69	582	333	312	312
query70	1130	1105	1114	1105
query71	435	269	276	269
query72	6812	4127	3914	3914
query73	781	334	343	334
query74	9934	8941	9001	8941
query75	3450	2641	2719	2641
query76	2337	890	965	890
query77	708	298	289	289
query78	9977	9204	9798	9204
query79	1693	541	549	541
query80	847	438	463	438
query81	559	243	249	243
query82	302	146	137	137
query83	332	140	138	138
query84	293	76	73	73
query85	990	295	282	282
query86	382	299	310	299
query87	4508	4371	4379	4371
query88	3342	2340	2312	2312
query89	407	289	287	287
query90	2096	192	188	188
query91	182	149	145	145
query92	66	50	48	48
query93	1291	528	531	528
query94	910	306	291	291
query95	347	255	261	255
query96	605	277	282	277
query97	3222	3117	3073	3073
query98	225	206	194	194
query99	1561	1287	1272	1272
Total cold run time: 316348 ms
Total hot run time: 195793 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd4483cdd60743f4a23e0b657409f7fcaa58f3b0, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.53	0.51	0.54
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.49	0.51
query10	0.55	0.54	0.54
query11	0.15	0.10	0.11
query12	0.14	0.10	0.10
query13	0.62	0.60	0.60
query14	2.96	3.11	2.97
query15	0.89	0.83	0.82
query16	0.38	0.38	0.37
query17	1.04	1.10	1.08
query18	0.20	0.19	0.20
query19	1.96	1.87	2.01
query20	0.01	0.02	0.01
query21	15.33	0.57	0.58
query22	2.60	3.00	1.83
query23	17.41	0.91	0.83
query24	3.25	1.33	0.51
query25	0.25	0.12	0.08
query26	0.31	0.14	0.14
query27	0.05	0.05	0.04
query28	10.80	1.11	1.08
query29	12.52	3.30	3.27
query30	0.25	0.06	0.06
query31	2.88	0.38	0.37
query32	3.29	0.46	0.46
query33	2.98	3.01	3.03
query34	17.08	4.39	4.37
query35	4.43	4.42	4.47
query36	0.66	0.49	0.50
query37	0.07	0.06	0.06
query38	0.04	0.04	0.03
query39	0.02	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.67 s
Total hot run time: 32.47 s

gavinchou
gavinchou previously approved these changes Sep 23, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@freemandealer
Copy link
Contributor Author

run beut

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41887 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04947cbeba1ba6cba17af9f372bf585e717d03e3, data reload: false

------ Round 1 ----------------------------------
q1	17576	7530	7252	7252
q2	2022	279	278	278
q3	12160	1082	1225	1082
q4	10586	804	733	733
q5	7744	3118	3096	3096
q6	235	152	148	148
q7	1023	626	609	609
q8	9425	2100	2051	2051
q9	7235	6468	6469	6468
q10	7013	2256	2342	2256
q11	442	251	249	249
q12	405	215	217	215
q13	17775	2968	2973	2968
q14	250	211	208	208
q15	568	525	517	517
q16	663	612	622	612
q17	973	827	874	827
q18	7245	6668	6655	6655
q19	1414	1092	1036	1036
q20	591	295	286	286
q21	4005	3337	3339	3337
q22	1130	1004	1013	1004
Total cold run time: 110480 ms
Total hot run time: 41887 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7308	7279	7228	7228
q2	358	237	248	237
q3	3092	3002	2973	2973
q4	2081	1827	1753	1753
q5	5703	5608	5736	5608
q6	234	143	149	143
q7	2197	1801	1878	1801
q8	3301	3447	3476	3447
q9	8842	8853	8805	8805
q10	3508	3505	3496	3496
q11	570	482	482	482
q12	815	616	595	595
q13	10622	3118	3178	3118
q14	289	277	274	274
q15	575	525	501	501
q16	704	675	677	675
q17	1869	1605	1568	1568
q18	8299	7816	7791	7791
q19	1739	1565	1614	1565
q20	2121	1898	1910	1898
q21	5558	5358	5315	5315
q22	1143	1079	1071	1071
Total cold run time: 70928 ms
Total hot run time: 60344 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192135 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04947cbeba1ba6cba17af9f372bf585e717d03e3, data reload: false

query1	951	403	386	386
query2	6243	2087	2081	2081
query3	8679	203	203	203
query4	33798	23684	23457	23457
query5	3716	485	503	485
query6	278	166	168	166
query7	4186	305	304	304
query8	280	215	225	215
query9	9515	2668	2675	2668
query10	490	289	270	270
query11	17957	15154	15119	15119
query12	160	96	96	96
query13	1537	432	411	411
query14	9514	7167	7574	7167
query15	250	171	184	171
query16	8120	438	410	410
query17	1700	621	586	586
query18	2184	325	336	325
query19	377	157	151	151
query20	125	117	111	111
query21	211	113	112	112
query22	4638	4591	4758	4591
query23	35069	34644	34494	34494
query24	10842	3034	2862	2862
query25	600	407	408	407
query26	722	163	165	163
query27	2008	287	298	287
query28	6663	2457	2438	2438
query29	736	421	454	421
query30	264	156	156	156
query31	1021	767	782	767
query32	102	52	61	52
query33	643	318	310	310
query34	917	502	488	488
query35	875	725	714	714
query36	1092	950	925	925
query37	145	86	92	86
query38	3991	3930	3920	3920
query39	1482	1411	1417	1411
query40	208	97	95	95
query41	51	48	50	48
query42	110	92	95	92
query43	531	490	482	482
query44	1175	844	813	813
query45	196	164	166	164
query46	1145	820	769	769
query47	1909	1808	1868	1808
query48	477	368	374	368
query49	892	425	412	412
query50	817	416	423	416
query51	7112	6933	6844	6844
query52	101	96	111	96
query53	246	176	177	176
query54	1039	456	469	456
query55	81	78	77	77
query56	314	265	253	253
query57	1212	1092	1084	1084
query58	234	228	236	228
query59	3341	3157	2920	2920
query60	295	272	271	271
query61	104	105	117	105
query62	805	654	663	654
query63	224	181	178	178
query64	4289	651	658	651
query65	3259	3219	3181	3181
query66	783	290	327	290
query67	15970	15568	15579	15568
query68	4423	592	577	577
query69	526	304	301	301
query70	1186	1156	1119	1119
query71	376	272	274	272
query72	7552	4074	3999	3999
query73	776	339	335	335
query74	10234	9171	9092	9092
query75	3445	2660	2666	2660
query76	2827	930	937	930
query77	629	289	297	289
query78	10099	9169	9179	9169
query79	2291	560	565	560
query80	1133	447	451	447
query81	573	244	243	243
query82	627	142	146	142
query83	303	139	138	138
query84	272	83	77	77
query85	1201	290	292	290
query86	420	295	296	295
query87	4547	4366	4333	4333
query88	3419	2350	2350	2350
query89	393	284	280	280
query90	1849	187	189	187
query91	188	146	146	146
query92	59	47	52	47
query93	2466	543	553	543
query94	881	305	302	302
query95	360	265	265	265
query96	624	285	280	280
query97	3260	3113	3104	3104
query98	230	207	191	191
query99	1722	1287	1301	1287
Total cold run time: 297892 ms
Total hot run time: 192135 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04947cbeba1ba6cba17af9f372bf585e717d03e3, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.49	0.50	0.51
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.55	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	3.08	3.04	2.95
query15	0.90	0.83	0.82
query16	0.37	0.37	0.37
query17	1.05	1.04	1.08
query18	0.20	0.19	0.19
query19	1.98	1.80	2.00
query20	0.01	0.00	0.01
query21	15.38	0.62	0.58
query22	2.95	2.17	2.20
query23	17.10	0.95	0.81
query24	3.23	1.62	0.67
query25	0.19	0.05	0.10
query26	0.63	0.15	0.14
query27	0.05	0.05	0.04
query28	10.35	1.09	1.07
query29	12.58	3.23	3.23
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.26	0.47	0.46
query33	2.96	3.00	3.02
query34	17.08	4.39	4.34
query35	4.53	4.42	4.40
query36	0.64	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.73 s
Total hot run time: 32.67 s

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41324 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 310a25b582662cd3deeaec5a2651225a50ca6405, data reload: false

------ Round 1 ----------------------------------
q1	17619	7493	7301	7301
q2	2048	281	275	275
q3	12086	1091	1187	1091
q4	10558	708	716	708
q5	7769	2897	2882	2882
q6	242	152	151	151
q7	997	627	605	605
q8	9404	1953	1978	1953
q9	7727	6492	6457	6457
q10	6995	2282	2305	2282
q11	441	255	255	255
q12	414	234	219	219
q13	17765	3016	2973	2973
q14	255	225	215	215
q15	577	513	524	513
q16	714	615	631	615
q17	996	563	572	563
q18	7179	6733	6745	6733
q19	1404	1054	1029	1029
q20	597	292	288	288
q21	4135	3215	3414	3215
q22	1097	1011	1001	1001
Total cold run time: 111019 ms
Total hot run time: 41324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7203	7223	7255	7223
q2	346	245	244	244
q3	3045	2984	2916	2916
q4	2104	1876	1805	1805
q5	5766	5649	5818	5649
q6	236	146	148	146
q7	2207	1794	1808	1794
q8	3363	3549	3537	3537
q9	8919	8951	8879	8879
q10	3612	3593	3544	3544
q11	591	505	501	501
q12	834	616	628	616
q13	8064	3187	3163	3163
q14	323	273	279	273
q15	585	512	520	512
q16	692	665	685	665
q17	1868	1633	1578	1578
q18	8314	8003	7899	7899
q19	1731	1603	1590	1590
q20	2144	1881	1896	1881
q21	5588	5439	5493	5439
q22	1110	1074	1054	1054
Total cold run time: 68645 ms
Total hot run time: 60908 ms

@freemandealer
Copy link
Contributor Author

run beut

@doris-robot
Copy link

TPC-DS: Total hot run time: 193865 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 310a25b582662cd3deeaec5a2651225a50ca6405, data reload: false

query1	964	386	373	373
query2	6237	2104	2045	2045
query3	8685	208	217	208
query4	33953	23596	23463	23463
query5	3556	456	478	456
query6	265	187	165	165
query7	4198	316	311	311
query8	286	225	223	223
query9	9257	2690	2674	2674
query10	439	310	281	281
query11	17847	15279	15237	15237
query12	149	103	101	101
query13	1536	444	430	430
query14	8724	8013	7701	7701
query15	284	170	179	170
query16	7982	453	458	453
query17	1679	647	591	591
query18	2022	319	340	319
query19	374	157	167	157
query20	122	123	115	115
query21	218	115	111	111
query22	4758	4507	4555	4507
query23	34882	34641	34656	34641
query24	11139	2987	2937	2937
query25	649	421	414	414
query26	1168	167	167	167
query27	2295	300	306	300
query28	7718	2444	2453	2444
query29	840	441	443	441
query30	259	162	162	162
query31	1051	794	793	793
query32	100	54	56	54
query33	769	309	306	306
query34	925	505	509	505
query35	905	751	731	731
query36	1116	957	944	944
query37	157	91	89	89
query38	4073	3998	3938	3938
query39	1495	1449	1403	1403
query40	213	98	113	98
query41	57	50	48	48
query42	119	100	103	100
query43	550	508	508	508
query44	1263	828	808	808
query45	198	169	169	169
query46	1182	741	738	738
query47	1931	1825	1840	1825
query48	469	361	385	361
query49	891	418	425	418
query50	839	431	430	430
query51	7097	6869	6948	6869
query52	106	92	86	86
query53	258	208	185	185
query54	1222	474	468	468
query55	78	80	79	79
query56	306	272	274	272
query57	1240	1098	1095	1095
query58	242	250	231	231
query59	3204	3054	3190	3054
query60	306	270	276	270
query61	106	104	102	102
query62	842	664	642	642
query63	223	194	190	190
query64	3945	654	652	652
query65	3278	3219	3224	3219
query66	766	316	343	316
query67	16183	15681	15632	15632
query68	4412	595	573	573
query69	567	305	310	305
query70	1187	1137	1102	1102
query71	417	278	290	278
query72	7734	3958	4086	3958
query73	769	350	358	350
query74	10332	8913	9074	8913
query75	3874	2712	2650	2650
query76	3396	866	900	866
query77	637	305	317	305
query78	9995	10347	9708	9708
query79	2403	612	638	612
query80	1399	455	458	455
query81	577	244	241	241
query82	578	137	142	137
query83	299	135	144	135
query84	277	81	82	81
query85	744	306	287	287
query86	463	303	308	303
query87	4523	4310	4343	4310
query88	4133	2420	2395	2395
query89	413	294	291	291
query90	1916	190	192	190
query91	184	150	149	149
query92	63	48	50	48
query93	1912	567	569	567
query94	831	300	266	266
query95	356	265	267	265
query96	627	287	283	283
query97	3238	3177	3129	3129
query98	227	212	198	198
query99	1575	1290	1304	1290
Total cold run time: 300248 ms
Total hot run time: 193865 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 310a25b582662cd3deeaec5a2651225a50ca6405, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.54	0.51
query6	1.14	0.73	0.73
query7	0.01	0.01	0.01
query8	0.04	0.04	0.03
query9	0.59	0.50	0.50
query10	0.56	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	3.08	2.97	2.94
query15	0.90	0.84	0.82
query16	0.38	0.38	0.38
query17	1.06	1.08	1.11
query18	0.20	0.19	0.20
query19	1.99	1.85	1.98
query20	0.02	0.01	0.01
query21	15.36	0.60	0.58
query22	2.55	2.55	2.17
query23	17.32	0.88	0.86
query24	2.90	0.96	1.19
query25	0.26	0.18	0.09
query26	0.41	0.13	0.15
query27	0.04	0.03	0.03
query28	10.46	1.10	1.06
query29	12.55	3.28	3.27
query30	0.26	0.06	0.05
query31	2.87	0.37	0.38
query32	3.28	0.47	0.46
query33	2.96	2.99	3.02
query34	17.05	4.44	4.44
query35	4.55	4.47	4.47
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.27 s
Total hot run time: 33.36 s

@freemandealer
[enhancement](cloud) dump error file when footer parse failure
8de5ca7 
dump error file to local disk in cloud mode for better debug.
limit the error file directory to avoid disk capacity problem.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
refine
a1e11f1 
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
add config
566fa7a 
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
merge
f8a5baa 
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 24, 2024
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41313 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8a5baad0d165adb44b1bb349576f5d62e5e8de4, data reload: false

------ Round 1 ----------------------------------
q1	17908	7495	7275	7275
q2	2009	292	291	291
q3	12146	1086	1257	1086
q4	10567	722	781	722
q5	7792	2920	2908	2908
q6	237	155	151	151
q7	998	624	638	624
q8	9365	1993	1998	1993
q9	6522	6408	6428	6408
q10	7008	2305	2289	2289
q11	438	250	244	244
q12	417	224	226	224
q13	17792	3035	3008	3008
q14	244	228	223	223
q15	590	530	537	530
q16	675	641	609	609
q17	990	565	607	565
q18	7236	6577	6747	6577
q19	1401	1075	950	950
q20	559	295	283	283
q21	3994	3393	3374	3374
q22	1093	979	1007	979
Total cold run time: 109981 ms
Total hot run time: 41313 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7330	7224	7236	7224
q2	335	235	228	228
q3	3133	3033	2990	2990
q4	2105	1859	1851	1851
q5	5646	5685	5740	5685
q6	241	152	153	152
q7	2287	1850	1827	1827
q8	3405	3583	3586	3583
q9	9065	9116	9046	9046
q10	3559	3558	3585	3558
q11	595	488	488	488
q12	826	621	641	621
q13	9058	3160	3142	3142
q14	310	283	295	283
q15	594	530	526	526
q16	715	676	674	674
q17	1847	1617	1629	1617
q18	8347	7988	7965	7965
q19	1741	1534	1560	1534
q20	2124	1891	1938	1891
q21	5417	5398	5601	5398
q22	1146	1040	1011	1011
Total cold run time: 69826 ms
Total hot run time: 61294 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.31% (9623/25790)
Line Coverage: 28.71% (79615/277338)
Region Coverage: 28.16% (41178/146242)
Branch Coverage: 24.78% (20976/84646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f8a5baad0d165adb44b1bb349576f5d62e5e8de4_f8a5baad0d165adb44b1bb349576f5d62e5e8de4/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f8a5baad0d165adb44b1bb349576f5d62e5e8de4, data reload: false

query1	946	390	385	385
query2	6372	2116	2004	2004
query3	8686	201	215	201
query4	33730	23561	23442	23442
query5	3374	459	455	455
query6	263	167	164	164
query7	4186	305	312	305
query8	287	238	225	225
query9	9680	2683	2670	2670
query10	490	288	302	288
query11	18061	15312	15326	15312
query12	150	99	98	98
query13	1528	432	416	416
query14	10247	7737	7704	7704
query15	253	175	180	175
query16	7801	529	501	501
query17	1655	608	595	595
query18	1523	318	320	318
query19	379	161	158	158
query20	126	115	124	115
query21	216	113	105	105
query22	4625	4499	4617	4499
query23	35441	34608	34948	34608
query24	10675	2794	2845	2794
query25	599	409	405	405
query26	720	163	169	163
query27	2007	316	315	315
query28	6433	2473	2433	2433
query29	694	437	446	437
query30	253	157	153	153
query31	999	804	777	777
query32	101	54	57	54
query33	651	301	308	301
query34	951	498	490	490
query35	871	741	732	732
query36	1134	925	974	925
query37	147	87	95	87
query38	4057	3880	3908	3880
query39	1476	1444	1416	1416
query40	213	101	99	99
query41	53	47	50	47
query42	114	104	94	94
query43	533	491	498	491
query44	1158	804	783	783
query45	203	166	168	166
query46	1126	720	713	713
query47	1916	1827	1902	1827
query48	479	367	380	367
query49	874	412	416	412
query50	820	402	403	402
query51	7065	6766	6747	6747
query52	102	89	84	84
query53	259	180	188	180
query54	1184	475	471	471
query55	80	77	77	77
query56	282	258	259	258
query57	1217	1122	1104	1104
query58	238	246	247	246
query59	3331	3018	2857	2857
query60	292	265	282	265
query61	99	104	117	104
query62	833	676	681	676
query63	221	188	187	187
query64	3573	651	645	645
query65	3247	3250	3200	3200
query66	788	304	310	304
query67	16110	15426	15596	15426
query68	4531	551	570	551
query69	559	293	289	289
query70	1207	1136	1135	1135
query71	405	279	270	270
query72	7676	4178	4100	4100
query73	785	342	350	342
query74	10444	9028	8968	8968
query75	4067	2653	2713	2653
query76	3511	933	879	879
query77	705	311	296	296
query78	10135	9403	9342	9342
query79	2372	583	591	583
query80	1188	433	453	433
query81	559	242	241	241
query82	621	143	138	138
query83	373	137	133	133
query84	279	84	82	82
query85	1616	312	281	281
query86	455	310	306	306
query87	4451	4295	4365	4295
query88	3872	2367	2353	2353
query89	425	294	289	289
query90	2004	188	185	185
query91	178	142	142	142
query92	64	49	50	49
query93	2542	544	548	544
query94	995	295	298	295
query95	357	252	250	250
query96	622	277	275	275
query97	3231	3164	3141	3141
query98	233	204	199	199
query99	1560	1302	1321	1302
Total cold run time: 300219 ms
Total hot run time: 192678 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8a5baad0d165adb44b1bb349576f5d62e5e8de4, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.02
query3	0.24	0.07	0.07
query4	1.65	0.10	0.10
query5	0.51	0.49	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.56	0.51	0.50
query10	0.56	0.59	0.56
query11	0.15	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.98	2.94	2.96
query15	0.90	0.82	0.83
query16	0.38	0.38	0.41
query17	1.05	1.05	1.04
query18	0.20	0.19	0.19
query19	1.90	1.79	1.96
query20	0.01	0.01	0.01
query21	15.36	0.60	0.58
query22	2.67	3.23	1.34
query23	17.59	0.85	0.74
query24	2.84	1.47	0.39
query25	0.32	0.13	0.09
query26	0.47	0.14	0.14
query27	0.04	0.04	0.04
query28	10.95	1.09	1.08
query29	12.56	3.24	3.24
query30	0.24	0.06	0.06
query31	2.89	0.39	0.37
query32	3.26	0.48	0.47
query33	3.01	3.05	3.02
query34	16.95	4.44	4.49
query35	4.51	4.48	4.50
query36	0.68	0.47	0.49
query37	0.09	0.05	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.97 s
Total hot run time: 31.74 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 5db9b1a into apache:master Sep 29, 2024
23 of 27 checks passed
gavinchou pushed a commit that referenced this pull request Sep 29, 2024
@freemandealer @gavinchou
[enhancement](cloud) dump error file when footer parse failure (#40943)
9899b5d 
dump error file to local disk in cloud mode for better debug. limit the
error file directory to avoid disk capacity problem.
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@freemandealer @cjj2010
[enhancement](cloud) dump error file when footer parse failure (apach…
404b77f 
…e#40943)

dump error file to local disk in cloud mode for better debug. limit the
error file directory to avoid disk capacity problem.
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
@freemandealer @amorynan
[enhancement](cloud) dump error file when footer parse failure (apach…
f32087e 
…e#40943)

dump error file to local disk in cloud mode for better debug. limit the
error file directory to avoid disk capacity problem.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@freemandealer @doris-robot @gavinchou @Yukang-Lian