Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](log) print detail error for segment compaction failure #30503

Merged
merged 2 commits into from
Jan 30, 2024
Merged

[enhancement](log) print detail error for segment compaction failure #30503

merged 2 commits into from
Jan 30, 2024

Conversation

zhannngchen
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

print detail error for segment compaction failure

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring previously approved these changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37600 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45c02519f21439f727cb148cfc09be920d1479f2, data reload: false

------ Round 1 ----------------------------------
q1	17908	4962	5053	4962
q2	2474	147	144	144
q3	11038	954	922	922
q4	5078	733	728	728
q5	7830	2873	2974	2873
q6	189	123	124	123
q7	1165	741	703	703
q8	9264	1981	2021	1981
q9	7203	6262	6288	6262
q10	8102	2460	2447	2447
q11	436	205	211	205
q12	777	292	291	291
q13	17988	3311	3259	3259
q14	271	240	243	240
q15	524	497	485	485
q16	454	398	411	398
q17	929	535	488	488
q18	6980	6140	6099	6099
q19	1546	1406	1301	1301
q20	614	366	358	358
q21	6676	3121	3029	3029
q22	824	316	302	302
Total cold run time: 108270 ms
Total hot run time: 37600 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5118	4864	5054	4864
q2	336	230	238	230
q3	2979	2643	2744	2643
q4	1883	1517	1582	1517
q5	5372	5289	5262	5262
q6	204	113	116	113
q7	2101	1674	1804	1674
q8	3085	3199	3275	3199
q9	8479	8668	13300	8668
q10	5574	3453	3435	3435
q11	543	459	456	456
q12	767	559	582	559
q13	16966	3143	3096	3096
q14	278	259	262	259
q15	534	489	490	489
q16	523	482	488	482
q17	1993	1627	1715	1627
q18	8922	10410	8291	8291
q19	24066	1543	1523	1523
q20	4584	1913	1896	1896
q21	15015	4835	4801	4801
q22	1048	462	447	447
Total cold run time: 110370 ms
Total hot run time: 55531 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8620/23823)
Line Coverage: 28.25% (70518/249585)
Region Coverage: 27.24% (36377/133522)
Branch Coverage: 24.05% (18646/77536)
Coverage Report: http://coverage.selectdb-in.cc/coverage/45c02519f21439f727cb148cfc09be920d1479f2_45c02519f21439f727cb148cfc09be920d1479f2/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 173124 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45c02519f21439f727cb148cfc09be920d1479f2, data reload: false

query1	947	336	330	330
query2	6565	2070	1864	1864
query3	6690	204	197	197
query4	32188	22115	22183	22115
query5	4454	443	343	343
query6	241	157	156	156
query7	4600	258	263	258
query8	261	177	193	177
query9	8527	2270	2260	2260
query10	407	198	194	194
query11	17881	15407	15396	15396
query12	120	69	69	69
query13	1671	365	379	365
query14	9214	6458	6423	6423
query15	201	177	179	177
query16	5698	259	250	250
query17	947	473	476	473
query18	1817	260	252	252
query19	178	137	135	135
query20	69	69	69	69
query21	205	133	133	133
query22	4868	4715	4706	4706
query23	31195	30328	30376	30328
query24	12530	2748	2749	2748
query25	598	309	305	305
query26	1997	138	134	134
query27	3171	285	282	282
query28	7665	1827	1802	1802
query29	2233	620	607	607
query30	284	135	135	135
query31	905	714	710	710
query32	71	53	47	47
query33	696	208	203	203
query34	1183	456	458	456
query35	845	775	768	768
query36	1429	1263	1160	1160
query37	177	59	59	59
query38	3278	3195	3166	3166
query39	1328	1248	1242	1242
query40	346	86	82	82
query41	37	35	34	34
query42	87	77	81	77
query43	519	466	496	466
query44	1108	698	692	692
query45	190	177	176	176
query46	1065	648	649	648
query47	1628	1543	1568	1543
query48	375	311	302	302
query49	1230	282	273	273
query50	674	318	318	318
query51	5312	5187	5143	5143
query52	96	74	76	74
query53	330	258	259	258
query54	259	185	191	185
query55	76	73	71	71
query56	175	158	162	158
query57	1020	923	919	919
query58	179	155	148	148
query59	2793	2523	2549	2523
query60	204	175	175	175
query61	83	82	80	80
query62	669	365	344	344
query63	267	247	266	247
query64	6002	1791	1743	1743
query65	3284	3277	3200	3200
query66	1479	315	310	310
query67	14220	14298	14183	14183
query68	4035	482	492	482
query69	420	310	297	297
query70	1545	1546	1578	1546
query71	249	224	211	211
query72	3723	2802	2854	2802
query73	682	318	308	308
query74	6720	6334	6333	6333
query75	2799	2269	2251	2251
query76	2622	1001	941	941
query77	343	225	224	224
query78	9158	8691	8450	8450
query79	2841	504	503	503
query80	1980	321	316	316
query81	510	195	192	192
query82	680	86	82	82
query83	292	119	111	111
query84	281	76	76	76
query85	2122	333	314	314
query86	495	410	432	410
query87	3449	3251	3255	3251
query88	3667	2191	2180	2180
query89	407	345	349	345
query90	1858	182	182	182
query91	143	117	117	117
query92	53	43	47	43
query93	3714	416	398	398
query94	1391	157	159	157
query95	500	462	444	444
query96	611	324	312	312
query97	4254	4133	4120	4120
query98	210	195	182	182
query99	1105	689	678	678
Total cold run time: 283900 ms
Total hot run time: 173124 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45c02519f21439f727cb148cfc09be920d1479f2, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.23	0.05	0.06
query4	1.68	0.10	0.09
query5	0.53	0.51	0.51
query6	1.20	0.64	0.65
query7	0.02	0.02	0.01
query8	0.03	0.02	0.03
query9	0.54	0.50	0.50
query10	0.54	0.54	0.55
query11	0.11	0.08	0.08
query12	0.10	0.09	0.10
query13	0.61	0.61	0.61
query14	0.78	0.80	0.81
query15	0.79	0.77	0.77
query16	0.38	0.38	0.39
query17	1.01	0.97	1.02
query18	0.22	0.25	0.23
query19	1.92	1.76	1.75
query20	0.01	0.00	0.01
query21	15.44	0.59	0.57
query22	2.59	2.98	1.70
query23	17.20	0.76	0.83
query24	2.66	0.82	1.25
query25	0.34	0.14	0.09
query26	0.52	0.14	0.14
query27	0.05	0.05	0.05
query28	11.33	0.84	0.84
query29	12.49	3.26	3.22
query30	0.64	0.50	0.53
query31	2.78	0.35	0.34
query32	3.37	0.48	0.48
query33	3.23	3.23	3.26
query34	16.02	4.32	4.29
query35	4.39	4.33	4.26
query36	1.08	1.08	1.05
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.01	0.02
query42	0.03	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.33 s
Total hot run time: 30.69 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 45c02519f21439f727cb148cfc09be920d1479f2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@zhannngchen
Copy link
Contributor Author

run buildall

dataroaring
dataroaring approved these changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.25% (70522/249593)
Region Coverage: 27.24% (36375/133526)
Branch Coverage: 24.05% (18645/77542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4f8d1678f23c780c95b8fec8760678c1e083647a_4f8d1678f23c780c95b8fec8760678c1e083647a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f8d1678f23c780c95b8fec8760678c1e083647a, data reload: false

------ Round 1 ----------------------------------
q1	17722	4855	5088	4855
q2	2050	139	134	134
q3	10706	924	938	924
q4	4732	661	718	661
q5	7660	2841	2812	2812
q6	185	120	119	119
q7	1137	717	713	713
q8	9388	1987	2017	1987
q9	7245	6329	6329	6329
q10	8080	2417	2416	2416
q11	412	216	204	204
q12	734	282	282	282
q13	17989	3284	3271	3271
q14	272	245	236	236
q15	528	514	491	491
q16	455	386	398	386
q17	935	574	517	517
q18	6829	6195	6032	6032
q19	1569	1320	1326	1320
q20	617	335	338	335
q21	7068	3067	3187	3067
q22	811	296	297	296
Total cold run time: 107124 ms
Total hot run time: 37387 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5103	4862	5111	4862
q2	343	227	237	227
q3	3013	2641	2746	2641
q4	1759	1530	1526	1526
q5	5401	5153	5064	5064
q6	199	114	117	114
q7	2061	1793	1818	1793
q8	3116	3183	3157	3157
q9	8616	8666	13225	8666
q10	5275	3497	3487	3487
q11	570	462	474	462
q12	761	583	592	583
q13	16941	3136	3072	3072
q14	272	255	250	250
q15	530	488	488	488
q16	526	488	457	457
q17	1987	1636	1687	1636
q18	8891	9853	8757	8757
q19	23967	1585	1512	1512
q20	4649	1912	1895	1895
q21	14886	4920	4912	4912
q22	1031	463	471	463
Total cold run time: 109897 ms
Total hot run time: 56024 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f8d1678f23c780c95b8fec8760678c1e083647a, data reload: false

query1	937	330	323	323
query2	6600	2032	1811	1811
query3	6702	198	196	196
query4	33008	22135	21964	21964
query5	4454	344	355	344
query6	247	148	148	148
query7	4629	259	254	254
query8	245	169	175	169
query9	8561	2272	2262	2262
query10	411	197	188	188
query11	17779	15482	15309	15309
query12	122	72	68	68
query13	1684	365	365	365
query14	9177	6396	6425	6396
query15	212	172	179	172
query16	5701	246	246	246
query17	960	471	462	462
query18	1805	268	250	250
query19	182	132	136	132
query20	78	76	65	65
query21	199	132	128	128
query22	4889	4831	4765	4765
query23	31117	30358	30683	30358
query24	11665	2774	2793	2774
query25	544	307	302	302
query26	1126	139	134	134
query27	3040	281	281	281
query28	7205	1830	1813	1813
query29	1001	609	604	604
query30	284	141	134	134
query31	908	720	709	709
query32	73	52	48	48
query33	684	212	204	204
query34	1152	436	450	436
query35	876	745	756	745
query36	1460	1281	1216	1216
query37	90	54	60	54
query38	3261	3191	3213	3191
query39	1320	1257	1233	1233
query40	194	86	82	82
query41	38	37	37	37
query42	91	84	82	82
query43	510	498	478	478
query44	1112	688	691	688
query45	199	175	174	174
query46	1072	643	637	637
query47	1608	1523	1545	1523
query48	389	296	306	296
query49	1105	282	278	278
query50	700	301	300	300
query51	5234	5095	5119	5095
query52	82	85	74	74
query53	323	254	253	253
query54	253	178	190	178
query55	78	69	75	69
query56	171	164	162	162
query57	985	895	936	895
query58	183	148	152	148
query59	2595	2599	2567	2567
query60	204	181	168	168
query61	80	81	79	79
query62	621	351	339	339
query63	285	257	263	257
query64	4566	1732	1749	1732
query65	3289	3232	3209	3209
query66	1144	315	311	311
query67	14764	14131	14187	14131
query68	3731	499	486	486
query69	447	294	293	293
query70	1585	1586	1537	1537
query71	253	207	209	207
query72	3773	2784	2843	2784
query73	688	309	305	305
query74	6635	6313	6346	6313
query75	2815	2278	2318	2278
query76	2611	1012	982	982
query77	371	231	221	221
query78	9034	8878	8491	8491
query79	2893	491	479	479
query80	1908	317	320	317
query81	532	194	199	194
query82	692	90	74	74
query83	237	108	117	108
query84	278	73	73	73
query85	2197	337	324	324
query86	513	400	387	387
query87	3424	3242	3315	3242
query88	3736	2170	2151	2151
query89	421	361	358	358
query90	1855	185	179	179
query91	145	118	114	114
query92	50	41	44	41
query93	2334	425	403	403
query94	1398	159	156	156
query95	481	444	457	444
query96	613	324	320	320
query97	4246	4127	4136	4127
query98	208	194	218	194
query99	1053	699	746	699
Total cold run time: 277420 ms
Total hot run time: 172776 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f8d1678f23c780c95b8fec8760678c1e083647a, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.05	0.06
query4	1.68	0.09	0.09
query5	0.52	0.51	0.51
query6	1.23	0.62	0.65
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.50	0.51
query10	0.57	0.56	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.60	0.61	0.61
query14	0.77	0.80	0.80
query15	0.78	0.78	0.77
query16	0.39	0.38	0.38
query17	1.04	0.98	0.99
query18	0.23	0.27	0.25
query19	1.90	1.77	1.74
query20	0.02	0.01	0.01
query21	15.41	0.58	0.56
query22	2.38	2.29	1.54
query23	17.20	1.03	0.74
query24	2.37	0.72	0.12
query25	0.19	0.30	0.09
query26	0.37	0.14	0.14
query27	0.04	0.06	0.07
query28	13.16	0.84	0.83
query29	12.75	3.17	3.20
query30	0.65	0.56	0.53
query31	2.79	0.36	0.35
query32	3.35	0.48	0.47
query33	3.25	3.19	3.18
query34	15.80	4.25	4.25
query35	4.32	4.30	4.32
query36	1.13	1.07	1.06
query37	0.07	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.12	0.12
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 106.43 s
Total hot run time: 29.73 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4f8d1678f23c780c95b8fec8760678c1e083647a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@zhannngchen zhannngchen merged commit a2a6f9a into apache:master Jan 30, 2024
26 of 27 checks passed
yiguolei pushed a commit that referenced this pull request Jan 30, 2024
@zhannngchen @Doris-Extras
[enhancement](log) print detail error for segment compaction failure (#…
0932dad 
…30503)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5 reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhannngchen @doris-robot @gavinchou @dataroaring