Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](pipelineX) fix multi be may be missing profiles #29914

Merged
merged 1 commit into from
Jan 14, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Jan 12, 2024

Proposed changes

Reason: There might be multiple BEs performing 'markOneFragmentDone,' whereas the original consideration only accounted for a single BE scenario.

Additionally, for pipelineX, the profiles sent are on a per-Fragment basis, resulting in larger profile sizes for each individual transmission.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Mryange
Copy link
Contributor Author

Mryange commented Jan 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38542 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false

------ Round 1 ----------------------------------
q1	17948	5007	4929	4929
q2	2027	140	126	126
q3	10709	1157	1176	1157
q4	10535	779	838	779
q5	7794	3077	3150	3077
q6	207	122	123	122
q7	868	497	494	494
q8	9272	1954	1983	1954
q9	7318	6466	6431	6431
q10	8283	3184	3153	3153
q11	415	210	235	210
q12	361	202	200	200
q13	18117	3407	3398	3398
q14	250	212	211	211
q15	552	526	501	501
q16	424	387	378	378
q17	936	498	553	498
q18	7277	6701	6700	6700
q19	1588	1286	1467	1286
q20	522	303	300	300
q21	2806	2345	2415	2345
q22	347	306	293	293
Total cold run time: 108556 ms
Total hot run time: 38542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4980	4927	4932	4927
q2	324	195	199	195
q3	3314	3305	3298	3298
q4	2233	2233	2227	2227
q5	5793	5779	5772	5772
q6	192	118	121	118
q7	2285	1838	1864	1838
q8	3434	3527	3524	3524
q9	8827	8789	8756	8756
q10	3746	3808	3789	3789
q11	535	421	420	420
q12	787	595	601	595
q13	6496	3206	3208	3206
q14	293	258	260	258
q15	560	514	509	509
q16	505	475	453	453
q17	1992	1995	2002	1995
q18	8874	8313	8307	8307
q19	1623	1624	1608	1608
q20	2167	1924	1917	1917
q21	6038	5717	5630	5630
q22	556	458	463	458
Total cold run time: 65554 ms
Total hot run time: 59800 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178059 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false

query1	939	341	327	327
query2	6713	1896	1971	1896
query3	6715	205	204	204
query4	26132	22211	22234	22211
query5	8567	548	602	548
query6	274	186	193	186
query7	4617	277	264	264
query8	232	202	198	198
query9	9228	2742	2830	2742
query10	479	254	244	244
query11	16270	15356	15318	15318
query12	138	75	76	75
query13	1717	386	392	386
query14	11945	7257	7287	7257
query15	242	189	199	189
query16	5385	262	249	249
query17	1900	484	482	482
query18	1961	277	257	257
query19	291	151	147	147
query20	78	82	75	75
query21	191	126	126	126
query22	5074	4587	4756	4587
query23	31830	30783	30823	30783
query24	11007	2800	2800	2800
query25	549	329	329	329
query26	1434	161	150	150
query27	2782	273	281	273
query28	6883	1846	1855	1846
query29	1079	709	661	661
query30	273	136	139	136
query31	962	768	775	768
query32	97	65	58	58
query33	700	243	232	232
query34	1010	462	483	462
query35	861	741	717	717
query36	1352	1262	1230	1230
query37	103	71	71	71
query38	3379	3294	3231	3231
query39	1342	1286	1288	1286
query40	221	95	95	95
query41	38	35	35	35
query42	109	100	98	98
query43	546	486	524	486
query44	1097	680	697	680
query45	213	187	181	181
query46	1055	663	638	638
query47	1647	1516	1568	1516
query48	401	326	316	316
query49	1143	313	314	313
query50	707	323	325	323
query51	5347	5188	5182	5182
query52	110	94	97	94
query53	357	281	286	281
query54	929	452	466	452
query55	99	89	90	89
query56	207	189	203	189
query57	972	938	941	938
query58	218	202	201	201
query59	2789	2646	2652	2646
query60	248	228	214	214
query61	86	81	81	81
query62	651	405	372	372
query63	304	283	307	283
query64	4776	1684	1661	1661
query65	3366	3256	3248	3248
query66	1071	338	326	326
query67	15754	15383	15085	15085
query68	10992	514	499	499
query69	714	381	396	381
query70	1692	1571	1533	1533
query71	573	235	259	235
query72	4999	2812	2810	2810
query73	2091	315	304	304
query74	6956	6459	6351	6351
query75	4829	2350	2296	2296
query76	6345	1096	1054	1054
query77	693	292	275	275
query78	9675	8489	8526	8489
query79	1022	514	515	514
query80	553	347	342	342
query81	466	215	207	207
query82	195	86	92	86
query83	151	128	120	120
query84	282	76	77	76
query85	1032	344	334	334
query86	384	382	363	363
query87	3592	3427	3394	3394
query88	2828	2206	2190	2190
query89	487	392	397	392
query90	1896	206	199	199
query91	164	125	137	125
query92	67	56	56	56
query93	970	427	404	404
query94	1053	190	190	190
query95	531	489	475	475
query96	612	314	307	307
query97	4309	4150	4150	4150
query98	213	193	184	184
query99	1077	701	700	700
Total cold run time: 290635 ms
Total hot run time: 178059 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.7 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 21.8 seconds inserted 10000000 Rows, about 458K ops/s
storage size: 17183779072 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false

query1	0.06	0.05	0.05
query2	0.06	0.02	0.02
query3	0.25	0.11	0.11
query4	1.76	0.13	0.12
query5	0.52	0.50	0.52
query6	1.39	0.64	0.65
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.55	0.50	0.50
query10	0.55	0.56	0.55
query11	0.13	0.10	0.09
query12	0.12	0.10	0.09
query13	0.61	0.60	0.60
query14	0.79	0.78	0.79
query15	0.81	0.80	0.80
query16	0.34	0.36	0.36
query17	0.98	0.94	0.94
query18	0.25	0.25	0.25
query19	1.86	1.74	1.75
query20	0.01	0.01	0.01
query21	15.44	0.57	0.56
query22	2.32	2.65	2.42
query23	17.17	0.79	0.87
query24	16.15	0.56	0.59
query25	2.17	0.15	0.14
query26	0.14	0.14	0.13
query27	0.13	0.14	0.14
query28	6.99	0.86	0.81
query29	12.54	3.23	3.23
query30	0.53	0.48	0.47
query31	2.78	0.37	0.35
query32	3.36	0.49	0.49
query33	3.24	3.25	3.21
query34	16.03	4.18	4.15
query35	4.18	4.16	4.17
query36	1.09	1.03	1.03
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.13	0.14
query41	0.08	0.01	0.01
query42	0.03	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 115.8 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a0329a58b6853364746c92515260cac238f748fd with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       12.5 seconds inserted 10000000 Rows, about 800K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 12, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 67ef9e3 into apache:master Jan 14, 2024
18 of 20 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 15, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.0 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants