Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[scan](improvement) Adjust parallel scanners num #38430

Merged
merged 1 commit into from
Jul 29, 2024
Merged

[scan](improvement) Adjust parallel scanners num #38430

merged 1 commit into from
Jul 29, 2024

Conversation

HappenLee
Copy link
Contributor

Proposed changes

Before：

mysql [tpcds]>select  count(*) from (     select distinct c_last_name, c_first_name, d_date     from store_sales, date_dim, customer           where store_sales.ss_sold_date_sk = date_dim.d_date_sk       and store_sales.ss_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11   intersect     select distinct c_last_name, c_first_name, d_date     from catalog_sales, date_dim, customer           where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk       and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11   intersect     select distinct c_last_name, c_first_name, d_date     from web_sales, date_dim, customer           where web_sales.ws_sold_date_sk = date_dim.d_date_sk       and web_sales.ws_bill_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11 ) hot_cust limit 100;
+----------+
| count(*) |
+----------+
| 24874049 |
+----------+
1 row in set (53 sec)

After:

mysql [tpcds]>select  count(*) from (     select distinct c_last_name, c_first_name, d_date     from store_sales, date_dim, customer           where store_sales.ss_sold_date_sk = date_dim.d_date_sk       and store_sales.ss_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11   intersect     select distinct c_last_name, c_first_name, d_date     from catalog_sales, date_dim, customer           where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk       and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11   intersect     select distinct c_last_name, c_first_name, d_date     from web_sales, date_dim, customer           where web_sales.ws_sold_date_sk = date_dim.d_date_sk       and web_sales.ws_bill_customer_sk = customer.c_customer_sk       and d_month_seq between 1186 and 1186 + 11 ) hot_cust limit 100;
+----------+
| count(*) |
+----------+
| 24874049 |
+----------+
1 row in set (40.45 sec)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40224 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a082aff6e489a71d435a80f37064308a31f8fd7d, data reload: false

------ Round 1 ----------------------------------
q1	17623	4054	4004	4004
q2	2024	198	192	192
q3	10465	1236	1346	1236
q4	10168	857	904	857
q5	7682	2894	2954	2894
q6	222	141	140	140
q7	1018	611	610	610
q8	9416	1890	1973	1890
q9	8490	6606	6576	6576
q10	8776	3840	3836	3836
q11	432	251	243	243
q12	412	226	227	226
q13	17760	2960	2924	2924
q14	271	249	241	241
q15	518	487	479	479
q16	523	387	385	385
q17	960	938	906	906
q18	7935	7223	7160	7160
q19	1648	1210	1203	1203
q20	548	332	329	329
q21	4922	3631	3614	3614
q22	337	280	279	279
Total cold run time: 112150 ms
Total hot run time: 40224 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4057	4001	4011	4001
q2	319	215	217	215
q3	2980	2974	2958	2958
q4	1926	1964	1995	1964
q5	5489	5600	5426	5426
q6	219	132	138	132
q7	2203	1787	1789	1787
q8	3390	3354	3331	3331
q9	8634	8649	8628	8628
q10	4074	4065	4066	4065
q11	543	460	459	459
q12	739	578	549	549
q13	16424	3140	3116	3116
q14	297	279	279	279
q15	524	478	493	478
q16	461	434	432	432
q17	1737	1714	1697	1697
q18	8110	7566	7753	7566
q19	1759	1728	1712	1712
q20	2051	1833	1830	1830
q21	5187	4854	4914	4854
q22	533	498	457	457
Total cold run time: 71656 ms
Total hot run time: 55936 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174829 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a082aff6e489a71d435a80f37064308a31f8fd7d, data reload: false

query1	909	363	372	363
query2	6479	1708	1714	1708
query3	6647	211	227	211
query4	20389	17564	17432	17432
query5	3652	506	513	506
query6	275	179	173	173
query7	4595	305	301	301
query8	278	194	192	192
query9	8504	2396	2405	2396
query10	451	296	303	296
query11	10619	9986	9988	9986
query12	122	88	87	87
query13	1631	376	372	372
query14	9443	7565	7671	7565
query15	215	164	163	163
query16	6750	445	456	445
query17	972	579	606	579
query18	1970	284	278	278
query19	192	139	144	139
query20	94	87	91	87
query21	205	97	103	97
query22	4273	4070	4057	4057
query23	34087	33535	33784	33535
query24	9342	3039	3079	3039
query25	681	427	399	399
query26	1468	150	154	150
query27	2985	292	296	292
query28	7578	1985	1988	1985
query29	1117	467	439	439
query30	241	160	160	160
query31	923	775	784	775
query32	103	56	56	56
query33	728	345	347	345
query34	913	500	514	500
query35	910	782	775	775
query36	1028	885	897	885
query37	170	78	78	78
query38	2909	2761	2802	2761
query39	879	836	807	807
query40	261	113	114	113
query41	44	43	45	43
query42	122	101	100	100
query43	504	434	422	422
query44	1177	718	731	718
query45	212	177	182	177
query46	1100	802	797	797
query47	1801	1703	1729	1703
query48	375	292	292	292
query49	892	436	422	422
query50	910	436	448	436
query51	6771	6736	6686	6686
query52	96	88	88	88
query53	252	185	182	182
query54	631	456	459	456
query55	78	74	78	74
query56	298	281	304	281
query57	1142	1028	1046	1028
query58	271	269	274	269
query59	2552	2405	2380	2380
query60	326	286	294	286
query61	100	99	95	95
query62	896	667	663	663
query63	210	192	183	183
query64	5672	1913	1925	1913
query65	3182	3110	3092	3092
query66	1202	335	342	335
query67	15297	14686	14755	14686
query68	4443	573	568	568
query69	658	379	312	312
query70	1127	1088	1098	1088
query71	445	298	283	283
query72	7805	6692	6703	6692
query73	813	327	333	327
query74	6058	5660	5627	5627
query75	3396	2738	2722	2722
query76	2945	1232	1253	1232
query77	555	318	307	307
query78	9363	8958	8890	8890
query79	2483	530	531	530
query80	1333	515	503	503
query81	573	228	231	228
query82	764	132	134	132
query83	249	179	180	179
query84	258	79	81	79
query85	1171	322	332	322
query86	384	304	296	296
query87	3399	3096	3097	3096
query88	3010	2482	2485	2482
query89	382	300	295	295
query90	1680	200	197	197
query91	127	102	106	102
query92	58	49	53	49
query93	1655	606	612	606
query94	802	296	302	296
query95	375	274	274	274
query96	607	286	286	286
query97	3219	3024	2985	2985
query98	228	206	200	200
query99	1628	1310	1337	1310
Total cold run time: 263094 ms
Total hot run time: 174829 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a082aff6e489a71d435a80f37064308a31f8fd7d, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.03
query3	0.22	0.04	0.04
query4	1.69	0.07	0.07
query5	0.50	0.48	0.47
query6	1.15	0.72	0.71
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.51
query10	0.56	0.56	0.59
query11	0.14	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.60	0.59
query14	0.76	0.79	0.84
query15	0.89	0.85	0.86
query16	0.34	0.35	0.36
query17	0.99	0.99	0.99
query18	0.21	0.21	0.22
query19	1.86	1.70	1.71
query20	0.01	0.01	0.01
query21	15.42	0.77	0.65
query22	4.00	7.13	1.43
query23	18.01	1.28	1.30
query24	2.27	0.22	0.22
query25	0.19	0.09	0.09
query26	0.32	0.21	0.22
query27	0.46	0.24	0.24
query28	13.16	1.01	0.97
query29	12.55	3.33	3.32
query30	0.25	0.07	0.06
query31	2.87	0.40	0.40
query32	3.23	0.51	0.48
query33	2.93	3.00	2.97
query34	15.43	4.31	4.25
query35	4.31	4.29	4.32
query36	0.68	0.48	0.48
query37	0.20	0.17	0.17
query38	0.17	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.72 s
Total hot run time: 30.15 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 11b332f into apache:master Jul 29, 2024
29 of 31 checks passed
@HappenLee HappenLee mentioned this pull request Jul 29, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 31, 2024
@HappenLee
[performance](exec) Performance problem create too many scanner task (#…
3b234cf 
…38460)

## Proposed changes

cherry pick the pr: #38430

<!--Describe your changes.-->
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Aug 7, 2024
@HappenLee
Revert "[scan](improvement) Adjust parallel scanners num (apache#38430)"
4366f74 
This reverts commit 11b332f.
gavinchou pushed a commit that referenced this pull request Aug 7, 2024
@HappenLee
Revert "[scan](improvement) Adjust parallel scanners num (#38430)" (#…
99dce14 
…39046)

This reverts commit 11b332f.
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Aug 7, 2024
@HappenLee
Revert "[scan](improvement) Adjust parallel scanners num (apache#38430)"
c09831b 
Fix performance problem

This reverts commit 11b332f.
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Aug 8, 2024
@HappenLee
Revert "[scan](improvement) Adjust parallel scanners num (apache#38430)"
c08a262 
Fix ssb flat performance problem

This reverts commit 11b332f.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@HappenLee @wyxxxcat
Revert "[scan](improvement) Adjust parallel scanners num (apache#38430)…
5ef39ee 
…" (apache#39046)

This reverts commit 11b332f.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@HappenLee @doris-robot @yiguolei @Gabriel39