Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Performance](opt) opt the scan performance and remove unless code #35604

Merged
merged 1 commit into from
May 31, 2024
Merged

[Performance](opt) opt the scan performance and remove unless code #35604

merged 1 commit into from
May 31, 2024

Conversation

HappenLee
Copy link
Contributor

Proposed changes

Before

select sum(l_linenumber) from lineitem_bak;
+-------------------+
| sum(l_linenumber) |
+-------------------+
|        1800152937 |
+-------------------+
1 row in set (0.17 sec)

after:

select sum(l_linenumber) from lineitem_bak;
+-------------------+
| sum(l_linenumber) |
+-------------------+
|        1800152937 |
+-------------------+
1 row in set (0.14 sec)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee533245ef07a2a4520768e0f6c912cd7818772e, data reload: false

------ Round 1 ----------------------------------
q1	17918	4371	4234	4234
q2	2032	194	194	194
q3	10757	1243	1151	1151
q4	10247	804	761	761
q5	7477	2701	2649	2649
q6	224	134	135	134
q7	977	600	587	587
q8	9677	2117	2120	2117
q9	9208	6632	6619	6619
q10	9755	3812	3757	3757
q11	465	241	225	225
q12	457	209	210	209
q13	17765	2950	2950	2950
q14	252	213	215	213
q15	514	460	475	460
q16	519	371	375	371
q17	974	664	680	664
q18	8032	7474	7430	7430
q19	4075	1498	1438	1438
q20	660	319	312	312
q21	4945	3871	3256	3256
q22	348	279	277	277
Total cold run time: 117278 ms
Total hot run time: 40008 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4202	4185	4185
q2	357	280	256	256
q3	2939	2691	2706	2691
q4	1900	1583	1608	1583
q5	5235	5252	5237	5237
q6	222	126	124	124
q7	2090	1754	1705	1705
q8	3174	3341	3288	3288
q9	8360	8310	8304	8304
q10	3892	3652	3646	3646
q11	586	481	479	479
q12	744	575	572	572
q13	17376	2942	2905	2905
q14	291	283	273	273
q15	512	479	474	474
q16	475	402	418	402
q17	1777	1487	1453	1453
q18	7610	7648	7415	7415
q19	1643	1477	1511	1477
q20	1976	1792	1766	1766
q21	4796	4593	4645	4593
q22	576	509	478	478
Total cold run time: 70858 ms
Total hot run time: 53306 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.89% (9123/25416)
Line Coverage: 27.36% (74945/273896)
Region Coverage: 26.61% (38813/145865)
Branch Coverage: 23.45% (19749/84234)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ee533245ef07a2a4520768e0f6c912cd7818772e_ee533245ef07a2a4520768e0f6c912cd7818772e/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41101 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee533245ef07a2a4520768e0f6c912cd7818772e, data reload: false

------ Round 1 ----------------------------------
q1	17741	4439	4271	4271
q2	2023	181	185	181
q3	10550	1242	1066	1066
q4	10202	848	759	759
q5	7472	2676	2737	2676
q6	216	133	133	133
q7	951	623	618	618
q8	9221	2155	2111	2111
q9	9309	6715	6712	6712
q10	9860	3878	3942	3878
q11	432	238	252	238
q12	455	237	237	237
q13	17209	3164	3177	3164
q14	270	230	229	229
q15	525	470	480	470
q16	476	372	382	372
q17	956	728	796	728
q18	8648	7895	7821	7821
q19	5986	1594	1513	1513
q20	634	311	324	311
q21	5073	3289	4063	3289
q22	403	324	332	324
Total cold run time: 118612 ms
Total hot run time: 41101 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4525	4485	4500	4485
q2	370	262	270	262
q3	3147	2958	2874	2874
q4	1982	1605	1619	1605
q5	5359	5529	5487	5487
q6	219	123	124	123
q7	2200	1791	1794	1791
q8	3285	3433	3417	3417
q9	8587	8625	8738	8625
q10	4078	3666	3723	3666
q11	623	504	495	495
q12	823	610	666	610
q13	16533	3129	3145	3129
q14	325	264	257	257
q15	528	496	473	473
q16	497	439	445	439
q17	1820	1496	1469	1469
q18	7724	7556	7538	7538
q19	1675	1474	1609	1474
q20	1981	1775	1798	1775
q21	9367	4619	4755	4619
q22	577	561	521	521
Total cold run time: 76225 ms
Total hot run time: 55134 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170020 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee533245ef07a2a4520768e0f6c912cd7818772e, data reload: false

query1	915	385	379	379
query2	6441	2344	2385	2344
query3	6645	212	217	212
query4	19557	17288	17235	17235
query5	4105	434	427	427
query6	253	154	144	144
query7	4596	296	313	296
query8	316	287	297	287
query9	8451	2416	2385	2385
query10	452	280	269	269
query11	10608	10197	10043	10043
query12	134	89	83	83
query13	1626	366	373	366
query14	9759	7906	7526	7526
query15	235	191	195	191
query16	7413	260	263	260
query17	1309	517	539	517
query18	1943	274	270	270
query19	198	154	151	151
query20	97	83	84	83
query21	193	136	129	129
query22	4209	4052	3942	3942
query23	33550	33059	33043	33043
query24	5035	2838	2854	2838
query25	457	352	367	352
query26	691	156	165	156
query27	1848	319	326	319
query28	3771	2088	2095	2088
query29	832	614	602	602
query30	224	156	151	151
query31	917	759	733	733
query32	63	53	56	53
query33	423	289	268	268
query34	848	484	473	473
query35	707	636	604	604
query36	1061	930	895	895
query37	101	69	67	67
query38	2885	2805	2746	2746
query39	842	797	783	783
query40	198	126	124	124
query41	53	51	53	51
query42	104	96	97	96
query43	576	535	563	535
query44	1137	745	758	745
query45	182	176	171	171
query46	1049	700	711	700
query47	1843	1749	1761	1749
query48	362	298	302	298
query49	763	378	381	378
query50	772	381	395	381
query51	6699	6546	6618	6546
query52	110	92	88	88
query53	359	295	299	295
query54	573	439	435	435
query55	73	74	72	72
query56	257	234	243	234
query57	1137	1050	1059	1050
query58	220	220	203	203
query59	3519	3386	3225	3225
query60	268	254	263	254
query61	90	90	86	86
query62	558	461	442	442
query63	309	288	295	288
query64	2604	1770	1699	1699
query65	3179	3096	3122	3096
query66	719	325	328	325
query67	15220	14577	14714	14577
query68	4634	560	547	547
query69	444	270	272	270
query70	1150	1139	1092	1092
query71	392	286	264	264
query72	7660	5061	2771	2771
query73	723	330	325	325
query74	6054	5608	5653	5608
query75	3327	2637	2608	2608
query76	2263	1042	994	994
query77	388	271	272	271
query78	10200	9810	9752	9752
query79	2586	532	520	520
query80	1083	425	429	425
query81	517	224	219	219
query82	889	92	92	92
query83	242	177	173	173
query84	249	86	88	86
query85	979	267	265	265
query86	497	297	334	297
query87	3302	3116	3121	3116
query88	4440	2379	2358	2358
query89	487	407	399	399
query90	2002	192	195	192
query91	136	113	111	111
query92	65	51	52	51
query93	1950	515	506	506
query94	1153	198	191	191
query95	417	328	322	322
query96	604	272	268	268
query97	3195	3001	3005	3001
query98	241	232	219	219
query99	1213	854	873	854
Total cold run time: 250125 ms
Total hot run time: 170020 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee533245ef07a2a4520768e0f6c912cd7818772e, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.51	0.50	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.50	0.50
query10	0.54	0.56	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.79	0.78
query15	0.82	0.83	0.81
query16	0.37	0.36	0.37
query17	0.96	1.02	0.96
query18	0.21	0.22	0.24
query19	1.77	1.68	1.66
query20	0.01	0.01	0.01
query21	15.66	0.66	0.66
query22	4.94	7.03	1.46
query23	18.26	1.38	1.27
query24	1.96	0.23	0.22
query25	0.15	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.31	1.01	0.99
query29	13.53	3.30	3.27
query30	0.25	0.05	0.06
query31	2.90	0.38	0.38
query32	3.29	0.48	0.47
query33	2.87	2.83	2.96
query34	16.99	4.40	4.38
query35	4.49	4.53	4.58
query36	0.69	0.46	0.46
query37	0.17	0.17	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 110.92 s
Total hot run time: 30 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit b5034c2 into apache:master May 31, 2024
27 of 29 checks passed
dataroaring pushed a commit that referenced this pull request May 31, 2024
@HappenLee @dataroaring
[Performance](opt) opt the scan performance and remove unless code (#…
5a054f1 
…35604)

## Proposed changes

Before
```
select sum(l_linenumber) from lineitem_bak;
+-------------------+
| sum(l_linenumber) |
+-------------------+
|        1800152937 |
+-------------------+
1 row in set (0.17 sec)
```

after:
```
select sum(l_linenumber) from lineitem_bak;
+-------------------+
| sum(l_linenumber) |
+-------------------+
|        1800152937 |
+-------------------+
1 row in set (0.14 sec)
```
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Jun 1, 2024
@HappenLee
[Scanner](revert) revert the scanner change by apache#35604
6568196
HappenLee added a commit that referenced this pull request Jun 2, 2024
@HappenLee
[Scanner](revert) revert the scanner change by #35604 (#31508)
ae50d7a
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@HappenLee @dataroaring
[Scanner](revert) revert the scanner change by #35604 (#31508)
be733da
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@HappenLee @seawinde
[Scanner](revert) revert the scanner change by apache#35604 (apache#3…
ac38582 
…1508)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged not-merge/2.1 reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@HappenLee @doris-robot @Gabriel39 @zhangstar333 @yiguolei @dataroaring