Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout #30515

Merged
merged 1 commit into from
Jan 29, 2024
Merged

[Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout #30515

merged 1 commit into from
Jan 29, 2024

Conversation

HappenLee
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 36974 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 606479db7504dd1c1ebd79f210db79a94d3d1d88, data reload: false

------ Round 1 ----------------------------------
q1	17653	4476	4411	4411
q2	2045	143	132	132
q3	10683	934	928	928
q4	4673	780	823	780
q5	7683	2768	2840	2768
q6	186	119	118	118
q7	1122	729	699	699
q8	9303	2008	2014	2008
q9	7209	6342	6317	6317
q10	8088	2445	2421	2421
q11	420	214	209	209
q12	784	277	268	268
q13	18004	3345	3302	3302
q14	281	257	240	240
q15	536	518	489	489
q16	464	411	404	404
q17	948	488	529	488
q18	6745	6029	5966	5966
q19	1547	1415	1352	1352
q20	572	336	326	326
q21	6950	3046	3170	3046
q22	793	319	302	302
Total cold run time: 106689 ms
Total hot run time: 36974 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4363	4449	4363
q2	337	230	244	230
q3	3062	2853	2819	2819
q4	1844	1680	1702	1680
q5	5256	5325	5299	5299
q6	191	115	115	115
q7	2158	1811	1836	1811
q8	3113	3278	3251	3251
q9	8361	8295	8323	8295
q10	5876	3564	3613	3564
q11	554	470	454	454
q12	766	586	608	586
q13	13544	3150	3102	3102
q14	278	254	255	254
q15	536	485	485	485
q16	519	448	488	448
q17	1864	1701	1687	1687
q18	8012	7590	7612	7590
q19	6714	1483	1534	1483
q20	2169	1900	1899	1899
q21	4937	4662	4588	4588
q22	551	500	500	500
Total cold run time: 74996 ms
Total hot run time: 54503 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.26% (70523/249587)
Region Coverage: 27.25% (36385/133525)
Branch Coverage: 24.05% (18645/77542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/606479db7504dd1c1ebd79f210db79a94d3d1d88_606479db7504dd1c1ebd79f210db79a94d3d1d88/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172333 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 606479db7504dd1c1ebd79f210db79a94d3d1d88, data reload: false

query1	935	338	327	327
query2	6563	2039	2024	2024
query3	6697	202	196	196
query4	26834	22215	22017	22017
query5	4450	348	360	348
query6	245	160	156	156
query7	4606	265	263	263
query8	243	197	174	174
query9	9262	2286	2248	2248
query10	415	203	217	203
query11	16724	15338	15263	15263
query12	126	73	66	66
query13	1690	376	375	375
query14	9222	6543	6544	6543
query15	235	179	174	174
query16	6567	268	245	245
query17	1896	464	479	464
query18	1933	257	244	244
query19	169	129	138	129
query20	73	66	69	66
query21	198	138	135	135
query22	4831	4729	4833	4729
query23	31407	30355	30194	30194
query24	9501	2744	2763	2744
query25	546	306	298	298
query26	822	140	138	138
query27	2847	285	287	285
query28	6308	1843	1827	1827
query29	947	628	606	606
query30	297	135	136	135
query31	890	717	733	717
query32	93	47	47	47
query33	538	209	206	206
query34	818	456	481	456
query35	831	772	745	745
query36	1250	1129	1216	1129
query37	91	58	57	57
query38	3263	3228	3190	3190
query39	1318	1244	1243	1243
query40	198	99	84	84
query41	38	38	34	34
query42	91	84	86	84
query43	538	459	495	459
query44	1124	686	686	686
query45	190	182	180	180
query46	1030	642	636	636
query47	1527	1457	1455	1455
query48	379	311	298	298
query49	1107	287	280	280
query50	675	303	305	303
query51	5249	5222	5063	5063
query52	93	83	80	80
query53	335	254	267	254
query54	221	190	194	190
query55	79	75	72	72
query56	179	158	157	157
query57	964	841	911	841
query58	176	149	152	149
query59	2461	2391	2246	2246
query60	196	180	176	176
query61	83	85	84	84
query62	589	348	363	348
query63	277	263	266	263
query64	4028	1796	1724	1724
query65	3290	3218	3202	3202
query66	1132	317	326	317
query67	14467	14073	14235	14073
query68	5439	491	483	483
query69	476	311	311	311
query70	1552	1529	1553	1529
query71	282	214	206	206
query72	4065	2820	2842	2820
query73	691	314	311	311
query74	6748	6280	6241	6241
query75	2861	2358	2265	2265
query76	3444	938	948	938
query77	368	238	229	229
query78	9103	8696	8513	8513
query79	2687	491	496	491
query80	1983	333	315	315
query81	523	198	196	196
query82	771	84	84	84
query83	239	116	119	116
query84	303	69	74	69
query85	1958	356	334	334
query86	511	386	404	386
query87	3504	3243	3254	3243
query88	4113	2185	2153	2153
query89	427	358	348	348
query90	1921	188	186	186
query91	146	117	137	117
query92	52	42	45	42
query93	4209	423	435	423
query94	1256	158	156	156
query95	497	467	449	449
query96	623	307	303	303
query97	4225	4095	4123	4095
query98	199	190	183	183
query99	1026	654	654	654
Total cold run time: 272599 ms
Total hot run time: 172333 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 606479db7504dd1c1ebd79f210db79a94d3d1d88, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.52	0.52
query6	1.21	0.64	0.65
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.56	0.48	0.49
query10	0.55	0.56	0.55
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.60	0.62	0.61
query14	0.77	0.82	0.79
query15	0.81	0.78	0.77
query16	0.39	0.40	0.39
query17	1.02	1.04	1.02
query18	0.21	0.25	0.24
query19	1.91	1.80	1.72
query20	0.01	0.01	0.02
query21	15.40	0.58	0.57
query22	2.58	2.41	1.62
query23	17.56	0.87	0.80
query24	2.32	1.36	0.65
query25	0.33	0.22	0.17
query26	0.49	0.15	0.14
query27	0.05	0.04	0.05
query28	11.92	0.85	0.86
query29	12.58	3.12	3.22
query30	0.67	0.54	0.53
query31	2.79	0.36	0.35
query32	3.35	0.47	0.48
query33	3.20	3.22	3.22
query34	16.01	4.33	4.27
query35	4.28	4.24	4.29
query36	1.10	1.05	1.05
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.87 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 606479db7504dd1c1ebd79f210db79a94d3d1d88 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

morningman
morningman approved these changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 6e3458b into apache:master Jan 29, 2024
28 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Jan 30, 2024
@HappenLee @Doris-Extras
[Try_Fix](scan) try fix the scanner schedule logic to prevent excessi…
129463f 
…ve memory usage and timeout (#30515)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@HappenLee @doris-robot @morningman @zclllyybb