Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement]Support cgroup v2 for workload group #39374

Merged
merged 1 commit into from
Aug 16, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Aug 14, 2024

Proposed changes

Support cgroup v2 for wokrload group.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wangbo
Copy link
Contributor Author

wangbo commented Aug 14, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 14, 2024
yiguolei
yiguolei previously approved these changes Aug 14, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37978 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c35e4c21e95102e9251f3bba813cb441e04e027, data reload: false

------ Round 1 ----------------------------------
q1	17617	4422	4314	4314
q2	2026	180	177	177
q3	10520	1195	1064	1064
q4	10147	822	761	761
q5	7772	2842	2820	2820
q6	226	142	141	141
q7	965	601	602	601
q8	9329	2060	2092	2060
q9	7347	6570	6571	6570
q10	7072	2248	2134	2134
q11	453	244	243	243
q12	413	228	222	222
q13	18830	2999	2994	2994
q14	277	236	235	235
q15	509	480	497	480
q16	537	382	384	382
q17	998	709	784	709
q18	7360	6819	6781	6781
q19	4782	1060	960	960
q20	681	326	340	326
q21	3851	2983	3021	2983
q22	1088	1031	1021	1021
Total cold run time: 112800 ms
Total hot run time: 37978 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4448	4211	4239	4211
q2	397	277	271	271
q3	2893	2626	2627	2626
q4	1878	1636	1602	1602
q5	5412	5592	5620	5592
q6	238	140	136	136
q7	2198	1800	1738	1738
q8	3292	3428	3429	3428
q9	8712	8684	8897	8684
q10	3545	3222	3258	3222
q11	641	512	514	512
q12	827	610	618	610
q13	17472	3091	3141	3091
q14	320	284	302	284
q15	527	483	485	483
q16	498	437	446	437
q17	1833	1547	1525	1525
q18	8184	7832	7425	7425
q19	1743	1621	1485	1485
q20	2093	1843	1812	1812
q21	8219	5027	5128	5027
q22	1079	1058	1029	1029
Total cold run time: 76449 ms
Total hot run time: 55230 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184693 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c35e4c21e95102e9251f3bba813cb441e04e027, data reload: false

query1	904	387	363	363
query2	6461	1939	1791	1791
query3	6638	210	224	210
query4	33217	23248	23232	23232
query5	4184	476	485	476
query6	273	168	158	158
query7	4587	289	277	277
query8	245	202	199	199
query9	8676	2475	2418	2418
query10	426	264	259	259
query11	15522	14873	15005	14873
query12	149	103	99	99
query13	1638	369	356	356
query14	9572	6644	6872	6644
query15	222	168	173	168
query16	7576	496	473	473
query17	1083	564	555	555
query18	1947	286	289	286
query19	184	154	162	154
query20	112	109	108	108
query21	204	107	98	98
query22	4248	4039	3969	3969
query23	33937	33472	33082	33082
query24	11669	2815	2857	2815
query25	641	397	399	397
query26	1179	158	157	157
query27	2675	281	280	280
query28	6979	2054	2058	2054
query29	876	427	426	426
query30	303	148	149	148
query31	990	722	724	722
query32	96	56	57	56
query33	752	293	283	283
query34	925	458	455	455
query35	820	732	726	726
query36	1089	937	918	918
query37	157	84	81	81
query38	4016	3785	3771	3771
query39	1427	1389	1379	1379
query40	204	121	120	120
query41	52	45	47	45
query42	129	102	97	97
query43	493	457	476	457
query44	1202	727	724	724
query45	198	163	163	163
query46	1118	733	719	719
query47	1884	1788	1762	1762
query48	360	296	285	285
query49	1033	429	422	422
query50	812	408	409	408
query51	6778	6731	6758	6731
query52	105	89	96	89
query53	252	187	182	182
query54	880	456	453	453
query55	75	78	75	75
query56	267	348	244	244
query57	1125	1033	1031	1031
query58	235	230	234	230
query59	3012	2664	2673	2664
query60	290	262	258	258
query61	97	95	110	95
query62	833	654	654	654
query63	214	181	184	181
query64	5228	2323	1717	1717
query65	3160	3173	3148	3148
query66	768	364	338	338
query67	15615	14864	14829	14829
query68	6645	550	556	550
query69	650	352	278	278
query70	1137	1087	1131	1087
query71	567	272	266	266
query72	7612	2203	2008	2008
query73	810	319	314	314
query74	9200	8775	8648	8648
query75	4987	2682	2707	2682
query76	4829	957	1021	957
query77	837	308	297	297
query78	9824	10994	9674	9674
query79	9438	537	526	526
query80	1378	489	488	488
query81	579	224	224	224
query82	287	137	133	133
query83	331	143	143	143
query84	266	74	73	73
query85	710	321	264	264
query86	389	294	296	294
query87	4383	4182	4210	4182
query88	4543	2257	2248	2248
query89	390	285	281	281
query90	2334	191	184	184
query91	118	93	93	93
query92	64	47	49	47
query93	1594	536	528	528
query94	988	298	264	264
query95	354	256	250	250
query96	582	263	265	263
query97	3210	3045	2995	2995
query98	218	197	203	197
query99	1515	1218	1276	1218
Total cold run time: 303787 ms
Total hot run time: 184693 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c35e4c21e95102e9251f3bba813cb441e04e027, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.05
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.48
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.61	0.59
query14	0.75	0.77	0.77
query15	0.86	0.82	0.82
query16	0.34	0.37	0.37
query17	1.04	1.06	1.07
query18	0.24	0.23	0.22
query19	1.98	1.81	1.76
query20	0.01	0.02	0.00
query21	15.39	0.76	0.65
query22	3.95	6.74	2.80
query23	18.34	1.38	1.31
query24	2.14	0.22	0.22
query25	0.14	0.08	0.08
query26	0.30	0.21	0.22
query27	0.46	0.22	0.23
query28	13.31	1.03	1.01
query29	12.66	3.26	3.26
query30	0.24	0.05	0.05
query31	2.89	0.39	0.40
query32	3.26	0.49	0.48
query33	2.97	2.96	2.99
query34	17.11	4.38	4.33
query35	4.44	4.49	4.41
query36	0.66	0.49	0.49
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.05	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.92 s
Total hot run time: 31.73 s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 15, 2024
@wangbo wangbo force-pushed the 0813_cgroup_v2 branch 8 times, most recently from 1d27c91 to ea75104 Compare August 15, 2024 06:30
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@wangbo
Copy link
Contributor Author

wangbo commented Aug 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38417 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02ed46cfaf0c26fe20ee9dbade675f768c4637a1, data reload: false

------ Round 1 ----------------------------------
q1	17838	4497	4318	4318
q2	2042	207	236	207
q3	11774	1158	1190	1158
q4	10542	789	696	696
q5	7841	2880	2841	2841
q6	264	155	153	153
q7	995	663	646	646
q8	9580	2115	2124	2115
q9	8817	6609	6601	6601
q10	7104	2251	2289	2251
q11	482	271	271	271
q12	428	250	257	250
q13	18892	2973	3071	2973
q14	306	260	259	259
q15	558	532	546	532
q16	534	422	409	409
q17	1005	667	724	667
q18	7563	6841	6724	6724
q19	6183	1101	1010	1010
q20	865	355	363	355
q21	3968	2984	2942	2942
q22	1140	1052	1039	1039
Total cold run time: 118721 ms
Total hot run time: 38417 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4808	4320	4335	4320
q2	408	304	313	304
q3	2885	2723	2787	2723
q4	1996	1690	1742	1690
q5	5651	5705	5613	5613
q6	253	144	143	143
q7	2198	1734	1748	1734
q8	3390	3485	3473	3473
q9	8781	8676	8773	8676
q10	3640	3339	3298	3298
q11	624	539	540	539
q12	821	653	660	653
q13	15838	3063	3243	3063
q14	337	304	304	304
q15	556	539	521	521
q16	528	470	466	466
q17	1863	1596	1565	1565
q18	8055	8045	7634	7634
q19	6929	1638	1698	1638
q20	2191	1904	1901	1901
q21	11916	5162	5293	5162
q22	1173	1108	1095	1095
Total cold run time: 84841 ms
Total hot run time: 56515 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195844 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02ed46cfaf0c26fe20ee9dbade675f768c4637a1, data reload: false

query1	1374	923	878	878
query2	6721	2081	2010	2010
query3	10622	3907	3863	3863
query4	58748	26174	23348	23348
query5	6034	694	693	693
query6	547	177	181	177
query7	6493	310	317	310
query8	458	353	353	353
query9	9600	2494	2473	2473
query10	608	304	311	304
query11	18355	15022	15577	15022
query12	192	126	125	125
query13	1696	453	431	431
query14	12923	7501	7634	7501
query15	296	191	202	191
query16	7683	517	503	503
query17	1212	638	615	615
query18	2065	348	340	340
query19	328	166	160	160
query20	137	132	138	132
query21	257	151	143	143
query22	4446	4374	4349	4349
query23	34218	33725	33975	33725
query24	5691	2973	3044	2973
query25	588	432	468	432
query26	722	192	184	184
query27	1754	315	302	302
query28	3632	2148	2134	2134
query29	752	440	446	440
query30	218	179	181	179
query31	1023	813	780	780
query32	126	75	79	75
query33	578	340	330	330
query34	873	488	496	488
query35	869	752	761	752
query36	1069	948	915	915
query37	153	98	102	98
query38	3872	3858	3860	3858
query39	1518	1485	1490	1485
query40	240	153	152	152
query41	164	133	132	132
query42	140	116	110	110
query43	532	506	501	501
query44	1099	797	793	793
query45	219	196	190	190
query46	1108	773	762	762
query47	1932	1817	1868	1817
query48	420	349	344	344
query49	912	573	576	573
query50	863	460	461	460
query51	6833	6747	6650	6650
query52	124	112	107	107
query53	295	228	223	223
query54	616	515	495	495
query55	88	86	92	86
query56	324	299	307	299
query57	1183	1104	1123	1104
query58	298	304	332	304
query59	3049	2824	2850	2824
query60	345	330	322	322
query61	146	147	139	139
query62	816	702	688	688
query63	253	224	225	224
query64	4421	2390	1788	1788
query65	3229	3198	3189	3189
query66	1043	658	706	658
query67	15501	15119	15054	15054
query68	7923	600	602	600
query69	690	417	346	346
query70	1597	1200	1140	1140
query71	545	312	309	309
query72	6987	2285	2076	2076
query73	2646	364	367	364
query74	9150	8833	8692	8692
query75	4219	2718	2773	2718
query76	5068	1019	1048	1019
query77	848	443	494	443
query78	10939	9495	9030	9030
query79	9555	569	566	566
query80	941	600	602	600
query81	608	260	259	259
query82	497	161	153	153
query83	252	215	211	211
query84	302	98	95	95
query85	762	345	346	345
query86	371	331	319	319
query87	4432	4176	4265	4176
query88	4191	2559	2570	2559
query89	517	332	327	327
query90	2317	234	234	234
query91	156	125	122	122
query92	125	74	75	74
query93	3868	548	550	548
query94	1107	316	293	293
query95	391	297	296	296
query96	625	288	286	286
query97	3263	3046	3065	3046
query98	241	232	215	215
query99	1530	1324	1331	1324
Total cold run time: 341307 ms
Total hot run time: 195844 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02ed46cfaf0c26fe20ee9dbade675f768c4637a1, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.53	0.54	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.12
query13	0.60	0.61	0.58
query14	0.78	0.80	0.80
query15	0.86	0.81	0.81
query16	0.36	0.38	0.38
query17	1.05	1.02	0.97
query18	0.23	0.22	0.22
query19	1.81	1.68	1.71
query20	0.01	0.02	0.01
query21	15.41	0.86	0.66
query22	4.44	6.78	2.07
query23	18.29	1.48	1.30
query24	2.07	0.23	0.23
query25	0.16	0.09	0.09
query26	0.29	0.22	0.22
query27	0.46	0.24	0.22
query28	13.31	1.02	1.01
query29	12.63	3.31	3.28
query30	0.37	0.18	0.18
query31	2.79	0.40	0.40
query32	3.28	0.49	0.48
query33	2.92	2.97	2.97
query34	17.03	4.34	4.34
query35	4.41	4.40	4.45
query36	0.66	0.49	0.49
query37	0.20	0.17	0.17
query38	0.17	0.16	0.16
query39	0.05	0.05	0.05
query40	0.17	0.14	0.15
query41	0.11	0.06	0.07
query42	0.08	0.06	0.06
query43	0.06	0.06	0.05
Total cold run time: 110.17 s
Total hot run time: 31.23 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 16, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@wangbo wangbo changed the title [Improment]Support cgroup v2 for wokrload group [Improvement]Support cgroup v2 for workload group Aug 16, 2024
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo wangbo merged commit 0c0ef27 into apache:master Aug 16, 2024
27 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
## Proposed changes
Support cgroup v2 for wokrload group.
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Oct 25, 2024
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Oct 25, 2024
@wangbo wangbo mentioned this pull request Oct 25, 2024
yiguolei pushed a commit that referenced this pull request Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants