Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](auth)Fix concurrency issue during role manager upgrade #42419

Merged
merged 1 commit into from
Nov 4, 2024
Merged

[fix](auth)Fix concurrency issue during role manager upgrade #42419

merged 1 commit into from
Nov 4, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Oct 24, 2024 •
edited
Loading

Change the map to a concurrency safe map，because there is a possibility of concurrent read and write operations

For example, when a node performs the rectifyPrivs operation, it needs to traverse 'roles' and may be adding or deleting roles at the same time

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Oct 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41413 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f67f94431d19751d1d7fbf52dd5619f178c11fc1, data reload: false

------ Round 1 ----------------------------------
q1	17598	7548	7357	7357
q2	2062	176	149	149
q3	10557	1154	1142	1142
q4	10221	869	858	858
q5	7780	3128	3075	3075
q6	240	155	159	155
q7	1016	625	610	610
q8	9425	1945	1951	1945
q9	6750	6457	6387	6387
q10	7172	2445	2455	2445
q11	466	257	258	257
q12	523	220	229	220
q13	17796	3025	3024	3024
q14	245	212	213	212
q15	571	522	517	517
q16	659	597	589	589
q17	1023	494	546	494
q18	7490	6726	6764	6726
q19	1350	912	1001	912
q20	488	181	180	180
q21	4090	3176	3256	3176
q22	1091	983	997	983
Total cold run time: 108613 ms
Total hot run time: 41413 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7326	7258	7304	7258
q2	325	237	222	222
q3	2973	2808	2854	2808
q4	1977	1757	1727	1727
q5	5460	5520	5571	5520
q6	234	146	145	145
q7	2160	1717	1726	1717
q8	3264	3429	3445	3429
q9	8639	8671	8595	8595
q10	3546	3482	3444	3444
q11	587	480	507	480
q12	803	568	607	568
q13	14497	3089	3054	3054
q14	317	264	268	264
q15	559	511	507	507
q16	694	644	639	639
q17	1842	1575	1564	1564
q18	7929	7556	7319	7319
q19	1693	1512	1551	1512
q20	2052	1828	1860	1828
q21	5426	5461	5110	5110
q22	1122	1038	1004	1004
Total cold run time: 73425 ms
Total hot run time: 58714 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f67f94431d19751d1d7fbf52dd5619f178c11fc1, data reload: false

query1	960	375	374	374
query2	3794	2122	2050	2050
query3	2334	213	234	213
query4	32847	23517	23571	23517
query5	3117	494	474	474
query6	217	174	170	170
query7	3445	300	292	292
query8	268	238	270	238
query9	7263	2733	2724	2724
query10	410	288	289	288
query11	17572	15229	15398	15229
query12	157	106	104	104
query13	1220	457	410	410
query14	9432	7791	7476	7476
query15	250	173	180	173
query16	6419	440	479	440
query17	1092	564	582	564
query18	1736	299	309	299
query19	283	158	150	150
query20	116	113	111	111
query21	221	107	107	107
query22	4305	4315	4179	4179
query23	34980	34213	34148	34148
query24	11486	2756	2869	2756
query25	650	401	422	401
query26	1311	168	165	165
query27	2280	297	299	297
query28	6392	2437	2454	2437
query29	832	440	429	429
query30	340	162	162	162
query31	1043	804	834	804
query32	90	58	59	58
query33	662	302	298	298
query34	924	524	536	524
query35	1096	910	922	910
query36	1088	963	947	947
query37	229	84	90	84
query38	4384	4417	4320	4320
query39	1493	1502	1427	1427
query40	285	107	102	102
query41	52	48	47	47
query42	124	105	100	100
query43	541	499	496	496
query44	1215	805	812	805
query45	203	167	171	167
query46	1138	703	706	703
query47	1925	1816	1822	1816
query48	428	324	332	324
query49	1240	451	423	423
query50	805	382	395	382
query51	7066	6983	6970	6970
query52	102	96	89	89
query53	261	179	183	179
query54	688	440	427	427
query55	81	77	81	77
query56	287	287	269	269
query57	1351	1189	1169	1169
query58	265	237	255	237
query59	3154	3131	2942	2942
query60	294	266	264	264
query61	107	107	105	105
query62	854	692	660	660
query63	224	189	188	188
query64	4639	641	611	611
query65	3265	3259	3212	3212
query66	1233	316	305	305
query67	16012	15569	15618	15569
query68	4586	540	556	540
query69	501	299	297	297
query70	1220	1170	1095	1095
query71	342	272	295	272
query72	6565	3922	4009	3922
query73	772	369	363	363
query74	9618	8981	9068	8981
query75	3441	2733	2724	2724
query76	2760	1003	966	966
query77	410	305	289	289
query78	10575	9591	9822	9591
query79	1104	604	598	598
query80	731	480	463	463
query81	559	242	243	242
query82	311	141	143	141
query83	177	139	141	139
query84	250	75	78	75
query85	1016	292	285	285
query86	327	282	297	282
query87	4744	4699	4686	4686
query88	3587	2231	2169	2169
query89	388	282	280	280
query90	2033	189	189	189
query91	139	110	107	107
query92	63	51	49	49
query93	1066	539	539	539
query94	984	307	292	292
query95	355	254	249	249
query96	618	282	291	282
query97	2855	2734	2697	2697
query98	220	194	199	194
query99	1538	1330	1325	1325
Total cold run time: 278432 ms
Total hot run time: 192710 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f67f94431d19751d1d7fbf52dd5619f178c11fc1, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.40	0.41	0.39
query6	1.17	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.53
query10	0.56	0.53	0.55
query11	0.14	0.10	0.10
query12	0.16	0.11	0.10
query13	0.63	0.60	0.61
query14	2.72	2.78	2.73
query15	0.91	0.83	0.85
query16	0.37	0.39	0.38
query17	1.01	1.05	1.04
query18	0.24	0.24	0.23
query19	1.86	1.86	1.96
query20	0.01	0.02	0.01
query21	15.37	0.59	0.58
query22	2.81	2.31	2.47
query23	16.88	1.12	0.79
query24	3.07	1.19	0.79
query25	0.11	0.19	0.10
query26	0.58	0.14	0.15
query27	0.06	0.05	0.04
query28	10.92	1.11	1.08
query29	12.58	3.34	3.29
query30	0.24	0.06	0.06
query31	2.88	0.37	0.38
query32	3.28	0.46	0.46
query33	2.97	3.03	3.03
query34	16.80	4.43	4.43
query35	4.54	4.44	4.51
query36	0.65	0.49	0.49
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.98 s
Total hot run time: 32.91 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit ab27598 into apache:master Nov 4, 2024
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 4, 2024
@zddr
[fix](auth)Fix concurrency issue during role manager upgrade (#42419)
ac602f4 
Change the map to a concurrency safe map，because there is a possibility
of concurrent read and write operations

For example, when a node performs the rectifyPrivs operation, it needs
to traverse 'roles' and may be adding or deleting roles at the same time
@github-actions github-actions bot mentioned this pull request Nov 4, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Nov 4, 2024
@zddr
[fix](auth)Fix concurrency issue during role manager upgrade (#42419)
df8d0a0 
Change the map to a concurrency safe map，because there is a possibility
of concurrent read and write operations

For example, when a node performs the rectifyPrivs operation, it needs
to traverse 'roles' and may be adding or deleting roles at the same time
@github-actions github-actions bot mentioned this pull request Nov 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
@github-actions @zddr
branch-3.0: [fix](auth)Fix concurrency issue during role manager upgr…
23f7bb9 
…ade (#43193)

Cherry-picked from #42419

Co-authored-by: zhangdong <493738387@qq.com>
yiguolei pushed a commit that referenced this pull request Nov 9, 2024
@github-actions @zddr
branch-2.1: [fix](auth)Fix concurrency issue during role manager upgr…
486dfe9 
…ade (#43194)

Cherry-picked from #42419

Co-authored-by: zhangdong <493738387@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@seawinde seawinde seawinde approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @doris-robot @morrySnow @seawinde @yiguolei @dataroaring