Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature-wip](Cloud) Support Azure Cloud Blob in FE #35990

Merged
merged 2 commits into from
Jun 24, 2024
Merged

[feature-wip](Cloud) Support Azure Cloud Blob in FE #35990

merged 2 commits into from
Jun 24, 2024

Conversation

ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jun 6, 2024

Proposed changes

As one subsequent pr of #35307, this pr tries to support Azure Cloud in FE. After this pr, we should be able to use Azure Blob for Cold Heat Separation and Storage Compute Separation along with backup restore to Azure Cloud.

For the data lakes support, you can add corresponding sdks to FE and implements the corresponding features.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@ByteYue ByteYue force-pushed the support_azure_in_fe branch from 242c485 to 1dc8472 Compare June 14, 2024 06:37
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

3 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2216755f57ab53919ab696505aeefab6a827e43, data reload: false

------ Round 1 ----------------------------------
q1	17600	5569	4257	4257
q2	2020	185	204	185
q3	10444	1116	1150	1116
q4	10187	752	822	752
q5	7452	2685	2692	2685
q6	224	137	136	136
q7	961	619	596	596
q8	9222	2073	2099	2073
q9	9065	6531	6496	6496
q10	8992	3721	3714	3714
q11	457	237	239	237
q12	460	232	226	226
q13	17781	2978	2999	2978
q14	271	220	221	220
q15	527	479	472	472
q16	521	383	377	377
q17	985	658	701	658
q18	8122	7507	7449	7449
q19	4628	1586	1434	1434
q20	635	332	338	332
q21	4917	3213	3987	3213
q22	387	325	333	325
Total cold run time: 115858 ms
Total hot run time: 39931 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4401	4246	4271	4246
q2	372	258	268	258
q3	2968	2917	2856	2856
q4	2035	1757	1746	1746
q5	5615	5557	5505	5505
q6	226	130	138	130
q7	2238	1883	1847	1847
q8	3302	3446	3471	3446
q9	8748	8721	8863	8721
q10	4115	3877	3797	3797
q11	593	486	493	486
q12	828	643	631	631
q13	15928	3172	3204	3172
q14	312	272	273	272
q15	527	476	483	476
q16	482	427	421	421
q17	1844	1509	1519	1509
q18	8185	7845	7739	7739
q19	1795	1628	1729	1628
q20	3126	1896	1889	1889
q21	5215	5053	4850	4850
q22	598	526	521	521
Total cold run time: 73453 ms
Total hot run time: 56146 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173609 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2216755f57ab53919ab696505aeefab6a827e43, data reload: false

query1	922	395	381	381
query2	6457	2504	2327	2327
query3	6640	205	205	205
query4	19210	17558	17467	17467
query5	3571	471	471	471
query6	237	154	158	154
query7	4572	299	303	299
query8	339	285	269	269
query9	8488	2398	2406	2398
query10	566	288	278	278
query11	10806	10235	10132	10132
query12	116	85	95	85
query13	1632	357	362	357
query14	10139	6307	6012	6012
query15	222	186	192	186
query16	7532	265	266	265
query17	1494	552	512	512
query18	1916	275	270	270
query19	198	153	151	151
query20	92	83	82	82
query21	205	135	134	134
query22	4481	4271	4209	4209
query23	33692	33699	33623	33623
query24	11148	3057	2919	2919
query25	598	389	370	370
query26	712	157	157	157
query27	2345	342	325	325
query28	5688	2101	2088	2088
query29	901	630	617	617
query30	249	159	154	154
query31	977	755	766	755
query32	93	50	55	50
query33	713	286	278	278
query34	996	468	463	463
query35	756	648	635	635
query36	1130	961	983	961
query37	141	74	75	74
query38	2922	2837	2843	2837
query39	908	843	828	828
query40	201	124	131	124
query41	60	58	56	56
query42	108	103	101	101
query43	606	546	518	518
query44	1165	715	731	715
query45	194	162	164	162
query46	1073	709	743	709
query47	1877	1779	1815	1779
query48	378	292	297	292
query49	838	401	406	401
query50	779	389	382	382
query51	6802	6686	6681	6681
query52	104	91	88	88
query53	358	292	283	283
query54	867	438	439	438
query55	73	73	74	73
query56	266	252	253	252
query57	1137	1060	1044	1044
query58	265	246	247	246
query59	3648	3278	3066	3066
query60	308	269	277	269
query61	91	90	85	85
query62	620	459	453	453
query63	314	292	299	292
query64	8496	2218	1777	1777
query65	3166	3121	3074	3074
query66	776	335	355	335
query67	15657	15043	14937	14937
query68	6152	538	532	532
query69	628	459	422	422
query70	1131	1083	1157	1083
query71	479	275	278	275
query72	7687	5781	5899	5781
query73	760	326	325	325
query74	5927	5589	5542	5542
query75	3925	2687	2694	2687
query76	3540	1034	949	949
query77	666	303	303	303
query78	10523	9870	9682	9682
query79	3403	505	513	505
query80	2619	489	482	482
query81	553	225	227	225
query82	544	104	102	102
query83	294	178	191	178
query84	287	98	93	93
query85	1156	330	331	330
query86	379	288	302	288
query87	3282	3057	3077	3057
query88	3670	2351	2458	2351
query89	489	382	374	374
query90	1840	193	186	186
query91	124	97	96	96
query92	55	47	49	47
query93	4629	495	495	495
query94	1155	192	188	188
query95	404	306	309	306
query96	606	262	263	262
query97	3207	3050	3023	3023
query98	209	203	187	187
query99	1348	859	843	843
Total cold run time: 275702 ms
Total hot run time: 173609 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2216755f57ab53919ab696505aeefab6a827e43, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.49	0.48
query6	1.16	0.73	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.53	0.50	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.59	0.60	0.59
query14	0.78	0.78	0.77
query15	0.83	0.81	0.82
query16	0.37	0.36	0.36
query17	0.95	0.95	0.96
query18	0.22	0.25	0.23
query19	1.79	1.67	1.72
query20	0.01	0.01	0.01
query21	15.41	0.65	0.64
query22	4.18	7.31	1.98
query23	18.32	1.41	1.26
query24	2.09	0.21	0.22
query25	0.16	0.09	0.08
query26	0.28	0.17	0.17
query27	0.08	0.08	0.08
query28	13.21	1.01	1.00
query29	12.64	3.32	3.25
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.46	0.46
query33	2.83	2.90	2.88
query34	17.25	4.45	4.42
query35	4.50	4.48	4.49
query36	0.66	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.05
Total cold run time: 109.43 s
Total hot run time: 30.49 s

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

5 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@ByteYue ByteYue force-pushed the support_azure_in_fe branch from c22cf2c to e4ce2a4 Compare June 19, 2024 06:46
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 19, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c3fe5a9f444ae04f3bb954f3ae8265599e7729f, data reload: false

------ Round 1 ----------------------------------
q1	17605	5149	4279	4279
q2	2020	187	193	187
q3	10467	1122	1111	1111
q4	10193	797	899	797
q5	7482	2716	2651	2651
q6	217	137	134	134
q7	966	607	601	601
q8	9213	2076	2109	2076
q9	9023	6518	6483	6483
q10	8913	3718	3701	3701
q11	452	240	236	236
q12	405	228	215	215
q13	17788	2976	2972	2972
q14	271	222	227	222
q15	522	480	477	477
q16	545	377	374	374
q17	977	674	720	674
q18	8152	7559	7331	7331
q19	7162	1590	1403	1403
q20	666	331	337	331
q21	4932	3137	3134	3134
q22	397	327	329	327
Total cold run time: 118368 ms
Total hot run time: 39716 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4244	4259	4244
q2	372	262	261	261
q3	3025	2874	2853	2853
q4	1979	1710	1770	1710
q5	5625	5494	5462	5462
q6	226	137	130	130
q7	2249	1866	1845	1845
q8	3276	3453	3428	3428
q9	8643	8746	8672	8672
q10	4142	3794	3811	3794
q11	614	502	504	502
q12	778	642	643	642
q13	16982	3139	3190	3139
q14	321	278	273	273
q15	546	467	491	467
q16	492	434	445	434
q17	1828	1502	1483	1483
q18	8116	7886	7663	7663
q19	1821	1606	1593	1593
q20	2421	1864	1857	1857
q21	5156	4982	4790	4790
q22	627	543	550	543
Total cold run time: 73628 ms
Total hot run time: 55785 ms

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40071 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30f6581baeb5fb10ad9f22092e2043221338d15d, data reload: false

------ Round 1 ----------------------------------
q1	17642	4419	4350	4350
q2	2026	187	191	187
q3	10454	1198	1119	1119
q4	10192	782	720	720
q5	7455	2700	2639	2639
q6	223	135	131	131
q7	952	606	595	595
q8	9225	2103	2088	2088
q9	8974	6568	6467	6467
q10	9049	3769	3720	3720
q11	474	241	236	236
q12	483	227	225	225
q13	18708	2976	2967	2967
q14	278	217	222	217
q15	529	477	491	477
q16	512	381	379	379
q17	972	631	651	631
q18	8135	7431	7524	7431
q19	7480	1492	1476	1476
q20	666	334	328	328
q21	5011	3356	4044	3356
q22	387	334	332	332
Total cold run time: 119827 ms
Total hot run time: 40071 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4420	4200	4191	4191
q2	362	270	263	263
q3	3134	2950	2904	2904
q4	1936	1743	1733	1733
q5	5582	5523	5456	5456
q6	233	134	135	134
q7	2279	1862	1869	1862
q8	3317	3442	3427	3427
q9	8730	8878	8744	8744
q10	4182	3714	3729	3714
q11	584	493	478	478
q12	814	652	614	614
q13	17251	3160	3177	3160
q14	293	286	263	263
q15	544	484	494	484
q16	520	442	434	434
q17	1838	1536	1493	1493
q18	8149	8114	7766	7766
q19	1804	1600	1496	1496
q20	2105	1880	1854	1854
q21	8918	4811	4835	4811
q22	670	579	564	564
Total cold run time: 77665 ms
Total hot run time: 55845 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30f6581baeb5fb10ad9f22092e2043221338d15d, data reload: false

query1	918	399	375	375
query2	6444	2356	2397	2356
query3	6633	210	216	210
query4	19114	17288	17292	17288
query5	3694	492	493	492
query6	247	163	162	162
query7	4586	293	290	290
query8	312	292	292	292
query9	8594	2372	2349	2349
query10	575	300	297	297
query11	10610	10008	10022	10008
query12	131	90	86	86
query13	1643	385	382	382
query14	10160	7666	7748	7666
query15	238	187	194	187
query16	7967	278	275	275
query17	1896	562	536	536
query18	2074	282	287	282
query19	202	167	154	154
query20	94	84	84	84
query21	210	137	128	128
query22	4348	4206	4053	4053
query23	34185	33838	33625	33625
query24	10795	2940	3014	2940
query25	578	376	362	362
query26	693	158	158	158
query27	2349	333	321	321
query28	6067	2168	2160	2160
query29	901	635	605	605
query30	262	157	150	150
query31	986	754	739	739
query32	96	55	52	52
query33	658	300	285	285
query34	895	491	468	468
query35	776	644	643	643
query36	1142	1008	939	939
query37	142	86	75	75
query38	2935	2837	2820	2820
query39	872	844	869	844
query40	210	121	124	121
query41	53	49	52	49
query42	117	97	100	97
query43	545	525	534	525
query44	1068	729	740	729
query45	196	169	166	166
query46	1063	714	747	714
query47	1887	1776	1782	1776
query48	389	309	298	298
query49	865	403	408	403
query50	765	409	394	394
query51	7010	6865	6767	6767
query52	110	94	95	94
query53	359	295	282	282
query54	850	450	452	450
query55	73	73	73	73
query56	290	274	261	261
query57	1103	1061	1056	1056
query58	262	261	235	235
query59	3235	3264	3005	3005
query60	310	265	328	265
query61	100	91	91	91
query62	594	452	433	433
query63	312	284	291	284
query64	8514	2244	1721	1721
query65	3206	3126	3095	3095
query66	745	318	321	318
query67	15442	15071	15010	15010
query68	5373	560	539	539
query69	630	460	394	394
query70	1202	1055	1121	1055
query71	464	275	271	271
query72	7334	5849	5439	5439
query73	781	326	325	325
query74	5890	5668	5559	5559
query75	3705	2674	2623	2623
query76	3365	898	964	898
query77	627	308	314	308
query78	10575	9946	9846	9846
query79	3048	510	517	510
query80	2119	469	463	463
query81	532	221	221	221
query82	1444	110	109	109
query83	324	175	169	169
query84	268	91	97	91
query85	1415	335	267	267
query86	473	334	324	324
query87	3306	3088	3159	3088
query88	3901	2490	2451	2451
query89	477	383	376	376
query90	1745	193	189	189
query91	128	102	98	98
query92	58	51	54	51
query93	2558	511	499	499
query94	1126	188	192	188
query95	403	317	314	314
query96	591	277	275	275
query97	3283	3032	3014	3014
query98	221	208	198	198
query99	1199	858	841	841
Total cold run time: 273260 ms
Total hot run time: 174843 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30f6581baeb5fb10ad9f22092e2043221338d15d, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.52	0.53
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.79	0.78
query15	0.85	0.82	0.82
query16	0.36	0.36	0.37
query17	0.99	0.97	0.99
query18	0.25	0.24	0.25
query19	1.91	1.73	1.80
query20	0.02	0.00	0.01
query21	15.40	0.67	0.66
query22	4.11	6.98	2.45
query23	18.45	1.43	1.28
query24	2.08	0.24	0.22
query25	0.16	0.09	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.26	1.04	1.00
query29	12.64	3.38	3.35
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.28	0.47	0.46
query33	2.87	2.86	2.92
query34	17.07	4.46	4.56
query35	4.46	4.51	4.45
query36	0.64	0.46	0.45
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.15	0.15
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 31.18 s

@ByteYue ByteYue mentioned this pull request Jun 22, 2024
Merged
@ByteYue ByteYue requested review from morningman and gavinchou June 22, 2024 05:48
gavinchou pushed a commit that referenced this pull request Jun 22, 2024
@ByteYue
[chore](Regression) Add provider conf in regression's pipeline conf (#…
abef1dd 
…36695)

Previously the conf of pipeline doesn't contain the provider of the
Object Storage it uses. When adding azure's support like #35990, user
should specify the vendor they use or just doesn't set the "provider"
property when using S3-like object storage. So i add this conf.
@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 22, 2024

run buildall

@ByteYue
add one glob implement
6bc8824 
add provider in conf

handle export

forbid azure tvf

check provider property
@ByteYue ByteYue force-pushed the support_azure_in_fe branch from 30f6581 to 6bc8824 Compare June 22, 2024 13:08
@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 22, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40091 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bc88240356670f795da527450534a91747056ad, data reload: false

------ Round 1 ----------------------------------
q1	17672	4922	4251	4251
q2	2042	190	198	190
q3	10531	1166	1131	1131
q4	10199	852	787	787
q5	7480	2697	2724	2697
q6	223	132	135	132
q7	968	620	597	597
q8	9204	2079	2059	2059
q9	8968	6478	6484	6478
q10	9002	3791	3750	3750
q11	438	238	236	236
q12	451	248	228	228
q13	17772	2977	2948	2948
q14	269	215	223	215
q15	528	474	486	474
q16	537	382	382	382
q17	984	743	697	697
q18	8013	7480	7515	7480
q19	7278	1423	1462	1423
q20	658	332	336	332
q21	4944	3275	3270	3270
q22	407	334	344	334
Total cold run time: 118568 ms
Total hot run time: 40091 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4287	4242	4242
q2	380	264	269	264
q3	3067	2909	2940	2909
q4	2000	1787	1828	1787
q5	5509	5508	5468	5468
q6	225	131	148	131
q7	2281	1872	1893	1872
q8	3271	3455	3405	3405
q9	8602	8772	8783	8772
q10	4190	3703	3826	3703
q11	576	492	505	492
q12	823	632	640	632
q13	17155	3171	3162	3162
q14	307	267	280	267
q15	515	484	490	484
q16	484	455	435	435
q17	1812	1493	1488	1488
q18	8218	7872	7820	7820
q19	1814	1667	1582	1582
q20	2145	1860	1838	1838
q21	7120	4878	4768	4768
q22	631	546	573	546
Total cold run time: 75492 ms
Total hot run time: 56067 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bc88240356670f795da527450534a91747056ad, data reload: false

query1	917	387	382	382
query2	6447	2470	2402	2402
query3	6631	212	230	212
query4	19425	17265	17353	17265
query5	3537	488	459	459
query6	236	161	178	161
query7	4584	304	289	289
query8	343	303	307	303
query9	8425	2373	2352	2352
query10	563	298	283	283
query11	10501	10148	10091	10091
query12	121	98	91	91
query13	1640	383	378	378
query14	9326	7548	6418	6418
query15	243	193	190	190
query16	8091	281	257	257
query17	1899	545	523	523
query18	2094	275	286	275
query19	198	149	152	149
query20	93	85	83	83
query21	214	128	123	123
query22	4224	4077	3961	3961
query23	33815	33763	33542	33542
query24	9629	2850	2913	2850
query25	587	373	375	373
query26	709	151	161	151
query27	2231	330	330	330
query28	6008	2126	2120	2120
query29	916	656	644	644
query30	256	151	149	149
query31	979	767	740	740
query32	95	56	55	55
query33	645	289	290	289
query34	895	483	484	483
query35	708	637	626	626
query36	1153	942	994	942
query37	137	73	73	73
query38	2986	2907	2855	2855
query39	901	858	840	840
query40	208	144	127	127
query41	59	51	51	51
query42	110	99	109	99
query43	593	542	526	526
query44	1085	728	738	728
query45	191	159	168	159
query46	1065	684	723	684
query47	1865	1764	1781	1764
query48	376	311	296	296
query49	840	415	423	415
query50	761	389	386	386
query51	6893	6643	6746	6643
query52	102	95	92	92
query53	359	308	290	290
query54	836	435	452	435
query55	75	75	71	71
query56	276	263	264	263
query57	1176	1046	1064	1046
query58	234	247	232	232
query59	3529	3227	3214	3214
query60	292	273	278	273
query61	97	92	90	90
query62	590	489	424	424
query63	318	293	287	287
query64	8523	2186	1750	1750
query65	3161	3135	3087	3087
query66	741	324	337	324
query67	15341	15217	14737	14737
query68	4636	543	539	539
query69	623	454	408	408
query70	1207	1056	1085	1056
query71	449	280	271	271
query72	7372	5508	2745	2745
query73	758	374	324	324
query74	5948	5530	5414	5414
query75	3403	2637	2678	2637
query76	2728	960	863	863
query77	609	303	294	294
query78	10706	9894	9751	9751
query79	2207	516	502	502
query80	907	486	453	453
query81	586	221	225	221
query82	742	103	103	103
query83	268	173	170	170
query84	241	91	83	83
query85	1339	285	267	267
query86	471	312	291	291
query87	3350	3079	3064	3064
query88	4101	2452	2439	2439
query89	469	374	376	374
query90	1703	188	187	187
query91	130	100	112	100
query92	59	49	54	49
query93	2258	515	515	515
query94	1117	197	195	195
query95	412	383	317	317
query96	602	273	276	273
query97	3280	3004	3021	3004
query98	215	206	193	193
query99	1121	857	854	854
Total cold run time: 266115 ms
Total hot run time: 170158 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bc88240356670f795da527450534a91747056ad, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.08
query5	0.49	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.55	0.50	0.49
query10	0.55	0.56	0.54
query11	0.15	0.12	0.12
query12	0.15	0.11	0.11
query13	0.58	0.58	0.60
query14	0.78	0.78	0.77
query15	0.84	0.81	0.81
query16	0.37	0.37	0.37
query17	1.04	1.03	1.03
query18	0.23	0.24	0.25
query19	1.80	1.67	1.69
query20	0.01	0.02	0.02
query21	15.41	0.65	0.65
query22	4.30	7.00	1.89
query23	18.60	1.43	1.26
query24	2.17	0.24	0.23
query25	0.16	0.08	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.09
query28	13.19	1.02	0.98
query29	12.58	3.47	3.36
query30	0.26	0.07	0.06
query31	2.86	0.39	0.39
query32	3.30	0.48	0.46
query33	2.89	2.92	2.89
query34	17.15	4.39	4.39
query35	4.48	4.48	4.50
query36	0.66	0.46	0.48
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 109.9 s
Total hot run time: 30.66 s

@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 23, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39996 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfe9a19f9d1963cdbc88a15b7918c587f09ef8aa, data reload: false

------ Round 1 ----------------------------------
q1	18028	4366	4357	4357
q2	2005	192	204	192
q3	10444	1075	1018	1018
q4	10203	820	785	785
q5	8138	2693	2646	2646
q6	222	142	131	131
q7	964	624	614	614
q8	9220	2092	2060	2060
q9	9436	6591	6608	6591
q10	9264	3900	3840	3840
q11	960	297	240	240
q12	789	243	240	240
q13	19492	3032	3016	3016
q14	265	223	223	223
q15	510	485	492	485
q16	536	379	380	379
q17	969	792	696	696
q18	8113	7482	7307	7307
q19	7581	1408	1477	1408
q20	651	318	341	318
q21	4859	3119	3862	3119
q22	386	342	331	331
Total cold run time: 123035 ms
Total hot run time: 39996 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4427	4229	4318	4229
q2	382	279	289	279
q3	3155	2897	2908	2897
q4	2049	1731	1664	1664
q5	5488	5424	5566	5424
q6	225	130	133	130
q7	2283	1847	1858	1847
q8	3282	3418	3430	3418
q9	8778	8749	8769	8749
q10	3940	3841	3745	3745
q11	572	523	494	494
q12	801	664	634	634
q13	17165	3141	3172	3141
q14	311	274	295	274
q15	526	472	488	472
q16	496	424	414	414
q17	1817	1527	1507	1507
q18	8053	7847	7864	7847
q19	1814	1573	1742	1573
q20	3114	1847	1848	1847
q21	5210	4846	4913	4846
q22	825	571	554	554
Total cold run time: 74713 ms
Total hot run time: 55985 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173160 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dfe9a19f9d1963cdbc88a15b7918c587f09ef8aa, data reload: false

query1	921	391	377	377
query2	6383	2404	2343	2343
query3	6630	207	216	207
query4	19435	17256	17327	17256
query5	3708	486	480	480
query6	239	168	180	168
query7	4586	297	301	297
query8	314	294	302	294
query9	8607	2370	2342	2342
query10	576	295	293	293
query11	10537	10114	9803	9803
query12	122	84	100	84
query13	1632	373	396	373
query14	9981	7128	6226	6226
query15	220	190	190	190
query16	7342	270	265	265
query17	1415	542	510	510
query18	1874	275	284	275
query19	195	153	152	152
query20	89	82	81	81
query21	209	126	125	125
query22	4401	4050	4044	4044
query23	33663	33508	33755	33508
query24	11309	2924	2875	2875
query25	604	365	383	365
query26	934	167	158	158
query27	2402	343	328	328
query28	6331	2133	2124	2124
query29	876	665	640	640
query30	282	160	164	160
query31	1025	828	750	750
query32	91	55	56	55
query33	781	304	279	279
query34	1043	502	494	494
query35	751	649	653	649
query36	1141	962	955	955
query37	141	79	76	76
query38	2950	2844	2798	2798
query39	905	855	827	827
query40	213	133	131	131
query41	61	53	56	53
query42	118	101	111	101
query43	569	554	584	554
query44	1222	734	748	734
query45	197	161	165	161
query46	1071	706	690	690
query47	1856	1781	1767	1767
query48	389	310	311	310
query49	905	431	438	431
query50	757	389	404	389
query51	6930	6774	6667	6667
query52	98	100	90	90
query53	359	286	292	286
query54	931	465	458	458
query55	79	78	78	78
query56	310	273	288	273
query57	1163	1051	1064	1051
query58	272	251	260	251
query59	3665	3102	3369	3102
query60	324	291	302	291
query61	112	113	109	109
query62	616	448	439	439
query63	325	292	291	291
query64	8774	2348	1841	1841
query65	3216	3097	3119	3097
query66	974	334	335	334
query67	15625	15063	14989	14989
query68	4564	550	551	550
query69	687	533	390	390
query70	1202	1157	1165	1157
query71	468	282	279	279
query72	9218	5500	5841	5500
query73	748	329	331	329
query74	5876	5585	5524	5524
query75	3636	2610	2665	2610
query76	2910	918	940	918
query77	679	316	317	316
query78	10381	9791	9705	9705
query79	8534	586	535	535
query80	1457	491	478	478
query81	563	213	236	213
query82	1321	106	100	100
query83	208	170	170	170
query84	277	83	88	83
query85	1368	305	265	265
query86	451	321	342	321
query87	3277	3071	3104	3071
query88	5110	2460	2446	2446
query89	561	377	374	374
query90	1842	190	190	190
query91	134	102	100	100
query92	63	50	53	50
query93	6497	509	513	509
query94	1171	195	188	188
query95	409	332	306	306
query96	601	272	269	269
query97	3158	3055	2997	2997
query98	218	197	194	194
query99	1275	829	888	829
Total cold run time: 284640 ms
Total hot run time: 173160 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dfe9a19f9d1963cdbc88a15b7918c587f09ef8aa, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.11	0.08
query5	0.48	0.47	0.48
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.49
query10	0.55	0.54	0.53
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.79	0.76	0.80
query15	0.84	0.80	0.80
query16	0.37	0.36	0.35
query17	1.01	0.95	1.01
query18	0.20	0.24	0.25
query19	1.79	1.71	1.71
query20	0.01	0.01	0.01
query21	15.43	0.65	0.65
query22	4.81	6.68	1.89
query23	18.28	1.48	1.15
query24	2.22	0.24	0.22
query25	0.15	0.09	0.09
query26	0.27	0.17	0.18
query27	0.08	0.07	0.08
query28	13.17	1.01	1.00
query29	12.60	3.39	3.39
query30	0.25	0.07	0.06
query31	2.84	0.38	0.39
query32	3.32	0.47	0.46
query33	2.86	2.94	2.90
query34	17.05	4.40	4.39
query35	4.48	4.52	4.50
query36	0.62	0.46	0.46
query37	0.19	0.17	0.16
query38	0.15	0.16	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.88 s
Total hot run time: 30.5 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit e11f516 into apache:master Jun 24, 2024
25 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jun 26, 2024
@ByteYue @dataroaring
[chore](Regression) Add provider conf in regression's pipeline conf (#…
70189e4 
…36695)

Previously the conf of pipeline doesn't contain the provider of the
Object Storage it uses. When adding azure's support like #35990, user
should specify the vendor they use or just doesn't set the "provider"
property when using S3-like object storage. So i add this conf.
dataroaring pushed a commit that referenced this pull request Jun 26, 2024
@ByteYue @dataroaring
[feature-wip](Cloud) Support Azure Cloud Blob in FE (#35990)
84e9642 
As one subsequent pr of #35307, this pr tries to support Azure Cloud in
FE. After this pr, we should be able to use Azure Blob for Cold Heat
Separation and Storage Compute Separation along with backup restore to
Azure Cloud.

For the data lakes support, you can add corresponding sdks to FE and
implements the corresponding features.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ByteYue @doris-robot @dataroaring @gavinchou @morningman