Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](merge-cloud) Add meta service client and proxy #29832

Merged
merged 1 commit into from
Jan 11, 2024
Merged

[feature](merge-cloud) Add meta service client and proxy #29832

merged 1 commit into from
Jan 11, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jan 11, 2024 •
edited by dataroaring
Loading

Proposed changes

FE gets versions, txn info from metaservice via MetaServiceClient.
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@w41ter
Copy link
Contributor Author

w41ter commented Jan 11, 2024

run buildall

dataroaring
dataroaring previously approved these changes Jan 11, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

deardeng
deardeng approved these changes Jan 11, 2024
View reviewed changes
Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM

@w41ter w41ter force-pushed the cherry-pick/fe_cloud_rpc branch from 08dcb0a to 9041d1f Compare January 11, 2024 03:04
@w41ter
Copy link
Contributor Author

w41ter commented Jan 11, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 11, 2024
@w41ter @deardeng
@mymeiyi @SWJTU-ZhangLei @gavinchou @luwei16 @Lchangliang @liaoxin01 @panDing19
[feature](merge-cloud) Add meta service client and proxy
a11197f 
Co-authored-by: deardeng <565620795@qq.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: Lei Zhang <27994433+SWJTU-ZhangLei@users.noreply.github.com>
Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: Lightman <31928846+Lchangliang@users.noreply.github.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: panDing19 <56944854+panDing19@users.noreply.github.com>
@w41ter w41ter force-pushed the cherry-pick/fe_cloud_rpc branch from 9041d1f to a11197f Compare January 11, 2024 03:22
@w41ter
Copy link
Contributor Author

w41ter commented Jan 11, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a11197fb2bbf81c26744a6d1e52641f64572cb35, data reload: false

------ Round 1 ----------------------------------
q1	17989	5349	5030	5030
q2	2049	155	138	138
q3	10748	1115	1218	1115
q4	10293	787	854	787
q5	7792	3214	3199	3199
q6	204	123	126	123
q7	875	493	491	491
q8	9287	1994	2016	1994
q9	7393	6524	6447	6447
q10	8307	3111	3073	3073
q11	418	199	195	195
q12	358	195	198	195
q13	18102	3424	3428	3424
q14	243	215	209	209
q15	559	521	511	511
q16	435	379	386	379
q17	950	563	528	528
q18	7410	6755	6747	6747
q19	1609	1316	1343	1316
q20	541	307	307	307
q21	2790	2419	2505	2419
q22	363	318	307	307
Total cold run time: 108715 ms
Total hot run time: 38934 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4966	4963	5015	4963
q2	315	201	203	201
q3	3371	3330	3311	3311
q4	2255	2265	2258	2258
q5	5822	5802	5825	5802
q6	197	117	122	117
q7	2359	1855	1867	1855
q8	3459	3546	3577	3546
q9	8938	8883	8835	8835
q10	3763	3815	3841	3815
q11	552	440	429	429
q12	796	621	604	604
q13	6357	3172	3162	3162
q14	288	259	264	259
q15	565	512	506	506
q16	509	460	447	447
q17	2053	2023	2036	2023
q18	8811	8376	8305	8305
q19	1626	1590	1611	1590
q20	2178	1964	1926	1926
q21	6029	5618	5766	5618
q22	599	510	502	502
Total cold run time: 65808 ms
Total hot run time: 60074 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178016 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a11197fb2bbf81c26744a6d1e52641f64572cb35, data reload: false

query1	928	340	336	336
query2	6705	1925	2088	1925
query3	6712	210	214	210
query4	26173	22096	22015	22015
query5	5128	481	512	481
query6	264	188	190	188
query7	4623	283	276	276
query8	272	223	207	207
query9	9150	2749	2835	2749
query10	471	230	250	230
query11	16120	15191	15156	15156
query12	138	79	76	76
query13	1624	335	333	333
query14	11826	7241	7257	7241
query15	234	195	197	195
query16	6488	274	268	268
query17	1887	494	479	479
query18	1941	279	270	270
query19	278	143	144	143
query20	86	80	79	79
query21	189	91	88	88
query22	4938	4848	4754	4754
query23	32247	31192	31205	31192
query24	11110	2825	2795	2795
query25	565	335	334	334
query26	1667	154	158	154
query27	2827	282	278	278
query28	7099	1856	1859	1856
query29	1521	387	386	386
query30	279	144	152	144
query31	1006	783	793	783
query32	98	56	60	56
query33	722	276	266	266
query34	910	437	446	437
query35	917	718	717	717
query36	1299	1184	1249	1184
query37	100	69	67	67
query38	3408	3304	3276	3276
query39	1337	1281	1293	1281
query40	197	95	91	91
query41	38	34	34	34
query42	101	92	99	92
query43	554	497	483	483
query44	1066	703	727	703
query45	216	187	184	184
query46	1040	658	658	658
query47	1636	1645	1581	1581
query48	395	310	303	303
query49	1160	315	327	315
query50	774	347	320	320
query51	5335	5184	5219	5184
query52	86	95	89	89
query53	224	153	152	152
query54	1386	559	571	559
query55	105	87	90	87
query56	222	200	195	195
query57	1034	962	929	929
query58	249	211	215	211
query59	2668	2554	2560	2554
query60	255	220	224	220
query61	82	82	80	80
query62	650	463	461	461
query63	169	146	149	146
query64	4903	1709	1700	1700
query65	3360	3263	3250	3250
query66	1219	331	342	331
query67	15725	15028	15012	15012
query68	10065	550	536	536
query69	570	282	279	279
query70	1720	1458	1453	1453
query71	475	233	221	221
query72	4484	2850	2863	2850
query73	1853	317	308	308
query74	6991	6509	6445	6445
query75	4662	2338	2289	2289
query76	6299	1117	1113	1113
query77	676	287	286	286
query78	9555	8623	8781	8623
query79	1037	508	503	503
query80	538	348	358	348
query81	473	209	217	209
query82	177	97	90	90
query83	174	142	136	136
query84	262	63	55	55
query85	957	287	287	287
query86	437	356	382	356
query87	3509	3376	3394	3376
query88	2894	2257	2254	2254
query89	325	272	246	246
query90	1841	201	201	201
query91	166	130	138	130
query92	58	56	56	56
query93	1095	436	434	434
query94	782	187	192	187
query95	504	433	425	425
query96	628	321	319	319
query97	4283	4152	4190	4152
query98	208	191	181	181
query99	1078	839	840	839
Total cold run time: 286912 ms
Total hot run time: 178016 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a11197fb2bbf81c26744a6d1e52641f64572cb35, data reload: false

query1	0.07	0.06	0.05
query2	0.06	0.02	0.02
query3	0.25	0.11	0.11
query4	1.79	0.12	0.12
query5	0.52	0.52	0.52
query6	1.35	0.64	0.62
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.58	0.51	0.51
query10	0.56	0.55	0.55
query11	0.13	0.09	0.08
query12	0.12	0.10	0.09
query13	0.62	0.60	0.60
query14	0.77	0.82	0.81
query15	0.82	0.80	0.79
query16	0.36	0.35	0.36
query17	0.96	0.99	0.98
query18	0.27	0.27	0.26
query19	1.83	1.76	1.71
query20	0.01	0.01	0.01
query21	15.39	0.58	0.57
query22	3.05	2.47	1.91
query23	17.20	0.86	0.74
query24	2.46	4.01	1.38
query25	2.00	0.15	0.15
query26	0.13	0.14	0.13
query27	0.15	0.14	0.14
query28	9.00	0.80	0.79
query29	12.66	3.24	3.31
query30	0.51	0.48	0.49
query31	2.78	0.35	0.35
query32	3.38	0.50	0.48
query33	3.25	3.23	3.23
query34	15.78	4.18	4.15
query35	4.17	4.21	4.16
query36	1.10	1.06	1.04
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.06	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.52 s
Total hot run time: 31.44 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a11197fb2bbf81c26744a6d1e52641f64572cb35 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.8 seconds inserted 10000000 Rows, about 480K ops/s
storage size: 17183941313 Bytes

dataroaring
dataroaring approved these changes Jan 11, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit d673afa into apache:master Jan 11, 2024
19 checks passed
yiguolei pushed a commit that referenced this pull request Jan 12, 2024
@w41ter @deardeng
@mymeiyi @SWJTU-ZhangLei @gavinchou @luwei16 @Lchangliang @liaoxin01 @panDing19 @Doris-Extras
[feature](merge-cloud) Add meta service client and proxy (#29832)
90c98a3 
Co-authored-by: deardeng <565620795@qq.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: Lei Zhang <27994433+SWJTU-ZhangLei@users.noreply.github.com>
Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: Lightman <31928846+Lchangliang@users.noreply.github.com>
Co-authored-by: Xin Liao <liaoxinbit@126.com>
Co-authored-by: panDing19 <56944854+panDing19@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deardeng deardeng deardeng approved these changes

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@w41ter @doris-robot @deardeng @gavinchou @dataroaring