Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](partitionkey) serialize expr in PartitionKey via gson #37277

Merged
merged 4 commits into from
Jul 5, 2024
Merged

[fix](partitionkey) serialize expr in PartitionKey via gson #37277

merged 4 commits into from
Jul 5, 2024

Conversation

dataroaring
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

gavinchou
gavinchou previously approved these changes Jul 4, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added meta-change and removed approved Indicates a PR has been approved by one committer. labels Jul 4, 2024
@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring dataroaring changed the title [fix](partitionkey) keep date type for partitionkey [fix](partitionkey) [fix](meta) serialize expr in PartitionKey via gson Jul 4, 2024
@dataroaring dataroaring changed the title [fix](partitionkey) [fix](meta) serialize expr in PartitionKey via gson [fix](partitionkey) serialize expr in PartitionKey via gson Jul 4, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40124 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cdb1d1de6c7f8b0e5334d8fe070aa53e6b34968a, data reload: false

------ Round 1 ----------------------------------
q1	18211	4734	4309	4309
q2	2032	199	197	197
q3	10499	1143	1110	1110
q4	10220	778	861	778
q5	7518	2700	2656	2656
q6	222	140	137	137
q7	961	595	599	595
q8	9230	2084	2117	2084
q9	8941	6513	6510	6510
q10	9019	3766	3756	3756
q11	474	249	245	245
q12	413	234	236	234
q13	17764	2964	2982	2964
q14	259	218	226	218
q15	530	493	481	481
q16	528	376	389	376
q17	978	678	709	678
q18	8042	7486	7510	7486
q19	6427	1584	1431	1431
q20	667	344	338	338
q21	4900	3264	3205	3205
q22	395	336	341	336
Total cold run time: 118230 ms
Total hot run time: 40124 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4430	4236	4265	4236
q2	371	270	268	268
q3	3035	2863	2897	2863
q4	1969	1656	1734	1656
q5	5597	5495	5477	5477
q6	224	132	131	131
q7	2231	1876	1813	1813
q8	3238	3423	3448	3423
q9	8720	8759	8818	8759
q10	4126	3813	3800	3800
q11	608	495	504	495
q12	804	656	680	656
q13	17419	3168	3194	3168
q14	308	277	295	277
q15	529	483	479	479
q16	497	436	432	432
q17	1815	1580	1539	1539
q18	8127	7993	7921	7921
q19	1792	1684	1560	1560
q20	2207	1857	1892	1857
q21	5155	4758	4932	4758
q22	627	573	537	537
Total cold run time: 73829 ms
Total hot run time: 56105 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cdb1d1de6c7f8b0e5334d8fe070aa53e6b34968a, data reload: false

query1	923	398	380	380
query2	6414	2420	2392	2392
query3	6637	209	212	209
query4	20768	17514	17231	17231
query5	3639	498	509	498
query6	287	192	178	178
query7	4591	290	285	285
query8	326	280	285	280
query9	8467	2409	2396	2396
query10	562	307	280	280
query11	10640	9968	10148	9968
query12	117	82	80	80
query13	1650	364	363	363
query14	10239	6152	7837	6152
query15	229	182	189	182
query16	7765	308	303	303
query17	1725	564	521	521
query18	1945	269	275	269
query19	210	156	157	156
query20	89	82	85	82
query21	219	132	127	127
query22	4398	4037	3941	3941
query23	33851	33616	33624	33616
query24	11178	2922	2917	2917
query25	634	380	436	380
query26	1244	166	156	156
query27	2518	326	323	323
query28	7235	2138	2140	2138
query29	916	659	641	641
query30	259	159	153	153
query31	1032	776	735	735
query32	87	53	53	53
query33	763	298	289	289
query34	1058	478	495	478
query35	758	647	634	634
query36	1131	958	941	941
query37	167	88	89	88
query38	2934	2793	2833	2793
query39	902	840	847	840
query40	215	130	160	130
query41	56	49	51	49
query42	105	103	108	103
query43	598	580	564	564
query44	1196	742	720	720
query45	192	170	163	163
query46	1097	723	722	722
query47	1847	1743	1779	1743
query48	371	310	296	296
query49	877	403	413	403
query50	775	390	388	388
query51	6867	6817	6720	6720
query52	108	89	93	89
query53	365	298	287	287
query54	890	435	435	435
query55	72	72	75	72
query56	287	262	265	262
query57	1105	1050	1028	1028
query58	256	242	257	242
query59	3834	3334	3206	3206
query60	337	272	269	269
query61	106	90	90	90
query62	624	451	446	446
query63	321	290	291	290
query64	8860	2242	1752	1752
query65	3138	3091	3116	3091
query66	944	335	341	335
query67	15471	15105	14915	14915
query68	4578	529	520	520
query69	536	308	309	308
query70	1176	1169	1128	1128
query71	424	285	282	282
query72	8889	5527	5433	5433
query73	775	325	321	321
query74	5918	5561	5482	5482
query75	3651	2701	2688	2688
query76	2717	1033	943	943
query77	459	299	299	299
query78	9537	8930	8811	8811
query79	2101	521	512	512
query80	2366	469	465	465
query81	553	221	217	217
query82	856	108	108	108
query83	302	172	163	163
query84	267	87	83	83
query85	2041	311	281	281
query86	448	318	320	318
query87	3240	3125	3061	3061
query88	3747	2364	2378	2364
query89	477	379	377	377
query90	1940	195	189	189
query91	128	112	112	112
query92	60	56	53	53
query93	2372	501	506	501
query94	1264	218	298	218
query95	415	312	327	312
query96	603	269	264	264
query97	3213	3073	3015	3015
query98	225	207	194	194
query99	1255	844	840	840
Total cold run time: 275954 ms
Total hot run time: 172120 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cdb1d1de6c7f8b0e5334d8fe070aa53e6b34968a, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.07
query5	0.47	0.49	0.47
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.47	0.50
query10	0.54	0.56	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.13
query13	0.58	0.59	0.58
query14	0.77	0.78	0.78
query15	0.86	0.81	0.81
query16	0.34	0.35	0.36
query17	0.97	1.04	1.04
query18	0.25	0.24	0.24
query19	1.90	1.83	1.69
query20	0.01	0.02	0.01
query21	15.53	0.73	0.65
query22	4.37	7.54	1.77
query23	18.33	1.44	1.22
query24	2.18	0.23	0.21
query25	0.15	0.09	0.08
query26	0.31	0.21	0.21
query27	0.45	0.25	0.22
query28	13.37	1.03	1.02
query29	12.90	3.32	3.29
query30	0.26	0.07	0.05
query31	2.86	0.40	0.38
query32	3.27	0.46	0.47
query33	2.87	2.88	2.90
query34	17.11	4.34	4.36
query35	4.36	4.43	4.42
query36	0.65	0.49	0.48
query37	0.19	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.49 s
Total hot run time: 30.43 s

@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40043 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0bea7cfbac50f097848cae56c11af86be2f95eb, data reload: false

------ Round 1 ----------------------------------
q1	17598	4340	4315	4315
q2	2023	191	184	184
q3	10463	1186	1155	1155
q4	10197	799	810	799
q5	7490	2674	2742	2674
q6	225	141	137	137
q7	958	587	605	587
q8	9219	2066	2072	2066
q9	9057	6485	6491	6485
q10	8812	3673	3698	3673
q11	450	234	245	234
q12	406	249	244	244
q13	17767	2965	2997	2965
q14	264	214	220	214
q15	523	499	486	486
q16	508	378	369	369
q17	964	677	726	677
q18	8013	7443	7391	7391
q19	3274	1467	1479	1467
q20	690	335	335	335
q21	4880	3987	3239	3239
q22	402	351	347	347
Total cold run time: 114183 ms
Total hot run time: 40043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4305	4255	4255
q2	367	261	272	261
q3	3001	2744	2796	2744
q4	1938	1684	1711	1684
q5	5508	5581	5459	5459
q6	222	132	148	132
q7	2217	1825	1898	1825
q8	3305	3386	3401	3386
q9	8683	8610	8729	8610
q10	4078	3944	3814	3814
q11	583	474	488	474
q12	762	624	641	624
q13	16052	3203	3165	3165
q14	297	281	273	273
q15	541	480	514	480
q16	504	434	429	429
q17	1818	1529	1482	1482
q18	8133	7846	7833	7833
q19	1852	1675	1582	1582
q20	2161	1893	1955	1893
q21	5192	4967	4919	4919
q22	643	559	579	559
Total cold run time: 72245 ms
Total hot run time: 55883 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170725 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0bea7cfbac50f097848cae56c11af86be2f95eb, data reload: false

query1	928	389	386	386
query2	6492	2436	2448	2436
query3	6637	205	208	205
query4	21145	17332	17345	17332
query5	3638	492	486	486
query6	273	183	186	183
query7	4591	296	308	296
query8	331	302	293	293
query9	8635	2396	2392	2392
query10	561	295	281	281
query11	10467	10044	10163	10044
query12	122	88	85	85
query13	1654	377	364	364
query14	10289	7230	7143	7143
query15	239	188	194	188
query16	7604	326	309	309
query17	1364	535	550	535
query18	1937	281	277	277
query19	203	149	154	149
query20	93	85	83	83
query21	210	131	131	131
query22	4293	3993	4037	3993
query23	33983	33710	33797	33710
query24	11065	2927	2903	2903
query25	585	380	401	380
query26	737	159	164	159
query27	2325	333	332	332
query28	6063	2151	2148	2148
query29	888	648	638	638
query30	258	157	163	157
query31	1059	765	770	765
query32	128	53	55	53
query33	664	289	292	289
query34	881	488	480	480
query35	798	641	622	622
query36	1112	1013	997	997
query37	142	87	82	82
query38	2970	2792	2889	2792
query39	883	855	821	821
query40	213	136	132	132
query41	56	53	55	53
query42	110	105	104	104
query43	620	553	543	543
query44	1090	722	734	722
query45	189	168	167	167
query46	1073	735	733	733
query47	1861	1760	1746	1746
query48	367	309	305	305
query49	861	440	433	433
query50	771	384	388	384
query51	6875	6780	6796	6780
query52	104	98	98	98
query53	360	311	302	302
query54	900	454	453	453
query55	75	78	74	74
query56	313	289	292	289
query57	1114	1056	1039	1039
query58	278	267	271	267
query59	3588	3128	3096	3096
query60	328	337	287	287
query61	99	92	90	90
query62	610	443	436	436
query63	327	297	300	297
query64	8530	2233	1760	1760
query65	3145	3105	3092	3092
query66	748	331	377	331
query67	15393	15002	14722	14722
query68	4549	529	539	529
query69	468	314	314	314
query70	1186	1131	1159	1131
query71	375	295	283	283
query72	7580	5540	2764	2764
query73	734	320	320	320
query74	5936	5554	5471	5471
query75	3424	2633	2681	2633
query76	2186	991	871	871
query77	624	305	301	301
query78	9572	8942	8833	8833
query79	2294	524	529	524
query80	2409	469	486	469
query81	615	221	220	220
query82	804	103	105	103
query83	304	174	168	168
query84	262	90	84	84
query85	2001	291	277	277
query86	479	300	319	300
query87	3282	3079	3091	3079
query88	3921	2378	2376	2376
query89	494	396	397	396
query90	1768	188	190	188
query91	132	111	104	104
query92	63	51	50	50
query93	2408	505	496	496
query94	1145	217	210	210
query95	405	385	316	316
query96	594	267	267	267
query97	3136	2976	3046	2976
query98	234	198	197	197
query99	1123	842	821	821
Total cold run time: 270752 ms
Total hot run time: 170725 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0bea7cfbac50f097848cae56c11af86be2f95eb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.22	0.06	0.06
query4	1.66	0.09	0.10
query5	0.50	0.50	0.49
query6	1.15	0.72	0.73
query7	0.03	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.56	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.76	0.79
query15	0.84	0.80	0.81
query16	0.34	0.37	0.36
query17	0.94	1.01	0.99
query18	0.23	0.23	0.24
query19	1.77	1.68	1.64
query20	0.02	0.01	0.01
query21	15.40	0.75	0.66
query22	4.12	7.70	1.74
query23	18.30	1.38	1.23
query24	2.18	0.22	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.22
query27	0.45	0.23	0.23
query28	13.26	1.02	1.00
query29	12.62	3.37	3.34
query30	0.26	0.06	0.06
query31	2.87	0.39	0.40
query32	3.25	0.48	0.47
query33	2.81	2.92	2.87
query34	17.17	4.33	4.35
query35	4.43	4.39	4.38
query36	0.66	0.47	0.46
query37	0.18	0.15	0.16
query38	0.14	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.59 s
Total hot run time: 30.34 s

@dataroaring dataroaring merged commit c8d499d into apache:master Jul 5, 2024
26 of 29 checks passed
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jul 5, 2024
@dataroaring
[fix](partitionkey) serialize expr in PartitionKey via gson (apache#3…
732e37d 
…7277)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@w41ter w41ter w41ter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dataroaring @doris-robot @gavinchou @w41ter