Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix] (inverted index) remove tmp columns in block #39369

Merged
merged 1 commit into from
Aug 15, 2024
Merged

[fix] (inverted index) remove tmp columns in block #39369

merged 1 commit into from
Aug 15, 2024

Conversation

csun5285
Copy link
Contributor

@csun5285 csun5285 commented Aug 14, 2024 •
edited
Loading

Proposed changes

remove tmp columns in block

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 14, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 37736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a25e1ac22d2a28cc1ab57d444f469642204f9696, data reload: false

------ Round 1 ----------------------------------
q1	18541	4450	4395	4395
q2	2483	184	175	175
q3	11143	1152	1117	1117
q4	10513	784	672	672
q5	8112	2810	2764	2764
q6	223	136	138	136
q7	975	598	601	598
q8	10150	2030	2064	2030
q9	7428	6526	6500	6500
q10	7058	2200	2187	2187
q11	461	240	233	233
q12	387	217	212	212
q13	17759	3039	3004	3004
q14	282	232	227	227
q15	519	479	489	479
q16	515	386	384	384
q17	975	686	673	673
q18	7299	6768	6752	6752
q19	4294	1015	1057	1015
q20	670	318	330	318
q21	4089	2853	2908	2853
q22	1113	1016	1012	1012
Total cold run time: 114989 ms
Total hot run time: 37736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4565	4275	4279	4275
q2	385	278	271	271
q3	2877	2622	2570	2570
q4	1882	1666	1601	1601
q5	5368	5384	5392	5384
q6	225	130	129	129
q7	2046	1690	1645	1645
q8	3144	3332	3361	3332
q9	8365	8332	8319	8319
q10	3387	3185	3134	3134
q11	589	510	502	502
q12	769	567	608	567
q13	16371	2980	2999	2980
q14	303	285	280	280
q15	533	473	501	473
q16	469	415	419	415
q17	1764	1500	1453	1453
q18	7566	7579	7383	7383
q19	1652	1536	1568	1536
q20	2009	1790	1820	1790
q21	5193	5233	5152	5152
q22	1130	1026	1004	1004
Total cold run time: 70592 ms
Total hot run time: 54195 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183763 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a25e1ac22d2a28cc1ab57d444f469642204f9696, data reload: false

query1	907	377	367	367
query2	6448	1905	1821	1821
query3	6637	210	216	210
query4	28056	23205	23020	23020
query5	4204	487	481	481
query6	274	186	166	166
query7	4601	293	305	293
query8	245	198	201	198
query9	8807	2460	2439	2439
query10	427	267	264	264
query11	17788	14946	15031	14946
query12	146	100	103	100
query13	1628	352	350	350
query14	9832	6624	6467	6467
query15	234	165	166	165
query16	7393	486	495	486
query17	1144	578	551	551
query18	1962	276	281	276
query19	186	144	137	137
query20	113	108	111	108
query21	206	97	99	97
query22	4168	3971	3988	3971
query23	34011	33177	33219	33177
query24	12523	2855	2794	2794
query25	671	376	398	376
query26	1785	156	156	156
query27	2975	269	272	269
query28	7844	2070	2040	2040
query29	1208	398	395	395
query30	301	148	151	148
query31	952	756	725	725
query32	105	52	55	52
query33	752	282	274	274
query34	998	461	466	461
query35	832	713	705	705
query36	1099	941	958	941
query37	278	81	90	81
query38	3912	3898	3778	3778
query39	1445	1401	1393	1393
query40	270	120	111	111
query41	45	44	44	44
query42	115	94	100	94
query43	501	467	452	452
query44	1189	731	724	724
query45	192	163	168	163
query46	1133	730	730	730
query47	1837	1743	1771	1743
query48	356	291	295	291
query49	1195	422	467	422
query50	792	403	400	400
query51	6778	6657	6705	6657
query52	97	98	87	87
query53	258	183	180	180
query54	1021	448	437	437
query55	76	73	75	73
query56	262	243	264	243
query57	1163	1036	1026	1026
query58	249	218	233	218
query59	2928	2674	2516	2516
query60	284	274	271	271
query61	98	111	94	94
query62	825	629	640	629
query63	213	183	189	183
query64	6378	2267	1700	1700
query65	3230	3158	3127	3127
query66	1379	331	334	331
query67	15410	14854	14808	14808
query68	4592	530	540	530
query69	386	276	274	274
query70	1186	1107	1123	1107
query71	396	270	269	269
query72	6422	2280	2049	2049
query73	766	319	317	317
query74	9238	8787	8882	8787
query75	3431	2684	2694	2684
query76	2401	1007	930	930
query77	485	310	307	307
query78	9532	9219	8932	8932
query79	2095	530	523	523
query80	1622	483	487	483
query81	594	227	222	222
query82	763	134	128	128
query83	260	152	142	142
query84	258	79	76	76
query85	2122	274	265	265
query86	508	301	312	301
query87	4321	4216	4235	4216
query88	4316	2307	2372	2307
query89	393	292	289	289
query90	1868	194	187	187
query91	116	92	93	92
query92	63	53	50	50
query93	2340	535	531	531
query94	989	263	285	263
query95	366	252	252	252
query96	594	277	278	277
query97	3192	3104	3069	3069
query98	212	208	195	195
query99	1589	1284	1277	1277
Total cold run time: 292787 ms
Total hot run time: 183763 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a25e1ac22d2a28cc1ab57d444f469642204f9696, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.50	0.48
query10	0.55	0.53	0.53
query11	0.15	0.12	0.11
query12	0.16	0.13	0.12
query13	0.59	0.60	0.59
query14	0.77	0.79	0.78
query15	0.88	0.82	0.82
query16	0.36	0.38	0.37
query17	0.99	1.03	1.02
query18	0.22	0.22	0.21
query19	1.81	1.80	1.79
query20	0.01	0.02	0.01
query21	15.38	0.76	0.66
query22	4.26	7.33	2.15
query23	18.29	1.35	1.24
query24	2.08	0.25	0.22
query25	0.16	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.25	1.03	1.00
query29	12.63	3.34	3.29
query30	0.24	0.05	0.05
query31	2.89	0.41	0.39
query32	3.27	0.50	0.48
query33	2.98	2.95	2.95
query34	16.93	4.39	4.39
query35	4.45	4.42	4.41
query36	0.67	0.48	0.48
query37	0.20	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.75 s
Total hot run time: 31.12 s

@csun5285
fix bug
6ab59be 
reset

fix
@csun5285
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ab59bee08b33451efac7d0314e36e4b7313ea7d, data reload: false

------ Round 1 ----------------------------------
q1	18385	4576	4437	4437
q2	2899	226	238	226
q3	10981	1273	1095	1095
q4	10644	723	790	723
q5	8326	2964	2921	2921
q6	277	158	162	158
q7	1017	664	654	654
q8	9414	2116	2118	2116
q9	7266	6592	6576	6576
q10	7088	2220	2235	2220
q11	495	279	280	279
q12	429	266	259	259
q13	17806	3008	2992	2992
q14	291	268	249	249
q15	569	516	535	516
q16	533	433	413	413
q17	1006	727	661	661
q18	7664	6921	6769	6769
q19	6156	1149	1088	1088
q20	743	358	372	358
q21	3946	2918	3055	2918
q22	1166	1032	1063	1032
Total cold run time: 117101 ms
Total hot run time: 38660 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4636	4301	4359	4301
q2	419	317	307	307
q3	2876	2650	2742	2650
q4	1956	1694	1647	1647
q5	5391	5400	5426	5400
q6	231	144	156	144
q7	2083	1736	1736	1736
q8	3218	3350	3343	3343
q9	8491	8416	8424	8416
q10	3429	3218	3177	3177
q11	597	509	510	509
q12	796	631	608	608
q13	16361	2992	3008	2992
q14	334	300	308	300
q15	565	506	514	506
q16	506	440	439	439
q17	1785	1520	1504	1504
q18	7775	7796	7546	7546
q19	1725	1609	1643	1609
q20	2089	1848	1803	1803
q21	11855	5029	5149	5029
q22	1102	1047	1036	1036
Total cold run time: 78220 ms
Total hot run time: 55002 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190253 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ab59bee08b33451efac7d0314e36e4b7313ea7d, data reload: false

query1	988	408	390	390
query2	6729	1985	1955	1955
query3	6654	236	247	236
query4	30691	23573	23451	23451
query5	4389	650	638	638
query6	299	190	200	190
query7	4616	317	314	314
query8	413	365	355	355
query9	8673	2501	2481	2481
query10	479	317	308	308
query11	17803	15003	15113	15003
query12	170	123	140	123
query13	1698	447	411	411
query14	9644	6930	6775	6775
query15	234	191	182	182
query16	7688	513	466	466
query17	1612	629	588	588
query18	1989	336	334	334
query19	206	169	164	164
query20	178	134	137	134
query21	250	143	136	136
query22	4463	4038	3912	3912
query23	34042	33284	33405	33284
query24	11743	2854	2961	2854
query25	688	421	409	409
query26	1262	179	180	179
query27	2826	300	302	300
query28	7447	2141	2125	2125
query29	934	440	439	439
query30	346	189	180	180
query31	1083	828	848	828
query32	118	79	82	79
query33	849	351	346	346
query34	947	525	541	525
query35	906	784	754	754
query36	1135	959	984	959
query37	186	104	102	102
query38	3961	3841	3980	3841
query39	1531	1469	1460	1460
query40	237	158	158	158
query41	139	138	138	138
query42	140	119	119	119
query43	539	495	505	495
query44	1250	779	789	779
query45	221	192	201	192
query46	1146	802	791	791
query47	1895	1837	1826	1826
query48	404	338	335	335
query49	1254	581	603	581
query50	860	470	479	470
query51	6854	6791	6772	6772
query52	118	109	112	109
query53	304	229	233	229
query54	962	517	505	505
query55	89	89	95	89
query56	335	313	313	313
query57	1200	1123	1120	1120
query58	319	292	323	292
query59	3032	2774	2818	2774
query60	358	327	332	327
query61	147	144	143	143
query62	870	711	684	684
query63	264	229	238	229
query64	5323	2422	1877	1877
query65	3234	3182	3177	3177
query66	1153	684	675	675
query67	15579	14975	14959	14959
query68	6499	588	584	584
query69	745	444	343	343
query70	1242	1101	1199	1101
query71	584	312	316	312
query72	7823	2317	2097	2097
query73	880	367	363	363
query74	9283	8826	8982	8826
query75	4844	2754	2774	2754
query76	4934	1081	1045	1045
query77	925	449	445	445
query78	9850	9260	8990	8990
query79	3542	557	547	547
query80	2226	597	605	597
query81	642	262	264	262
query82	1265	155	157	155
query83	339	222	216	216
query84	289	101	100	100
query85	1564	371	348	348
query86	467	335	326	326
query87	4404	4239	4193	4193
query88	4560	2521	2495	2495
query89	449	333	321	321
query90	1977	231	227	227
query91	151	126	124	124
query92	84	74	76	74
query93	4809	548	542	542
query94	922	322	337	322
query95	387	288	293	288
query96	625	284	288	284
query97	3277	3160	3100	3100
query98	253	241	223	223
query99	1566	1281	1338	1281
Total cold run time: 309503 ms
Total hot run time: 190253 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ab59bee08b33451efac7d0314e36e4b7313ea7d, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.66	0.08	0.08
query5	0.51	0.48	0.51
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.55	0.51	0.49
query10	0.55	0.56	0.55
query11	0.17	0.13	0.13
query12	0.16	0.14	0.13
query13	0.61	0.62	0.59
query14	0.78	0.78	0.78
query15	0.86	0.84	0.83
query16	0.38	0.35	0.36
query17	0.99	1.01	1.00
query18	0.24	0.22	0.22
query19	1.86	1.76	1.85
query20	0.02	0.02	0.01
query21	15.39	0.86	0.68
query22	4.00	7.06	2.27
query23	18.30	1.37	1.26
query24	2.16	0.22	0.22
query25	0.15	0.09	0.09
query26	0.31	0.23	0.22
query27	0.46	0.23	0.22
query28	13.26	1.03	1.02
query29	12.66	3.33	3.35
query30	0.44	0.25	0.24
query31	2.85	0.42	0.40
query32	3.24	0.49	0.50
query33	2.99	2.98	3.02
query34	17.12	4.46	4.38
query35	4.42	4.44	4.46
query36	0.67	0.50	0.50
query37	0.21	0.18	0.17
query38	0.17	0.17	0.17
query39	0.06	0.06	0.06
query40	0.18	0.14	0.15
query41	0.11	0.07	0.07
query42	0.08	0.07	0.07
query43	0.07	0.07	0.07
Total cold run time: 110.21 s
Total hot run time: 31.8 s

airborne12
airborne12 approved these changes Aug 15, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

xiaokang
xiaokang approved these changes Aug 15, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 4f16bf6 into apache:master Aug 15, 2024
26 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@csun5285 @dataroaring
[fix] (inverted index) remove tmp columns in block (#39369)
fa89b74
@csun5285 csun5285 mentioned this pull request Aug 18, 2024
Merged
xiaokang pushed a commit that referenced this pull request Aug 20, 2024
@csun5285
[fix] (inverted index) remove tmp columns in block (#39369) (#39533)
12ed295
@csun5285 csun5285 mentioned this pull request Aug 20, 2024
Merged
airborne12 pushed a commit that referenced this pull request Aug 21, 2024
@csun5285
[fix] (inverted index) remove tmp columns in block (#39369) (#39655)
4ac11eb 
## Proposed changes

pick from master #39369
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@csun5285 @Baymine
[fix] (inverted index) remove tmp columns in block (apache#39369) (ap…
c754efa 
…ache#39655)

## Proposed changes

pick from master apache#39369
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged p0_w reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@csun5285 @doris-robot @xiaokang @airborne12 @qidaye @dataroaring