Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](build index) Forbid building indexes for ngram BF indexes #47041

Merged
merged 6 commits into from
Jan 17, 2025

Conversation

airborne12
Copy link
Member

@airborne12 airborne12 commented Jan 15, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
DO NOT support build index for ngram bf index

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32118 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42de8ed953d19b95f764c88bfa4bf485c02807e0, data reload: false

------ Round 1 ----------------------------------
q1	17567	5452	5340	5340
q2	2048	311	165	165
q3	10416	1227	741	741
q4	10196	960	524	524
q5	7522	2339	2144	2144
q6	192	170	133	133
q7	926	782	598	598
q8	9236	1315	1175	1175
q9	5203	4864	4842	4842
q10	6841	2341	1885	1885
q11	474	270	260	260
q12	335	356	218	218
q13	17786	3698	3077	3077
q14	233	227	208	208
q15	528	488	464	464
q16	632	618	574	574
q17	551	853	312	312
q18	6987	6424	6493	6424
q19	1397	958	532	532
q20	311	323	191	191
q21	2836	2225	1999	1999
q22	362	347	312	312
Total cold run time: 102579 ms
Total hot run time: 32118 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5482	5476	5438	5438
q2	238	335	236	236
q3	2298	2659	2293	2293
q4	1465	1847	1394	1394
q5	4344	4746	4662	4662
q6	171	157	124	124
q7	1965	1963	1859	1859
q8	2605	2837	2670	2670
q9	7317	7284	7277	7277
q10	3068	3276	2795	2795
q11	584	511	488	488
q12	637	760	618	618
q13	3480	3911	3293	3293
q14	280	308	272	272
q15	512	485	475	475
q16	676	692	672	672
q17	1249	1720	1280	1280
q18	7655	7318	7419	7318
q19	765	1018	1130	1018
q20	1998	2144	1918	1918
q21	5720	5473	4941	4941
q22	616	629	583	583
Total cold run time: 53125 ms
Total hot run time: 51624 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194099 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42de8ed953d19b95f764c88bfa4bf485c02807e0, data reload: false

query1	1315	961	909	909
query2	6305	2069	2060	2060
query3	10962	4602	4559	4559
query4	60687	30849	23002	23002
query5	5461	601	458	458
query6	419	186	170	170
query7	5537	484	296	296
query8	314	236	230	230
query9	8087	2672	2669	2669
query10	440	287	246	246
query11	17570	15562	15527	15527
query12	163	113	124	113
query13	1408	549	424	424
query14	11491	6881	6830	6830
query15	214	221	195	195
query16	7036	639	468	468
query17	1127	777	559	559
query18	1558	420	302	302
query19	198	183	164	164
query20	120	112	113	112
query21	210	119	107	107
query22	4490	4879	4328	4328
query23	33778	33134	33208	33134
query24	5620	2335	2352	2335
query25	474	459	404	404
query26	633	290	155	155
query27	1537	466	329	329
query28	4141	2492	2458	2458
query29	528	573	427	427
query30	227	190	155	155
query31	932	894	829	829
query32	69	63	59	59
query33	471	372	320	320
query34	733	851	508	508
query35	846	858	749	749
query36	1019	1044	965	965
query37	126	100	82	82
query38	4293	4290	4293	4290
query39	1496	1456	1465	1456
query40	217	124	109	109
query41	56	54	56	54
query42	122	103	141	103
query43	524	542	502	502
query44	1301	824	845	824
query45	179	180	172	172
query46	872	1074	644	644
query47	1945	1964	1827	1827
query48	389	394	321	321
query49	690	484	411	411
query50	665	666	403	403
query51	7015	7195	6971	6971
query52	130	108	91	91
query53	233	256	186	186
query54	471	501	421	421
query55	82	81	88	81
query56	273	263	254	254
query57	1260	1211	1117	1117
query58	249	230	231	230
query59	3188	3138	3120	3120
query60	295	278	247	247
query61	115	115	112	112
query62	748	705	666	666
query63	218	188	190	188
query64	1141	1004	652	652
query65	3280	3187	3185	3185
query66	724	384	295	295
query67	16210	15824	15473	15473
query68	5722	814	533	533
query69	525	351	268	268
query70	1212	1131	1115	1115
query71	414	284	264	264
query72	6361	3908	3845	3845
query73	829	755	360	360
query74	10064	8919	8728	8728
query75	3236	3175	2646	2646
query76	3836	1270	767	767
query77	470	375	268	268
query78	9973	9886	9235	9235
query79	3562	804	585	585
query80	1217	518	441	441
query81	515	280	241	241
query82	1062	157	120	120
query83	245	178	159	159
query84	294	98	70	70
query85	754	348	299	299
query86	415	316	271	271
query87	4421	4432	4482	4432
query88	4539	2167	2178	2167
query89	411	332	286	286
query90	1528	192	189	189
query91	132	138	105	105
query92	71	55	53	53
query93	2980	861	531	531
query94	851	397	297	297
query95	329	258	251	251
query96	478	626	290	290
query97	2838	2854	2795	2795
query98	235	201	194	194
query99	1295	1388	1252	1252
Total cold run time: 315441 ms
Total hot run time: 194099 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42de8ed953d19b95f764c88bfa4bf485c02807e0, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.45	0.40	0.41
query6	1.18	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.55	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.60
query14	2.76	2.86	2.85
query15	0.90	0.84	0.82
query16	0.45	0.38	0.38
query17	1.03	1.08	0.99
query18	0.23	0.21	0.21
query19	1.88	1.81	1.98
query20	0.01	0.02	0.01
query21	15.36	0.93	0.60
query22	0.75	0.83	0.61
query23	15.33	1.41	0.61
query24	3.02	1.87	1.96
query25	0.15	0.22	0.14
query26	0.22	0.14	0.13
query27	0.04	0.04	0.05
query28	14.63	0.96	0.42
query29	12.58	3.94	3.28
query30	0.25	0.09	0.07
query31	2.81	0.59	0.40
query32	3.23	0.55	0.46
query33	3.02	3.01	3.06
query34	16.59	5.23	4.59
query35	4.49	4.53	4.56
query36	0.64	0.50	0.51
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.5 s
Total hot run time: 31.82 s

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 16, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32133 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 591763231de93a492b048e5001b5a59ba9fa5af0, data reload: false

------ Round 1 ----------------------------------
q1	17664	5578	5378	5378
q2	2057	306	164	164
q3	10531	1237	742	742
q4	10240	990	533	533
q5	8029	2391	2155	2155
q6	196	167	137	137
q7	916	769	596	596
q8	9228	1371	1132	1132
q9	5164	4927	4982	4927
q10	6842	2345	1910	1910
q11	465	270	255	255
q12	338	352	212	212
q13	17787	3741	3059	3059
q14	230	229	209	209
q15	505	479	456	456
q16	647	612	569	569
q17	568	859	331	331
q18	7083	6349	6337	6337
q19	2069	963	558	558
q20	307	315	188	188
q21	2813	2165	1977	1977
q22	367	338	308	308
Total cold run time: 104046 ms
Total hot run time: 32133 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5546	5494	5486	5486
q2	243	335	239	239
q3	2270	2684	2336	2336
q4	1399	1882	1367	1367
q5	4324	4738	4630	4630
q6	167	160	125	125
q7	2064	1963	1884	1884
q8	2621	2849	2687	2687
q9	7336	7222	7364	7222
q10	3006	3268	2795	2795
q11	561	496	481	481
q12	641	728	578	578
q13	3583	3975	3327	3327
q14	292	314	267	267
q15	505	486	458	458
q16	661	688	652	652
q17	1258	1753	1249	1249
q18	7677	7623	7280	7280
q19	837	1167	1055	1055
q20	2061	2048	1939	1939
q21	5640	5253	4922	4922
q22	605	589	564	564
Total cold run time: 53297 ms
Total hot run time: 51543 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193851 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 591763231de93a492b048e5001b5a59ba9fa5af0, data reload: false

query1	1309	976	933	933
query2	6311	2048	1949	1949
query3	10999	4431	4325	4325
query4	60743	28544	23069	23069
query5	5696	591	446	446
query6	430	191	183	183
query7	5589	500	305	305
query8	338	246	230	230
query9	8780	2690	2643	2643
query10	481	295	253	253
query11	17375	15171	15527	15171
query12	164	112	108	108
query13	1502	552	438	438
query14	11134	7246	6957	6957
query15	244	191	188	188
query16	7243	568	457	457
query17	1127	703	556	556
query18	1286	401	315	315
query19	187	182	153	153
query20	119	110	113	110
query21	220	125	106	106
query22	4636	4553	4547	4547
query23	34421	33436	33405	33405
query24	5562	2381	2434	2381
query25	475	474	400	400
query26	642	276	160	160
query27	1802	479	326	326
query28	4210	2490	2458	2458
query29	518	547	412	412
query30	212	194	152	152
query31	939	882	853	853
query32	66	58	56	56
query33	452	363	291	291
query34	736	860	520	520
query35	805	838	762	762
query36	1007	1045	962	962
query37	115	97	72	72
query38	4386	4273	4331	4273
query39	1501	1431	1442	1431
query40	198	117	106	106
query41	53	50	48	48
query42	123	107	102	102
query43	523	515	486	486
query44	1334	871	822	822
query45	185	176	169	169
query46	852	1070	656	656
query47	1928	1984	1896	1896
query48	383	408	331	331
query49	710	501	388	388
query50	650	676	389	389
query51	7062	7027	7036	7027
query52	104	99	95	95
query53	226	261	184	184
query54	479	498	424	424
query55	82	94	80	80
query56	264	303	256	256
query57	1220	1217	1125	1125
query58	241	227	233	227
query59	3173	3064	2792	2792
query60	271	265	251	251
query61	118	114	112	112
query62	696	722	668	668
query63	214	189	178	178
query64	1294	1017	653	653
query65	3259	3231	3163	3163
query66	654	386	300	300
query67	16148	15654	15436	15436
query68	4986	812	525	525
query69	481	297	267	267
query70	1135	1181	1176	1176
query71	420	281	258	258
query72	5931	3813	3840	3813
query73	811	772	352	352
query74	9848	9095	8765	8765
query75	3251	3135	2668	2668
query76	3803	1167	794	794
query77	469	364	277	277
query78	9914	9877	9340	9340
query79	3479	820	581	581
query80	1693	549	433	433
query81	550	286	234	234
query82	634	158	121	121
query83	273	165	162	162
query84	292	162	78	78
query85	795	352	287	287
query86	466	316	290	290
query87	4420	4520	4373	4373
query88	4573	2167	2145	2145
query89	392	331	284	284
query90	1556	189	202	189
query91	135	134	109	109
query92	63	58	51	51
query93	2911	879	525	525
query94	857	406	303	303
query95	324	264	266	264
query96	490	622	281	281
query97	2801	2866	2721	2721
query98	231	203	200	200
query99	1263	1395	1265	1265
Total cold run time: 315453 ms
Total hot run time: 193851 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 591763231de93a492b048e5001b5a59ba9fa5af0, data reload: false

query1	0.03	0.03	0.05
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.42	0.41
query6	1.16	0.65	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.51	0.49
query10	0.57	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.72	2.82	2.79
query15	0.90	0.83	0.83
query16	0.40	0.39	0.39
query17	1.03	1.07	1.06
query18	0.23	0.21	0.21
query19	1.92	1.81	2.00
query20	0.01	0.01	0.01
query21	15.41	0.93	0.59
query22	0.76	0.82	0.69
query23	15.16	1.43	0.57
query24	3.74	1.26	0.77
query25	0.27	0.17	0.20
query26	0.17	0.14	0.15
query27	0.05	0.05	0.06
query28	13.89	0.99	0.44
query29	12.56	4.00	3.28
query30	0.25	0.09	0.06
query31	2.81	0.60	0.39
query32	3.23	0.56	0.46
query33	3.03	3.00	3.02
query34	16.46	5.15	4.49
query35	4.56	4.44	4.49
query36	0.66	0.49	0.47
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.37 s
Total hot run time: 30.6 s

Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 5aac900 into apache:master Jan 17, 2025
27 of 28 checks passed
@airborne12 airborne12 deleted the fix1 branch January 17, 2025 04:06
github-actions bot pushed a commit that referenced this pull request Jan 17, 2025
Problem Summary:
DO NOT support build index for ngram bf index
airborne12 added a commit that referenced this pull request Jan 24, 2025
Problem Summary:
DO NOT support build index for ngram bf index
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
…he#47041)

Problem Summary:
DO NOT support build index for ngram bf index
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants