[test](join) Fuzzy disable runtime filters in BE #45654

mrhhsg · 2024-12-19T09:57:47Z

What problem does this PR solve?

Add the functionality to disable the runtime filter, in preparation for the spill feature.

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2024-12-19T09:57:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mrhhsg · 2024-12-19T09:57:55Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/exprs/runtime_filter.cpp

yiguolei · 2024-12-19T10:52:45Z

be/src/pipeline/exec/hashjoin_build_sink.cpp

-        RETURN_IF_ERROR(local_state._runtime_filter_slots->send_filter_size(
-                state, local_state._shared_state->build_block->rows(),
-                local_state._finish_dependency));
+        if (!local_state._runtime_filters_disabled) {


那现在的逻辑是，如果disable了runtime filter，实际是用runtime filter 等待超时的方式解决吗？没有机制，告诉coordinator 一个0或者啥的？

doris-robot · 2024-12-19T11:56:02Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10134/26073)
Line Coverage: 29.78% (85171/286033)
Region Coverage: 28.82% (43693/151587)
Branch Coverage: 25.36% (22200/87526)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00b8bb59f074a096bd031ac4bdf1a20b4503ab58_00b8bb59f074a096bd031ac4bdf1a20b4503ab58/report/index.html

mrhhsg · 2024-12-20T09:59:31Z

run buildall

doris-robot · 2024-12-20T11:45:32Z

TPC-H: Total hot run time: 40091 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8ec9fd3bdb9ad74bcb79b483104feaabb3388b5, data reload: false

------ Round 1 ----------------------------------
q1	17598	7605	7332	7332
q2	2067	180	167	167
q3	10663	1100	1186	1100
q4	10452	752	804	752
q5	7628	2684	2742	2684
q6	245	146	144	144
q7	997	632	611	611
q8	9247	1927	1923	1923
q9	6604	6515	6400	6400
q10	7005	2305	2331	2305
q11	475	260	263	260
q12	449	226	228	226
q13	17778	2924	2937	2924
q14	249	215	209	209
q15	569	505	507	505
q16	674	599	602	599
q17	1011	566	487	487
q18	7273	6724	6648	6648
q19	1383	1047	1023	1023
q20	479	188	195	188
q21	4103	3293	3422	3293
q22	379	321	311	311
Total cold run time: 107328 ms
Total hot run time: 40091 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7320	7244	7203	7203
q2	333	234	230	230
q3	2901	2841	3041	2841
q4	2080	1817	1878	1817
q5	5682	5709	5642	5642
q6	228	143	143	143
q7	2316	1837	1806	1806
q8	3429	3570	3524	3524
q9	8904	8992	8971	8971
q10	3586	3527	3574	3527
q11	608	525	540	525
q12	853	622	604	604
q13	13002	3238	3115	3115
q14	300	278	283	278
q15	565	507	502	502
q16	690	659	663	659
q17	1874	1633	1620	1620
q18	8538	7713	7935	7713
q19	1732	1618	1414	1414
q20	2114	1842	1862	1842
q21	5688	5572	5499	5499
q22	663	555	622	555
Total cold run time: 73406 ms
Total hot run time: 60030 ms

doris-robot · 2024-12-20T11:57:53Z

TPC-DS: Total hot run time: 197089 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8ec9fd3bdb9ad74bcb79b483104feaabb3388b5, data reload: false

query1	1318	945	924	924
query2	6282	2371	2377	2371
query3	11022	4752	4744	4744
query4	33445	23589	23510	23510
query5	4761	471	465	465
query6	275	187	183	183
query7	3987	314	311	311
query8	305	240	255	240
query9	9716	2735	2718	2718
query10	490	248	234	234
query11	17991	15130	15242	15130
query12	161	107	101	101
query13	1570	424	418	418
query14	9772	7043	6848	6848
query15	260	194	193	193
query16	8262	444	463	444
query17	1481	606	630	606
query18	2137	318	327	318
query19	307	164	166	164
query20	125	120	117	117
query21	216	110	112	110
query22	4730	4656	4395	4395
query23	35193	34047	33990	33990
query24	10534	2512	2641	2512
query25	664	402	383	383
query26	1275	157	156	156
query27	2492	327	342	327
query28	7506	2465	2473	2465
query29	854	412	422	412
query30	223	162	157	157
query31	1038	840	840	840
query32	98	59	60	59
query33	764	309	301	301
query34	1156	545	568	545
query35	911	766	777	766
query36	1150	974	990	974
query37	147	73	73	73
query38	4367	4314	4474	4314
query39	1543	1467	1452	1452
query40	222	110	113	110
query41	47	47	45	45
query42	129	106	112	106
query43	549	529	501	501
query44	1310	840	847	840
query45	192	169	174	169
query46	1224	707	729	707
query47	2063	1926	1950	1926
query48	436	338	328	328
query49	942	415	373	373
query50	838	397	397	397
query51	7431	7209	7136	7136
query52	111	94	93	93
query53	267	183	195	183
query54	1096	411	415	411
query55	93	88	85	85
query56	279	236	233	233
query57	1335	1146	1163	1146
query58	254	222	219	219
query59	3566	3218	3287	3218
query60	276	251	244	244
query61	110	108	110	108
query62	897	676	691	676
query63	226	195	192	192
query64	4140	705	646	646
query65	3316	3271	3282	3271
query66	765	301	309	301
query67	16572	15544	15438	15438
query68	5585	550	555	550
query69	492	260	264	260
query70	1234	1173	1140	1140
query71	479	253	247	247
query72	6915	4160	4095	4095
query73	829	362	389	362
query74	10297	8987	8852	8852
query75	3639	2633	2629	2629
query76	3783	1061	1068	1061
query77	577	281	268	268
query78	10208	9407	9660	9407
query79	1464	609	587	587
query80	1060	535	426	426
query81	523	235	231	231
query82	630	119	123	119
query83	206	141	153	141
query84	289	71	70	70
query85	1344	316	302	302
query86	411	315	307	307
query87	4500	4301	4665	4301
query88	3452	2249	2227	2227
query89	442	334	285	285
query90	2048	192	191	191
query91	143	103	105	103
query92	65	50	50	50
query93	1602	552	543	543
query94	762	282	291	282
query95	355	263	254	254
query96	625	283	284	283
query97	2908	2712	2656	2656
query98	215	188	195	188
query99	1634	1304	1300	1300
Total cold run time: 305930 ms
Total hot run time: 197089 ms

doris-robot · 2024-12-20T11:58:09Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10130/26070)
Line Coverage: 29.79% (85246/286175)
Region Coverage: 28.91% (43534/150561)
Branch Coverage: 25.42% (22168/87198)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d8ec9fd3bdb9ad74bcb79b483104feaabb3388b5_d8ec9fd3bdb9ad74bcb79b483104feaabb3388b5/report/index.html

doris-robot · 2024-12-20T12:03:39Z

ClickBench: Total hot run time: 33.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d8ec9fd3bdb9ad74bcb79b483104feaabb3388b5, data reload: false

query1	0.03	0.03	0.04
query2	0.09	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.09	0.08
query5	0.41	0.40	0.40
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.49	0.50
query10	0.54	0.57	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.14
query13	0.61	0.59	0.60
query14	2.89	2.85	2.82
query15	0.89	0.84	0.84
query16	0.37	0.38	0.37
query17	1.06	1.07	1.06
query18	0.19	0.20	0.18
query19	1.91	1.88	2.09
query20	0.01	0.02	0.01
query21	15.35	0.64	0.65
query22	4.63	6.66	2.11
query23	18.23	1.36	1.44
query24	2.31	0.21	0.21
query25	0.16	0.08	0.08
query26	0.27	0.18	0.17
query27	0.07	0.08	0.08
query28	13.25	1.16	1.14
query29	12.65	3.42	3.39
query30	0.25	0.06	0.06
query31	2.83	0.41	0.40
query32	3.22	0.49	0.48
query33	3.11	3.18	3.13
query34	16.90	4.53	4.49
query35	4.58	4.48	4.46
query36	0.66	0.48	0.49
query37	0.19	0.16	0.16
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.14
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 112.26 s
Total hot run time: 33.65 s

BiteTheDDDDt · 2024-12-20T13:43:36Z

be/src/pipeline/exec/hashjoin_build_sink.cpp

+        return Status::OK();
+    }
+
+    DCHECK(_runtime_filter_slots) << "_runtime_filter_slots should be initialized";


why check a object?

github-actions · 2024-12-23T03:25:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-23T03:25:23Z

PR approved by anyone and no changes requested.

mrhhsg · 2024-12-24T07:12:28Z

run buildall

doris-robot · 2024-12-24T08:43:27Z

TPC-H: Total hot run time: 39988 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac680479658ecde4be168926ce79f09efed70d0d, data reload: false

------ Round 1 ----------------------------------
q1	17629	7474	7266	7266
q2	2048	188	186	186
q3	10517	1139	1157	1139
q4	10226	729	745	729
q5	7611	2700	2751	2700
q6	241	147	148	147
q7	993	625	616	616
q8	9248	1876	1935	1876
q9	6675	6484	6409	6409
q10	6981	2344	2374	2344
q11	451	265	260	260
q12	420	221	224	221
q13	17772	2994	2954	2954
q14	251	213	218	213
q15	548	515	483	483
q16	670	580	582	580
q17	991	614	534	534
q18	7383	6655	6616	6616
q19	1368	1023	1003	1003
q20	471	190	189	189
q21	4142	3217	3254	3217
q22	374	306	320	306
Total cold run time: 107010 ms
Total hot run time: 39988 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7257	7193	7254	7193
q2	331	234	230	230
q3	2914	2750	2789	2750
q4	1965	1769	1711	1711
q5	5400	5413	5430	5413
q6	225	140	139	139
q7	2123	1735	1718	1718
q8	3265	3423	3417	3417
q9	8600	8664	8579	8579
q10	3529	3452	3442	3442
q11	591	491	493	491
q12	761	575	576	575
q13	12188	2960	2934	2934
q14	298	265	276	265
q15	546	495	485	485
q16	676	644	644	644
q17	1801	1584	1567	1567
q18	7912	7534	7418	7418
q19	1692	1568	1561	1561
q20	2075	1828	1873	1828
q21	5419	5396	5293	5293
q22	635	593	592	592
Total cold run time: 70203 ms
Total hot run time: 58245 ms

doris-robot · 2024-12-24T08:55:36Z

TPC-DS: Total hot run time: 189821 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac680479658ecde4be168926ce79f09efed70d0d, data reload: false

query1	968	404	379	379
query2	6524	2421	2433	2421
query3	6704	214	205	205
query4	34075	23170	23477	23170
query5	4383	492	472	472
query6	290	205	180	180
query7	4625	306	306	306
query8	307	257	231	231
query9	9517	2763	2763	2763
query10	487	283	254	254
query11	18207	15388	15165	15165
query12	173	104	103	103
query13	1734	454	416	416
query14	12067	7001	6730	6730
query15	320	178	189	178
query16	8112	436	430	430
query17	1743	583	572	572
query18	2148	336	327	327
query19	366	153	159	153
query20	124	120	112	112
query21	206	105	108	105
query22	4608	4390	4137	4137
query23	35032	33373	32842	32842
query24	11250	2450	2490	2450
query25	672	388	390	388
query26	1826	152	152	152
query27	3024	336	333	333
query28	7831	2484	2475	2475
query29	1024	415	408	408
query30	304	150	150	150
query31	1085	834	831	831
query32	102	58	63	58
query33	787	306	286	286
query34	1124	518	509	509
query35	901	740	730	730
query36	1097	929	965	929
query37	282	78	74	74
query38	4391	4160	4189	4160
query39	1464	1412	1431	1412
query40	278	104	106	104
query41	50	45	45	45
query42	110	97	98	97
query43	537	506	512	506
query44	1267	820	805	805
query45	184	172	176	172
query46	1173	720	733	720
query47	1961	1852	1857	1852
query48	420	322	316	316
query49	1280	393	385	385
query50	847	383	379	379
query51	7208	7070	6999	6999
query52	100	92	93	92
query53	262	187	191	187
query54	1188	410	407	407
query55	83	81	81	81
query56	247	242	237	237
query57	1294	1095	1100	1095
query58	241	223	224	223
query59	3129	3012	3117	3012
query60	261	240	248	240
query61	111	111	111	111
query62	943	752	731	731
query63	214	196	197	196
query64	5137	687	643	643
query65	3227	3173	3293	3173
query66	1418	319	305	305
query67	15667	15527	15550	15527
query68	5864	554	543	543
query69	441	249	251	249
query70	1245	1153	1150	1150
query71	456	255	255	255
query72	6632	4211	4135	4135
query73	824	368	367	367
query74	10142	8939	9053	8939
query75	3682	2751	2657	2657
query76	3574	1199	1174	1174
query77	588	271	292	271
query78	10320	9434	9562	9434
query79	1806	604	593	593
query80	864	445	439	439
query81	541	230	225	225
query82	634	124	119	119
query83	267	151	147	147
query84	237	77	67	67
query85	1256	303	297	297
query86	438	306	301	301
query87	4542	4424	4459	4424
query88	4067	2243	2167	2167
query89	415	295	308	295
query90	2184	187	182	182
query91	139	111	105	105
query92	63	52	57	52
query93	1887	559	542	542
query94	1065	305	275	275
query95	350	247	244	244
query96	656	277	282	277
query97	2895	2638	2704	2638
query98	218	197	194	194
query99	1667	1485	1428	1428
Total cold run time: 308680 ms
Total hot run time: 189821 ms

doris-robot · 2024-12-24T09:01:08Z

ClickBench: Total hot run time: 33.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac680479658ecde4be168926ce79f09efed70d0d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.44	0.42	0.41
query6	1.14	0.64	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.53
query10	0.55	0.58	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.12
query13	0.60	0.62	0.59
query14	2.74	2.76	2.73
query15	0.89	0.83	0.82
query16	0.38	0.40	0.39
query17	1.05	1.05	1.10
query18	0.22	0.20	0.21
query19	1.88	1.96	1.87
query20	0.01	0.02	0.01
query21	15.36	0.59	0.58
query22	2.39	1.94	2.78
query23	16.88	0.84	0.85
query24	3.54	1.63	1.29
query25	0.28	0.21	0.06
query26	0.39	0.14	0.13
query27	0.04	0.04	0.03
query28	9.34	1.10	1.08
query29	12.56	3.21	3.19
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.24	0.47	0.46
query33	3.08	3.15	3.16
query34	17.06	4.44	4.44
query35	4.48	4.48	4.41
query36	0.66	0.48	0.50
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.11	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.67 s
Total hot run time: 33.08 s

doris-robot · 2024-12-24T09:06:25Z

TeamCity be ut coverage result:
Function Coverage: 38.79% (10094/26022)
Line Coverage: 29.77% (85119/285925)
Region Coverage: 28.89% (43463/150469)
Branch Coverage: 25.42% (22155/87164)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac680479658ecde4be168926ce79f09efed70d0d_ac680479658ecde4be168926ce79f09efed70d0d/report/index.html

github-actions · 2024-12-30T11:41:47Z

PR approved by at least one committer and no changes requested.

Add the functionality to disable the runtime filter, in preparation for the spill feature.

pick from #45654 --------- Co-authored-by: Jerry Hu <hushenggang@selectdb.com>

Add the functionality to disable the runtime filter, in preparation for the spill feature.

pick from #45654 Co-authored-by: Jerry Hu <hushenggang@selectdb.com>

github-actions bot reviewed Dec 19, 2024

View reviewed changes

be/src/exprs/runtime_filter.cpp Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

[test](join) Fuzzy disable runtime filters in BE

d8ec9fd

mrhhsg force-pushed the fuzzy_disable_rf branch from 00b8bb5 to d8ec9fd Compare December 20, 2024 09:56

BiteTheDDDDt reviewed Dec 20, 2024

View reviewed changes

BiteTheDDDDt previously approved these changes Dec 23, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024

github-actions bot added the reviewed label Dec 23, 2024

Merge branch 'master' into fuzzy_disable_rf

ac68047

mrhhsg dismissed BiteTheDDDDt’s stale review via ac68047 December 24, 2024 07:11

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 24, 2024

BiteTheDDDDt approved these changes Dec 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 30, 2024

yiguolei approved these changes Dec 30, 2024

View reviewed changes

yiguolei merged commit 004e8f2 into apache:master Dec 30, 2024
24 of 26 checks passed

BiteTheDDDDt added dev/2.1.x dev/3.0.x labels Jan 13, 2025

github-actions bot added dev/2.1.x-conflict dev/3.0.x-conflict labels Jan 13, 2025

BiteTheDDDDt removed the dev/2.1.x-conflict label Jan 13, 2025

BiteTheDDDDt pushed a commit that referenced this pull request Jan 13, 2025

[test](join) Fuzzy disable runtime filters in BE (#45654)

0892c7e

Add the functionality to disable the runtime filter, in preparation for the spill feature.

BiteTheDDDDt mentioned this pull request Jan 13, 2025

[test](join) Fuzzy disable runtime filters in BE #45654 #46931

Merged

BiteTheDDDDt added a commit that referenced this pull request Jan 14, 2025

[test](join) Fuzzy disable runtime filters in BE #45654 (#46931)

989f189

pick from #45654 --------- Co-authored-by: Jerry Hu <hushenggang@selectdb.com>

BiteTheDDDDt added dev/2.1.8-merged and removed dev/2.1.x labels Jan 14, 2025

BiteTheDDDDt pushed a commit that referenced this pull request Jan 14, 2025

[test](join) Fuzzy disable runtime filters in BE (#45654)

aa42b67

Add the functionality to disable the runtime filter, in preparation for the spill feature.

BiteTheDDDDt mentioned this pull request Jan 14, 2025

[test](join) Fuzzy disable runtime filters in BE (#45654) #46963

Merged

BiteTheDDDDt added a commit that referenced this pull request Jan 14, 2025

[test](join) Fuzzy disable runtime filters in BE (#45654) (#46963)

e56c4f0

pick from #45654 Co-authored-by: Jerry Hu <hushenggang@selectdb.com>

wm1581066 added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test](join) Fuzzy disable runtime filters in BE #45654

[test](join) Fuzzy disable runtime filters in BE #45654

mrhhsg commented Dec 19, 2024 •

edited

Loading

hello-stephen commented Dec 19, 2024

mrhhsg commented Dec 19, 2024

github-actions bot left a comment

yiguolei Dec 19, 2024

doris-robot commented Dec 19, 2024

mrhhsg commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

BiteTheDDDDt Dec 20, 2024

github-actions bot commented Dec 23, 2024

github-actions bot commented Dec 23, 2024

mrhhsg commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

github-actions bot commented Dec 30, 2024

[test](join) Fuzzy disable runtime filters in BE #45654

[test](join) Fuzzy disable runtime filters in BE #45654

Conversation

mrhhsg commented Dec 19, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Dec 19, 2024

mrhhsg commented Dec 19, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

yiguolei Dec 19, 2024

Choose a reason for hiding this comment

doris-robot commented Dec 19, 2024

mrhhsg commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

BiteTheDDDDt Dec 20, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 23, 2024

github-actions bot commented Dec 23, 2024

mrhhsg commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

github-actions bot commented Dec 30, 2024

mrhhsg commented Dec 19, 2024 •

edited

Loading