[opt](mtmv) Support null aware left anti join rewrite by materialized view #45815

seawinde · 2024-12-23T10:39:16Z

What problem does this PR solve?

Support null aware left anti join rewrite by materialized view

such as mv def is as following, it's join type is NULL_AWARE_LEFT_ANTI_JOIN

            select  lineitem.L_LINENUMBER
            from lineitem
            where L_ORDERKEY not in (
            select o_custkey from orders_nullable
            );

if query is as following, should be rewritten successlly, this pr should this

            select  lineitem.L_LINENUMBER
            from lineitem
            where L_ORDERKEY not in (
            select o_custkey from orders_nullable
            ) and L_LINENUMBER = 4;

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Support null aware left anti join rewrite by materialized view

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-23T10:39:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2024-12-23T10:39:26Z

run buildall

doris-robot · 2024-12-23T10:59:02Z

TPC-H: Total hot run time: 40400 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d8cb608ec159203e7eb095da8eb869792498f31, data reload: false

------ Round 1 ----------------------------------
q1	17772	7603	7398	7398
q2	2067	202	187	187
q3	10584	1162	1222	1162
q4	10267	821	776	776
q5	7628	2751	2709	2709
q6	239	149	149	149
q7	989	627	619	619
q8	9255	1927	1937	1927
q9	6639	6466	6503	6466
q10	6999	2311	2321	2311
q11	465	281	267	267
q12	439	236	231	231
q13	17755	2926	2948	2926
q14	247	219	208	208
q15	554	501	498	498
q16	671	591	595	591
q17	998	576	561	561
q18	7707	6784	6671	6671
q19	1363	1042	962	962
q20	475	190	189	189
q21	4051	3464	3280	3280
q22	392	312	323	312
Total cold run time: 107556 ms
Total hot run time: 40400 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7349	7305	7266	7266
q2	324	235	235	235
q3	2948	2817	2818	2817
q4	2002	1761	1672	1672
q5	5364	5440	5447	5440
q6	216	139	141	139
q7	2134	1687	1711	1687
q8	3274	3421	3450	3421
q9	8625	8558	8599	8558
q10	3496	3467	3445	3445
q11	616	501	498	498
q12	782	584	569	569
q13	10637	2992	2980	2980
q14	292	258	261	258
q15	563	498	493	493
q16	664	640	646	640
q17	1815	1594	1568	1568
q18	8122	7474	7456	7456
q19	1711	1485	1364	1364
q20	2058	1850	1783	1783
q21	5531	5379	5267	5267
q22	665	584	592	584
Total cold run time: 69188 ms
Total hot run time: 58140 ms

doris-robot · 2024-12-23T11:11:11Z

TPC-DS: Total hot run time: 191226 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d8cb608ec159203e7eb095da8eb869792498f31, data reload: false

query1	975	392	375	375
query2	6519	2382	2436	2382
query3	6737	211	205	205
query4	33998	23539	23481	23481
query5	4320	485	469	469
query6	273	194	193	193
query7	4625	302	305	302
query8	307	241	238	238
query9	9700	2760	2762	2760
query10	493	256	247	247
query11	18124	15157	15253	15157
query12	166	106	112	106
query13	1686	442	431	431
query14	11500	7321	7498	7321
query15	299	189	180	180
query16	8102	453	458	453
query17	1789	600	583	583
query18	2124	338	307	307
query19	378	154	150	150
query20	119	114	112	112
query21	210	103	101	101
query22	4733	4524	4535	4524
query23	34446	33631	33698	33631
query24	11369	2482	2412	2412
query25	667	374	400	374
query26	1766	153	150	150
query27	2835	327	329	327
query28	8090	2450	2467	2450
query29	1022	407	406	406
query30	307	150	149	149
query31	1072	828	802	802
query32	96	58	78	58
query33	776	289	288	288
query34	999	509	508	508
query35	858	730	732	730
query36	1106	945	952	945
query37	276	75	77	75
query38	4336	4316	4127	4127
query39	1493	1458	1406	1406
query40	283	109	103	103
query41	50	47	44	44
query42	116	101	103	101
query43	551	512	491	491
query44	1285	808	804	804
query45	180	167	167	167
query46	1152	692	722	692
query47	1945	1857	1880	1857
query48	410	325	322	322
query49	1283	393	394	393
query50	789	391	387	387
query51	7137	6978	6986	6978
query52	108	94	89	89
query53	259	185	189	185
query54	1306	407	419	407
query55	81	76	83	76
query56	268	243	245	243
query57	1298	1133	1112	1112
query58	231	222	227	222
query59	3202	3143	2962	2962
query60	267	252	256	252
query61	112	107	121	107
query62	916	660	681	660
query63	214	190	199	190
query64	4927	685	651	651
query65	3306	3204	3261	3204
query66	1416	311	303	303
query67	15907	15619	15524	15524
query68	5810	568	570	568
query69	428	251	276	251
query70	1213	1138	1137	1137
query71	432	255	276	255
query72	6298	4089	4113	4089
query73	787	369	360	360
query74	10488	8878	8833	8833
query75	3435	2637	2643	2637
query76	3486	1070	986	986
query77	533	271	274	271
query78	10305	9465	9448	9448
query79	1686	616	598	598
query80	1034	425	438	425
query81	536	237	235	235
query82	943	120	120	120
query83	253	152	149	149
query84	237	80	70	70
query85	1409	316	324	316
query86	455	275	292	275
query87	4495	4468	4322	4322
query88	3518	2262	2223	2223
query89	421	283	300	283
query90	2193	186	186	186
query91	136	109	101	101
query92	63	55	54	54
query93	1719	565	557	557
query94	1117	275	290	275
query95	359	262	268	262
query96	612	274	283	274
query97	2849	2705	2717	2705
query98	220	195	200	195
query99	1552	1348	1325	1325
Total cold run time: 306712 ms
Total hot run time: 191226 ms

doris-robot · 2024-12-23T11:16:41Z

ClickBench: Total hot run time: 32.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1d8cb608ec159203e7eb095da8eb869792498f31, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.59	0.10	0.11
query5	0.42	0.39	0.42
query6	1.14	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.52
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.61	0.59
query14	2.73	2.71	2.72
query15	0.90	0.82	0.83
query16	0.39	0.39	0.38
query17	1.05	1.03	1.07
query18	0.23	0.22	0.22
query19	1.93	1.79	2.04
query20	0.01	0.01	0.01
query21	15.38	0.59	0.57
query22	2.72	1.98	1.91
query23	16.99	0.95	0.91
query24	2.59	1.30	0.13
query25	0.17	0.05	0.07
query26	0.34	0.14	0.14
query27	0.06	0.04	0.05
query28	11.71	1.11	1.09
query29	12.59	3.33	3.31
query30	0.25	0.07	0.06
query31	2.85	0.39	0.38
query32	3.25	0.47	0.45
query33	3.10	3.11	3.05
query34	17.11	4.53	4.53
query35	4.57	4.51	4.51
query36	0.68	0.47	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.66 s
Total hot run time: 32.04 s

morrySnow · 2024-12-25T03:46:50Z

fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java

+            LOG.info("computeOlapScan optStats isPresent {}, tableRowCount is {}",
+                    optStats.isPresent(), tableRowCount);
            if (optStats.isPresent()) {
                double selectedPartitionsRowCount = getSelectedPartitionRowCount(olapScan, tableRowCount);
+                LOG.info("computeOlapScan optStats is {}, selectedPartitionsRowCount is {}", optStats.get(),
+                        selectedPartitionsRowCount);


remove these logs

seawinde · 2024-12-25T11:36:00Z

run buildall

doris-robot · 2024-12-25T11:56:14Z

TPC-H: Total hot run time: 32246 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c0c2c81208431cc66f9adb61731d53346edd5836, data reload: false

------ Round 1 ----------------------------------
q1	17583	6075	5997	5997
q2	2044	290	163	163
q3	10438	1281	720	720
q4	10212	871	453	453
q5	7506	2131	1930	1930
q6	205	176	145	145
q7	878	744	602	602
q8	9236	1329	1128	1128
q9	5324	4865	4958	4865
q10	6769	2336	1837	1837
q11	464	273	256	256
q12	340	357	222	222
q13	17767	3544	2942	2942
q14	243	235	205	205
q15	562	521	496	496
q16	616	627	575	575
q17	541	835	315	315
q18	6954	6406	6408	6406
q19	1237	961	550	550
q20	304	322	204	204
q21	2709	2163	1925	1925
q22	357	323	310	310
Total cold run time: 102289 ms
Total hot run time: 32246 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6154	6168	6162	6162
q2	235	327	230	230
q3	2243	2628	2280	2280
q4	1423	1809	1339	1339
q5	4291	4724	4626	4626
q6	178	172	133	133
q7	1947	1844	1697	1697
q8	2461	2648	2588	2588
q9	6943	6933	6900	6900
q10	2924	3227	2689	2689
q11	560	506	508	506
q12	631	709	587	587
q13	3177	3598	3022	3022
q14	261	299	262	262
q15	552	496	504	496
q16	628	676	641	641
q17	1146	1653	1232	1232
q18	7156	7168	7016	7016
q19	761	1071	1035	1035
q20	1921	1977	1839	1839
q21	5384	5004	4844	4844
q22	602	617	595	595
Total cold run time: 51578 ms
Total hot run time: 50719 ms

doris-robot · 2024-12-25T12:08:20Z

TPC-DS: Total hot run time: 189740 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c0c2c81208431cc66f9adb61731d53346edd5836, data reload: false

query1	962	382	377	377
query2	6517	2351	2232	2232
query3	6712	214	209	209
query4	33622	23345	23700	23345
query5	4587	599	485	485
query6	281	217	192	192
query7	4622	492	320	320
query8	300	278	232	232
query9	9230	2769	2766	2766
query10	451	317	244	244
query11	17970	15252	15312	15252
query12	161	104	106	104
query13	1668	535	407	407
query14	11546	6809	6824	6809
query15	244	197	188	188
query16	8038	559	433	433
query17	1603	732	560	560
query18	2121	426	291	291
query19	202	174	145	145
query20	117	107	107	107
query21	205	119	101	101
query22	4504	4524	4145	4145
query23	34437	33165	33343	33165
query24	6666	2235	2229	2229
query25	498	458	379	379
query26	1222	266	148	148
query27	2599	446	333	333
query28	5692	2459	2448	2448
query29	748	537	419	419
query30	233	179	155	155
query31	986	881	801	801
query32	94	62	57	57
query33	496	342	310	310
query34	738	816	511	511
query35	801	812	733	733
query36	991	1041	972	972
query37	117	97	74	74
query38	4164	4372	4036	4036
query39	1494	1442	1434	1434
query40	200	113	99	99
query41	47	45	52	45
query42	120	103	100	100
query43	491	536	480	480
query44	1307	804	822	804
query45	180	170	160	160
query46	848	1032	655	655
query47	1927	1914	1844	1844
query48	374	410	339	339
query49	788	469	382	382
query50	602	642	386	386
query51	7102	7147	7044	7044
query52	99	100	92	92
query53	224	249	189	189
query54	472	474	403	403
query55	80	77	79	77
query56	246	256	239	239
query57	1187	1163	1108	1108
query58	232	219	225	219
query59	2952	3129	3058	3058
query60	282	259	252	252
query61	116	107	106	106
query62	871	802	741	741
query63	227	189	187	187
query64	4651	993	638	638
query65	3243	3134	3198	3134
query66	1074	435	305	305
query67	15898	15774	15428	15428
query68	9882	751	527	527
query69	463	283	246	246
query70	1180	1144	1109	1109
query71	447	279	250	250
query72	5837	3778	3863	3778
query73	665	747	366	366
query74	10036	9043	8925	8925
query75	4636	3142	2651	2651
query76	5540	1185	836	836
query77	1026	379	267	267
query78	10057	10161	10367	10161
query79	5171	923	577	577
query80	656	506	442	442
query81	473	272	237	237
query82	594	156	125	125
query83	201	166	153	153
query84	288	94	75	75
query85	852	367	295	295
query86	363	317	272	272
query87	4440	4732	4273	4273
query88	3871	2213	2185	2185
query89	458	338	296	296
query90	1902	184	183	183
query91	135	137	114	114
query92	70	55	53	53
query93	3097	870	537	537
query94	671	382	283	283
query95	333	261	248	248
query96	486	609	274	274
query97	2698	2794	2696	2696
query98	221	206	197	197
query99	1641	1546	1447	1447
Total cold run time: 301869 ms
Total hot run time: 189740 ms

doris-robot · 2024-12-25T12:13:42Z

ClickBench: Total hot run time: 31.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c0c2c81208431cc66f9adb61731d53346edd5836, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.44	0.40	0.40
query6	1.17	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.57	0.51	0.50
query10	0.55	0.57	0.55
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.59
query14	2.73	2.75	2.88
query15	0.88	0.82	0.83
query16	0.38	0.38	0.38
query17	1.06	0.99	1.05
query18	0.22	0.20	0.21
query19	1.91	1.99	1.87
query20	0.01	0.01	0.01
query21	15.35	0.98	0.58
query22	0.75	0.85	0.66
query23	15.25	1.43	0.59
query24	3.59	1.16	0.85
query25	0.21	0.13	0.18
query26	0.22	0.14	0.14
query27	0.05	0.05	0.04
query28	13.62	1.50	1.04
query29	12.58	3.92	3.24
query30	0.25	0.10	0.07
query31	2.81	0.59	0.38
query32	3.22	0.54	0.45
query33	3.06	3.07	3.13
query34	16.93	5.16	4.60
query35	4.54	4.49	4.51
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.39 s
Total hot run time: 31.32 s

… view

morrySnow · 2025-01-09T06:55:05Z

run buildall

doris-robot · 2025-01-09T07:56:01Z

TPC-H: Total hot run time: 32856 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78f4262cf357d24732930cee87d1620fbe9f6dbb, data reload: false

------ Round 1 ----------------------------------
q1	17588	6149	6016	6016
q2	2046	314	182	182
q3	10404	1275	775	775
q4	10246	886	445	445
q5	8372	2204	1992	1992
q6	208	185	144	144
q7	901	781	602	602
q8	9243	1396	1154	1154
q9	5175	4891	4897	4891
q10	6761	2309	1863	1863
q11	486	273	253	253
q12	350	357	228	228
q13	17759	3721	3082	3082
q14	235	242	213	213
q15	563	501	488	488
q16	640	625	589	589
q17	575	857	314	314
q18	7067	6566	6489	6489
q19	2325	1016	592	592
q20	308	319	182	182
q21	2938	2233	2051	2051
q22	369	339	311	311
Total cold run time: 104559 ms
Total hot run time: 32856 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6337	6184	6210	6184
q2	233	328	239	239
q3	2248	2629	2312	2312
q4	1422	1831	1391	1391
q5	4377	4753	4923	4753
q6	188	179	145	145
q7	2094	1973	1849	1849
q8	2641	2772	2732	2732
q9	7349	7215	7283	7215
q10	3097	3296	2726	2726
q11	582	514	520	514
q12	711	761	634	634
q13	3418	3901	3263	3263
q14	291	305	289	289
q15	561	493	519	493
q16	681	701	646	646
q17	1244	1745	1265	1265
q18	7741	7541	7311	7311
q19	853	1187	1116	1116
q20	1984	2028	1945	1945
q21	5842	5063	4984	4984
q22	619	602	617	602
Total cold run time: 54513 ms
Total hot run time: 52608 ms

doris-robot · 2025-01-09T08:08:09Z

TPC-DS: Total hot run time: 194106 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78f4262cf357d24732930cee87d1620fbe9f6dbb, data reload: false

query1	1310	933	924	924
query2	6276	2413	2360	2360
query3	10993	4714	4811	4714
query4	32783	23499	23251	23251
query5	4116	582	438	438
query6	283	199	186	186
query7	3981	500	305	305
query8	296	238	226	226
query9	9118	2695	2688	2688
query10	469	303	256	256
query11	17872	15193	14943	14943
query12	156	109	102	102
query13	1594	555	428	428
query14	10024	6993	7472	6993
query15	253	223	197	197
query16	7944	570	438	438
query17	1540	787	577	577
query18	2110	418	314	314
query19	230	187	159	159
query20	123	120	110	110
query21	203	125	104	104
query22	4480	4540	4252	4252
query23	33705	33149	32968	32968
query24	6585	2293	2310	2293
query25	480	449	393	393
query26	734	284	158	158
query27	2055	469	338	338
query28	5445	2476	2470	2470
query29	599	593	435	435
query30	208	182	152	152
query31	956	901	822	822
query32	77	63	59	59
query33	496	389	327	327
query34	790	840	539	539
query35	818	818	760	760
query36	1020	1039	974	974
query37	126	109	80	80
query38	4181	4193	4118	4118
query39	1522	1467	1442	1442
query40	205	116	104	104
query41	48	44	44	44
query42	116	107	100	100
query43	522	530	491	491
query44	1309	856	836	836
query45	178	179	163	163
query46	877	1062	656	656
query47	1916	1912	1850	1850
query48	410	443	326	326
query49	715	471	400	400
query50	646	660	401	401
query51	7041	7081	7002	7002
query52	99	99	90	90
query53	227	251	187	187
query54	492	499	415	415
query55	85	79	85	79
query56	255	247	245	245
query57	1173	1185	1145	1145
query58	260	225	221	221
query59	3249	3417	3081	3081
query60	274	269	243	243
query61	117	114	118	114
query62	831	805	699	699
query63	237	197	198	197
query64	3020	1072	660	660
query65	3278	3308	3228	3228
query66	891	409	302	302
query67	16377	15608	15371	15371
query68	8075	698	516	516
query69	480	303	250	250
query70	1115	1149	1135	1135
query71	455	286	254	254
query72	6255	3931	3809	3809
query73	665	750	359	359
query74	10304	9118	8747	8747
query75	4071	3178	2636	2636
query76	3620	1145	763	763
query77	749	363	275	275
query78	10078	10503	9331	9331
query79	2875	792	603	603
query80	638	538	564	538
query81	478	268	224	224
query82	535	144	122	122
query83	169	161	149	149
query84	233	94	69	69
query85	765	358	303	303
query86	336	307	300	300
query87	4525	4272	4296	4272
query88	4533	2215	2181	2181
query89	406	369	288	288
query90	1837	188	190	188
query91	136	131	102	102
query92	70	56	51	51
query93	1003	853	546	546
query94	654	372	293	293
query95	334	271	251	251
query96	482	603	289	289
query97	2879	2933	2736	2736
query98	234	199	189	189
query99	1567	1471	1394	1394
Total cold run time: 291355 ms
Total hot run time: 194106 ms

doris-robot · 2025-01-09T08:13:29Z

ClickBench: Total hot run time: 31.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78f4262cf357d24732930cee87d1620fbe9f6dbb, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.43	0.40
query6	1.15	0.65	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.56	0.56	0.55
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.60	0.61	0.60
query14	2.74	2.83	2.74
query15	0.90	0.82	0.81
query16	0.38	0.39	0.38
query17	0.95	1.06	1.01
query18	0.23	0.21	0.20
query19	1.84	1.80	1.98
query20	0.01	0.01	0.01
query21	15.39	0.89	0.57
query22	0.76	0.83	0.67
query23	15.25	1.43	0.52
query24	2.54	1.42	1.65
query25	0.20	0.28	0.08
query26	0.22	0.14	0.13
query27	0.07	0.04	0.06
query28	14.29	1.51	1.04
query29	12.58	4.03	3.36
query30	0.25	0.10	0.07
query31	2.83	0.62	0.37
query32	3.22	0.54	0.46
query33	3.05	3.14	3.03
query34	16.94	5.07	4.49
query35	4.44	4.44	4.46
query36	0.80	0.47	0.47
query37	0.09	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 31.58 s

morrySnow · 2025-01-09T08:31:33Z

need more test cases

seawinde · 2025-01-09T11:06:27Z

run buildall

doris-robot · 2025-01-09T11:26:32Z

TPC-H: Total hot run time: 32573 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fff7068e0eec01347b27cf3230df8ac1bf93662e, data reload: false

------ Round 1 ----------------------------------
q1	17575	6201	6037	6037
q2	2039	312	172	172
q3	10497	1280	724	724
q4	10257	871	437	437
q5	7872	2183	1949	1949
q6	208	187	149	149
q7	901	739	626	626
q8	9236	1361	1184	1184
q9	5178	4802	4910	4802
q10	6759	2319	1854	1854
q11	470	291	267	267
q12	347	356	222	222
q13	17759	3622	3055	3055
q14	241	238	218	218
q15	557	501	507	501
q16	637	618	587	587
q17	580	849	339	339
q18	7171	6467	6433	6433
q19	2258	963	530	530
q20	312	315	184	184
q21	2792	2260	1995	1995
q22	368	327	308	308
Total cold run time: 104014 ms
Total hot run time: 32573 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6295	6219	6209	6209
q2	240	325	240	240
q3	2273	2681	2284	2284
q4	1441	1827	1328	1328
q5	4330	4749	4842	4749
q6	192	187	144	144
q7	2024	1980	1848	1848
q8	2612	3030	2644	2644
q9	7303	7320	7174	7174
q10	3051	3326	2822	2822
q11	598	518	507	507
q12	689	756	593	593
q13	3433	3868	3236	3236
q14	276	294	279	279
q15	553	509	515	509
q16	646	695	637	637
q17	1199	1744	1265	1265
q18	7785	7309	7021	7021
q19	788	1102	991	991
q20	1915	1962	1816	1816
q21	5364	5095	4894	4894
q22	597	581	584	581
Total cold run time: 53604 ms
Total hot run time: 51771 ms

doris-robot · 2025-01-09T11:38:27Z

TPC-DS: Total hot run time: 187981 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fff7068e0eec01347b27cf3230df8ac1bf93662e, data reload: false

query1	967	368	378	368
query2	6515	2430	2385	2385
query3	6708	222	229	222
query4	33735	23385	23091	23091
query5	4354	616	451	451
query6	307	197	197	197
query7	4625	495	312	312
query8	315	256	250	250
query9	9598	2728	2710	2710
query10	464	310	269	269
query11	17982	15252	15130	15130
query12	173	107	105	105
query13	1650	520	400	400
query14	9613	6731	7490	6731
query15	282	201	178	178
query16	8318	617	416	416
query17	1606	738	570	570
query18	2133	407	328	328
query19	213	175	146	146
query20	112	112	110	110
query21	214	119	101	101
query22	4213	4271	4263	4263
query23	34205	32986	33106	32986
query24	6367	2257	2265	2257
query25	498	461	381	381
query26	1210	269	152	152
query27	2138	449	341	341
query28	5220	2452	2414	2414
query29	733	533	429	429
query30	224	177	151	151
query31	983	862	782	782
query32	82	63	81	63
query33	501	350	312	312
query34	754	827	526	526
query35	804	797	733	733
query36	1002	1002	923	923
query37	122	102	78	78
query38	4027	4267	3964	3964
query39	1482	1426	1397	1397
query40	208	113	102	102
query41	50	52	48	48
query42	131	104	103	103
query43	513	522	500	500
query44	1331	830	826	826
query45	177	167	168	167
query46	844	1025	634	634
query47	1802	1832	1762	1762
query48	389	390	319	319
query49	776	471	384	384
query50	633	631	391	391
query51	6868	7043	6945	6945
query52	101	103	90	90
query53	225	245	177	177
query54	485	479	408	408
query55	83	76	78	76
query56	243	250	240	240
query57	1179	1155	1096	1096
query58	238	234	229	229
query59	3123	3082	2955	2955
query60	273	262	254	254
query61	108	108	129	108
query62	843	777	715	715
query63	221	188	192	188
query64	4579	1011	635	635
query65	3237	3159	3123	3123
query66	1050	411	317	317
query67	15846	15678	15371	15371
query68	8438	687	518	518
query69	463	297	259	259
query70	1224	1147	1129	1129
query71	433	283	268	268
query72	6191	3894	3797	3797
query73	687	754	356	356
query74	10246	8864	8640	8640
query75	4542	3126	2650	2650
query76	3909	1179	769	769
query77	773	367	267	267
query78	9870	9866	9402	9402
query79	3322	826	591	591
query80	793	512	429	429
query81	457	277	229	229
query82	468	150	127	127
query83	197	171	159	159
query84	277	89	71	71
query85	778	342	288	288
query86	351	298	309	298
query87	4573	4377	4323	4323
query88	3264	2210	2218	2210
query89	413	325	291	291
query90	1927	189	188	188
query91	139	138	107	107
query92	64	61	55	55
query93	1876	835	534	534
query94	667	408	279	279
query95	337	277	251	251
query96	485	607	278	278
query97	2857	2880	2839	2839
query98	235	196	197	196
query99	1634	1513	1398	1398
Total cold run time: 291895 ms
Total hot run time: 187981 ms

doris-robot · 2025-01-09T11:43:46Z

ClickBench: Total hot run time: 30.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fff7068e0eec01347b27cf3230df8ac1bf93662e, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.40	0.40
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.59
query14	2.73	2.82	2.73
query15	0.90	0.81	0.83
query16	0.40	0.39	0.39
query17	1.07	0.99	1.06
query18	0.23	0.21	0.20
query19	1.91	1.70	1.93
query20	0.01	0.02	0.01
query21	15.35	0.94	0.58
query22	0.74	0.81	0.69
query23	15.22	1.37	0.57
query24	2.94	1.73	0.45
query25	0.20	0.08	0.14
query26	0.35	0.16	0.14
query27	0.07	0.05	0.05
query28	13.46	1.55	1.05
query29	12.59	3.94	3.31
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.45
query33	3.07	3.14	3.03
query34	16.82	5.08	4.43
query35	4.53	4.43	4.43
query36	0.63	0.51	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.64 s
Total hot run time: 30.5 s

seawinde · 2025-01-10T03:07:31Z

run buildall

zfr9527

LGTM

github-actions · 2025-01-10T03:08:41Z

PR approved by anyone and no changes requested.

doris-robot · 2025-01-10T03:30:13Z

TPC-H: Total hot run time: 33049 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7f46651f2893c72a25e44d1941e4671aab9c3c5, data reload: false

------ Round 1 ----------------------------------
q1	17587	6841	6094	6094
q2	2041	314	170	170
q3	10416	1305	754	754
q4	10208	875	447	447
q5	7527	2208	2086	2086
q6	214	182	149	149
q7	911	772	612	612
q8	9243	1430	1211	1211
q9	5278	4879	4870	4870
q10	6763	2316	1854	1854
q11	488	282	262	262
q12	356	361	227	227
q13	17777	3636	3075	3075
q14	243	238	221	221
q15	555	499	497	497
q16	623	611	590	590
q17	601	866	347	347
q18	6947	6555	6455	6455
q19	1203	959	568	568
q20	317	329	197	197
q21	3159	2275	2050	2050
q22	376	329	313	313
Total cold run time: 102833 ms
Total hot run time: 33049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6232	6265	6280	6265
q2	237	331	233	233
q3	2235	2675	2316	2316
q4	1450	1810	1393	1393
q5	4384	4744	4918	4744
q6	185	177	139	139
q7	2086	1950	1879	1879
q8	2653	2854	2726	2726
q9	7244	7255	7170	7170
q10	3017	3345	2847	2847
q11	588	539	493	493
q12	651	767	609	609
q13	3447	3836	3260	3260
q14	286	327	287	287
q15	578	523	502	502
q16	645	689	640	640
q17	1232	1737	1259	1259
q18	7925	7471	7303	7303
q19	848	1147	1109	1109
q20	1994	1993	1840	1840
q21	5669	5410	4907	4907
q22	638	609	581	581
Total cold run time: 54224 ms
Total hot run time: 52502 ms

doris-robot · 2025-01-10T03:42:11Z

TPC-DS: Total hot run time: 187609 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7f46651f2893c72a25e44d1941e4671aab9c3c5, data reload: false

query1	970	372	391	372
query2	6539	2402	2257	2257
query3	6721	220	222	220
query4	34095	23223	23661	23223
query5	4852	627	479	479
query6	306	208	201	201
query7	4618	502	299	299
query8	305	252	233	233
query9	9369	2753	2726	2726
query10	484	310	269	269
query11	18306	15164	15152	15152
query12	172	112	111	111
query13	1643	568	415	415
query14	10335	7255	6595	6595
query15	241	198	196	196
query16	8049	639	484	484
query17	1566	724	569	569
query18	2123	438	291	291
query19	216	181	146	146
query20	111	114	111	111
query21	208	116	96	96
query22	4205	4179	4179	4179
query23	33832	32561	33869	32561
query24	7125	2230	2221	2221
query25	459	467	387	387
query26	1049	259	149	149
query27	2451	460	334	334
query28	5512	2431	2420	2420
query29	506	540	412	412
query30	231	189	155	155
query31	973	858	799	799
query32	105	56	59	56
query33	525	344	294	294
query34	741	838	508	508
query35	814	810	748	748
query36	1019	1056	929	929
query37	131	99	76	76
query38	4056	4208	4007	4007
query39	1497	1395	1401	1395
query40	209	120	99	99
query41	63	54	53	53
query42	125	104	101	101
query43	515	520	493	493
query44	1349	834	822	822
query45	173	169	172	169
query46	849	1009	645	645
query47	1821	1799	1730	1730
query48	386	401	329	329
query49	772	481	375	375
query50	623	647	391	391
query51	6952	6968	6815	6815
query52	105	103	94	94
query53	225	246	179	179
query54	479	499	412	412
query55	88	84	82	82
query56	262	239	238	238
query57	1119	1154	1056	1056
query58	238	221	228	221
query59	3028	3082	3111	3082
query60	311	253	242	242
query61	113	109	120	109
query62	838	797	710	710
query63	227	188	189	188
query64	4537	1060	638	638
query65	3226	3174	3187	3174
query66	1040	435	314	314
query67	15640	15611	15274	15274
query68	8494	700	555	555
query69	465	295	257	257
query70	1212	1135	1124	1124
query71	436	294	271	271
query72	6063	3869	3794	3794
query73	677	757	355	355
query74	10085	9213	9035	9035
query75	4153	3130	2626	2626
query76	3934	1180	761	761
query77	768	366	268	268
query78	9965	9902	9445	9445
query79	2865	801	597	597
query80	618	541	427	427
query81	496	264	223	223
query82	520	157	117	117
query83	168	167	148	148
query84	233	87	77	77
query85	808	367	294	294
query86	412	327	292	292
query87	4455	4298	4345	4298
query88	4534	2186	2164	2164
query89	399	326	292	292
query90	1838	191	189	189
query91	134	136	111	111
query92	67	60	57	57
query93	1826	852	531	531
query94	674	406	289	289
query95	342	264	247	247
query96	499	603	272	272
query97	2948	2984	2804	2804
query98	234	214	199	199
query99	1483	1479	1380	1380
Total cold run time: 293578 ms
Total hot run time: 187609 ms

doris-robot · 2025-01-10T03:47:30Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7f46651f2893c72a25e44d1941e4671aab9c3c5, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.11	0.10
query5	0.41	0.41	0.41
query6	1.15	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.49
query10	0.54	0.56	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.61	0.60
query14	2.84	2.73	2.87
query15	0.89	0.83	0.83
query16	0.37	0.38	0.38
query17	1.02	1.05	1.07
query18	0.23	0.21	0.21
query19	1.95	1.88	1.95
query20	0.01	0.00	0.01
query21	15.38	0.97	0.59
query22	0.75	0.86	0.97
query23	14.94	1.48	0.53
query24	3.26	0.46	0.65
query25	0.21	0.12	0.22
query26	0.39	0.14	0.14
query27	0.09	0.05	0.03
query28	13.46	1.54	1.05
query29	12.56	3.98	3.26
query30	0.25	0.08	0.06
query31	2.84	0.62	0.38
query32	3.22	0.54	0.45
query33	3.17	3.09	3.11
query34	16.82	5.08	4.52
query35	4.52	4.41	4.41
query36	0.67	0.48	0.52
query37	0.10	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.9 s
Total hot run time: 30.99 s

morrySnow reviewed Dec 25, 2024

View reviewed changes

morrySnow added dev/2.1.x dev/3.0.x labels Dec 25, 2024

seawinde added 3 commits January 7, 2025 10:39

[opt](mtmv) Support null aware left anti join rewrite by materialized…

ca7bf08

… view

add test

c6b5c4b

remove useless code

78f4262

seawinde force-pushed the support_null_aware_left_anti_join_mv_rewrite branch from c0c2c81 to 78f4262 Compare January 7, 2025 03:43

morrySnow marked this pull request as draft January 9, 2025 07:09

add more test

fff7068

add more test

b7f4665

seawinde marked this pull request as ready for review January 10, 2025 03:07

zfr9527 approved these changes Jan 10, 2025

View reviewed changes

github-actions bot added the reviewed label Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](mtmv) Support null aware left anti join rewrite by materialized view #45815

[opt](mtmv) Support null aware left anti join rewrite by materialized view #45815

seawinde commented Dec 23, 2024

Thearas commented Dec 23, 2024

seawinde commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

morrySnow Dec 25, 2024

seawinde commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

morrySnow commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

morrySnow commented Jan 9, 2025

seawinde commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

seawinde commented Jan 10, 2025

zfr9527 left a comment

github-actions bot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

[opt](mtmv) Support null aware left anti join rewrite by materialized view #45815

Are you sure you want to change the base?

[opt](mtmv) Support null aware left anti join rewrite by materialized view #45815

Conversation

seawinde commented Dec 23, 2024

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Thearas commented Dec 23, 2024

seawinde commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

morrySnow Dec 25, 2024

Choose a reason for hiding this comment

seawinde commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

morrySnow commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

morrySnow commented Jan 9, 2025

seawinde commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

seawinde commented Jan 10, 2025

zfr9527 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025