Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](memory) Fix memory log compile by ASAN #33162

Merged
merged 3 commits into from
Apr 3, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Apr 2, 2024

Proposed changes

ASAN compiles BE, add markers in memory logs

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Apr 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39040 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9959bcb7eb4addca7253b3b1004a8bc79a407896, data reload: false

------ Round 1 ----------------------------------
q1	18626	4338	4197	4197
q2	2655	203	195	195
q3	11399	1265	1462	1265
q4	10715	889	1052	889
q5	7489	2991	2932	2932
q6	218	134	135	134
q7	990	569	551	551
q8	9424	2064	2049	2049
q9	6748	6237	6226	6226
q10	8402	3531	3502	3502
q11	417	243	237	237
q12	381	223	214	214
q13	17770	2896	2914	2896
q14	273	243	246	243
q15	530	497	484	484
q16	486	383	382	382
q17	968	923	930	923
q18	7269	6586	6466	6466
q19	1602	1550	1546	1546
q20	635	341	318	318
q21	3613	3180	3094	3094
q22	366	300	297	297
Total cold run time: 110976 ms
Total hot run time: 39040 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4066	4047	4047
q2	335	223	219	219
q3	2948	2971	2978	2971
q4	1895	1870	1853	1853
q5	5257	5236	5216	5216
q6	210	127	124	124
q7	1767	1352	1383	1352
q8	3240	3305	3287	3287
q9	8504	8497	8462	8462
q10	3776	3861	3829	3829
q11	537	448	436	436
q12	730	549	553	549
q13	16638	2919	2929	2919
q14	300	267	258	258
q15	513	475	470	470
q16	463	402	409	402
q17	1713	1695	1686	1686
q18	7653	7436	7269	7269
q19	1636	1644	1667	1644
q20	1947	1723	1717	1717
q21	5115	4678	4648	4648
q22	495	417	446	417
Total cold run time: 69749 ms
Total hot run time: 53775 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9959bcb7eb4addca7253b3b1004a8bc79a407896, data reload: false

query1	1235	1130	1125	1125
query2	6492	1796	1770	1770
query3	6665	220	221	220
query4	24351	21470	21140	21140
query5	4198	402	423	402
query6	271	181	187	181
query7	4585	309	306	306
query8	237	189	186	186
query9	8471	2291	2291	2291
query10	565	258	255	255
query11	14861	14697	14511	14511
query12	145	102	101	101
query13	1646	392	395	392
query14	8663	6831	6724	6724
query15	201	182	179	179
query16	7135	278	270	270
query17	1001	600	579	579
query18	1891	303	273	273
query19	196	159	150	150
query20	96	95	96	95
query21	195	128	122	122
query22	4927	4858	4811	4811
query23	33558	32808	32837	32808
query24	12508	3166	3122	3122
query25	695	397	402	397
query26	1905	168	159	159
query27	3006	333	330	330
query28	6763	1878	1861	1861
query29	1361	607	596	596
query30	310	168	173	168
query31	983	735	734	734
query32	102	60	62	60
query33	703	264	256	256
query34	1025	485	488	485
query35	843	696	690	690
query36	986	861	875	861
query37	282	86	78	78
query38	3574	3456	3361	3361
query39	1571	1532	1540	1532
query40	300	136	141	136
query41	48	47	47	47
query42	109	101	100	100
query43	444	388	400	388
query44	1091	716	703	703
query45	283	262	262	262
query46	1072	790	756	756
query47	1898	1819	1819	1819
query48	387	314	309	309
query49	1158	368	358	358
query50	793	392	403	392
query51	6783	6764	6685	6685
query52	111	94	101	94
query53	359	291	287	287
query54	320	241	253	241
query55	92	83	76	76
query56	242	232	229	229
query57	1243	1138	1132	1132
query58	265	223	234	223
query59	2558	2304	2286	2286
query60	280	258	242	242
query61	112	107	107	107
query62	693	455	457	455
query63	312	282	289	282
query64	6473	3346	3149	3149
query65	3060	2994	3019	2994
query66	1465	335	341	335
query67	15656	15206	15053	15053
query68	8731	588	585	585
query69	547	324	336	324
query70	1231	1109	1118	1109
query71	489	276	274	274
query72	6393	2570	2419	2419
query73	865	326	325	325
query74	6690	6282	6287	6282
query75	3334	2333	2278	2278
query76	5091	1140	1189	1140
query77	614	252	269	252
query78	10733	10211	10172	10172
query79	9792	526	524	524
query80	1380	426	424	424
query81	509	241	247	241
query82	392	112	103	103
query83	224	172	161	161
query84	271	92	93	92
query85	1019	283	285	283
query86	376	269	273	269
query87	3680	3562	3460	3460
query88	3729	2349	2370	2349
query89	535	370	370	370
query90	2049	178	184	178
query91	131	106	106	106
query92	62	50	54	50
query93	6755	539	537	537
query94	1324	197	188	188
query95	438	329	326	326
query96	609	271	268	268
query97	2641	2489	2477	2477
query98	231	217	203	203
query99	1207	836	846	836
Total cold run time: 297881 ms
Total hot run time: 181326 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9959bcb7eb4addca7253b3b1004a8bc79a407896, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.06
query5	0.48	0.48	0.50
query6	1.12	0.65	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.52
query10	0.57	0.57	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.59
query14	0.76	0.78	0.78
query15	0.84	0.84	0.84
query16	0.37	0.36	0.35
query17	1.00	0.98	0.99
query18	0.26	0.25	0.25
query19	1.79	1.75	1.75
query20	0.01	0.01	0.01
query21	15.43	0.76	0.68
query22	2.82	5.25	2.39
query23	17.78	1.41	1.14
query24	1.52	0.37	0.22
query25	0.15	0.09	0.08
query26	0.30	0.17	0.18
query27	0.08	0.09	0.08
query28	13.39	0.96	0.98
query29	12.60	3.40	3.34
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.27	0.47	0.48
query33	2.91	2.86	2.93
query34	15.49	4.37	4.39
query35	4.37	4.39	4.39
query36	0.68	0.47	0.48
query37	0.19	0.17	0.17
query38	0.17	0.16	0.15
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.09	0.05	0.05
query42	0.07	0.05	0.06
query43	0.05	0.05	0.05
Total cold run time: 105.47 s
Total hot run time: 30.91 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9959bcb7eb4addca7253b3b1004a8bc79a407896 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.4 seconds inserted 10000000 Rows, about 609K ops/s

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Apr 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8880/24911)
Line Coverage: 27.38% (72913/266309)
Region Coverage: 26.56% (37712/141984)
Branch Coverage: 23.35% (19223/82310)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bcfd6e284666df9894a112e13640aeb55418eac8_bcfd6e284666df9894a112e13640aeb55418eac8/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38587 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bcfd6e284666df9894a112e13640aeb55418eac8, data reload: false

------ Round 1 ----------------------------------
q1	17725	4331	4063	4063
q2	2012	193	188	188
q3	10613	1256	1399	1256
q4	10401	929	999	929
q5	7479	2950	2907	2907
q6	219	132	135	132
q7	1099	639	597	597
q8	9395	2024	2013	2013
q9	6594	6132	6133	6132
q10	8394	3493	3506	3493
q11	411	242	247	242
q12	386	221	218	218
q13	17780	2892	2891	2891
q14	272	238	242	238
q15	540	492	492	492
q16	501	379	374	374
q17	938	893	910	893
q18	7305	6463	6307	6307
q19	1614	1541	1543	1541
q20	607	328	316	316
q21	3546	3134	3058	3058
q22	361	308	307	307
Total cold run time: 108192 ms
Total hot run time: 38587 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4167	4015	4023	4015
q2	330	221	224	221
q3	2927	2923	2936	2923
q4	1883	1835	1817	1817
q5	5240	5216	5184	5184
q6	208	125	129	125
q7	2235	1797	1808	1797
q8	3195	3270	3239	3239
q9	8438	8436	8451	8436
q10	3730	3856	3845	3845
q11	539	461	459	459
q12	741	575	583	575
q13	15586	3052	3078	3052
q14	313	288	282	282
q15	541	476	477	476
q16	490	444	425	425
q17	1742	1729	1715	1715
q18	8132	7801	7696	7696
q19	1675	1693	1685	1685
q20	2010	1780	1743	1743
q21	5061	4996	4964	4964
q22	522	416	457	416
Total cold run time: 69705 ms
Total hot run time: 55090 ms

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 3, 2024

run buildall

Copy link
Contributor

github-actions bot commented Apr 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39512 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e998aad3cbdf5e73eae7e2f321ec994f8c3028a, data reload: false

------ Round 1 ----------------------------------
q1	18971	4269	4217	4217
q2	2605	201	193	193
q3	11144	1253	1413	1253
q4	10209	884	936	884
q5	7472	2968	2930	2930
q6	220	139	140	139
q7	1137	626	614	614
q8	9423	2042	2042	2042
q9	7422	6773	6733	6733
q10	8465	3538	3495	3495
q11	428	247	241	241
q12	380	223	218	218
q13	17785	2908	2902	2902
q14	275	250	250	250
q15	529	495	493	493
q16	526	392	375	375
q17	974	923	900	900
q18	7218	6520	6431	6431
q19	1610	1537	1521	1521
q20	609	326	321	321
q21	3545	3087	3056	3056
q22	365	304	306	304
Total cold run time: 111312 ms
Total hot run time: 39512 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4057	4053	4083	4053
q2	335	217	221	217
q3	2961	2964	2957	2957
q4	1865	1876	1866	1866
q5	5210	5233	5236	5233
q6	208	126	126	126
q7	2237	1791	1821	1791
q8	3218	3302	3278	3278
q9	9090	9121	9164	9121
q10	3770	3846	3820	3820
q11	541	448	459	448
q12	721	555	543	543
q13	10272	2898	2891	2891
q14	287	258	266	258
q15	514	479	465	465
q16	463	409	407	407
q17	1707	1672	1685	1672
q18	7714	7238	7276	7238
q19	1618	1634	1620	1620
q20	1926	1740	1728	1728
q21	4999	4793	4774	4774
q22	493	417	432	417
Total cold run time: 64206 ms
Total hot run time: 54923 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (8880/24919)
Line Coverage: 27.37% (72919/266400)
Region Coverage: 26.55% (37712/142046)
Branch Coverage: 23.34% (19220/82332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1e998aad3cbdf5e73eae7e2f321ec994f8c3028a_1e998aad3cbdf5e73eae7e2f321ec994f8c3028a/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 181986 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e998aad3cbdf5e73eae7e2f321ec994f8c3028a, data reload: false

query1	1204	1127	1101	1101
query2	6474	1896	1858	1858
query3	6669	209	222	209
query4	23969	21447	21488	21447
query5	4205	423	426	423
query6	276	185	186	185
query7	4604	315	314	314
query8	233	185	199	185
query9	8468	2294	2288	2288
query10	581	264	257	257
query11	14947	14535	14465	14465
query12	161	103	97	97
query13	1646	406	389	389
query14	8622	6861	6873	6861
query15	215	176	186	176
query16	7177	281	290	281
query17	985	631	602	602
query18	1920	295	288	288
query19	207	167	168	167
query20	101	95	96	95
query21	195	131	127	127
query22	4925	4826	4810	4810
query23	33354	32738	32673	32673
query24	12482	3176	3189	3176
query25	697	415	416	415
query26	1908	169	173	169
query27	2972	333	337	333
query28	6696	1862	1846	1846
query29	1362	607	608	607
query30	305	165	167	165
query31	1000	715	735	715
query32	106	64	60	60
query33	733	271	268	268
query34	990	517	508	508
query35	839	704	710	704
query36	1003	868	881	868
query37	290	83	81	81
query38	3627	3421	3408	3408
query39	1624	1531	1525	1525
query40	295	143	143	143
query41	53	47	46	46
query42	113	103	111	103
query43	450	419	401	401
query44	1094	708	714	708
query45	280	257	260	257
query46	1079	828	803	803
query47	1855	1798	1791	1791
query48	392	306	307	306
query49	1154	363	358	358
query50	812	393	397	393
query51	6872	6657	6717	6657
query52	118	94	102	94
query53	360	298	295	295
query54	320	252	244	244
query55	90	82	85	82
query56	251	235	229	229
query57	1214	1120	1130	1120
query58	252	236	224	224
query59	2543	2462	2386	2386
query60	262	250	249	249
query61	97	93	91	91
query62	686	458	451	451
query63	319	286	282	282
query64	6321	3048	3059	3048
query65	3088	3004	2997	2997
query66	1454	340	316	316
query67	15340	14908	14932	14908
query68	5165	573	586	573
query69	500	333	331	331
query70	1166	1107	1132	1107
query71	440	287	279	279
query72	6183	2595	2572	2572
query73	731	332	330	330
query74	6654	6312	6375	6312
query75	3033	2335	2302	2302
query76	3331	1033	1218	1033
query77	443	283	264	264
query78	10803	10197	10077	10077
query79	6101	554	547	547
query80	1637	454	451	451
query81	537	249	249	249
query82	993	111	118	111
query83	272	178	171	171
query84	285	98	96	96
query85	2040	341	335	335
query86	500	316	289	289
query87	3648	3516	3518	3516
query88	4253	2364	2346	2346
query89	501	387	379	379
query90	2053	183	182	182
query91	141	108	122	108
query92	65	54	56	54
query93	5307	548	529	529
query94	1229	198	205	198
query95	423	339	333	333
query96	605	275	272	272
query97	2688	2473	2457	2457
query98	240	219	211	211
query99	1266	856	844	844
Total cold run time: 288169 ms
Total hot run time: 181986 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e998aad3cbdf5e73eae7e2f321ec994f8c3028a, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.05	0.04
query4	1.68	0.07	0.06
query5	0.48	0.48	0.50
query6	1.13	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.53
query10	0.55	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.59	0.60
query14	0.78	0.79	0.78
query15	0.86	0.83	0.84
query16	0.34	0.35	0.35
query17	1.00	0.96	0.98
query18	0.26	0.26	0.25
query19	1.81	1.75	1.75
query20	0.02	0.01	0.01
query21	15.41	0.75	0.69
query22	3.55	4.98	2.48
query23	17.74	1.35	1.01
query24	1.79	0.26	0.23
query25	0.14	0.09	0.08
query26	0.29	0.17	0.18
query27	0.09	0.10	0.08
query28	13.45	0.96	0.98
query29	12.61	3.55	3.48
query30	0.26	0.06	0.06
query31	2.88	0.39	0.39
query32	3.27	0.48	0.47
query33	2.88	2.90	2.88
query34	15.49	4.36	4.35
query35	4.39	4.39	4.38
query36	0.66	0.48	0.48
query37	0.20	0.17	0.18
query38	0.17	0.17	0.15
query39	0.04	0.03	0.04
query40	0.18	0.15	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.5 s
Total hot run time: 30.99 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1e998aad3cbdf5e73eae7e2f321ec994f8c3028a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.5 seconds inserted 10000000 Rows, about 606K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2024
Copy link
Contributor

github-actions bot commented Apr 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Apr 3, 2024

PR approved by anyone and no changes requested.

@xinyiZzz xinyiZzz merged commit 51508cf into apache:master Apr 3, 2024
26 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
ASAN compiles BE, add markers in memory logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants