[refactor](readability) Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard #30554

JackDrogon · 2024-01-30T04:26:06Z

Proposed changes

Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

JackDrogon · 2024-01-30T04:26:18Z

run buildall

github-actions · 2024-01-30T04:35:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-30T04:44:53Z

TPC-H: Total hot run time: 37389 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3e55df2966ca4e2d2da80d7af9564e2c75828a98, data reload: false

------ Round 1 ----------------------------------
q1	17956	4720	4564	4564
q2	2465	154	137	137
q3	11369	943	934	934
q4	4840	815	713	713
q5	8017	2931	2922	2922
q6	189	122	121	121
q7	1168	754	742	742
q8	9746	2018	2053	2018
q9	7202	6382	6337	6337
q10	8095	2430	2414	2414
q11	411	217	232	217
q12	779	280	287	280
q13	18007	3326	3291	3291
q14	271	259	242	242
q15	538	489	488	488
q16	471	405	410	405
q17	945	618	547	547
q18	6750	5936	5960	5936
q19	1552	1368	1342	1342
q20	618	328	346	328
q21	6947	3159	3092	3092
q22	796	329	319	319
Total cold run time: 109132 ms
Total hot run time: 37389 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4496	4460	4400	4400
q2	331	246	242	242
q3	3020	2952	2888	2888
q4	1862	1707	1663	1663
q5	5242	5260	5269	5260
q6	189	113	114	113
q7	2196	1838	1744	1744
q8	3131	3259	3258	3258
q9	8468	8278	8266	8266
q10	5852	3538	3555	3538
q11	566	474	476	474
q12	763	589	584	584
q13	6917	3143	3095	3095
q14	278	250	274	250
q15	532	497	487	487
q16	532	494	486	486
q17	1839	1684	1720	1684
q18	8060	7550	7645	7550
q19	7627	1520	1537	1520
q20	2164	1912	1915	1912
q21	4947	4687	4752	4687
q22	602	497	470	470
Total cold run time: 69614 ms
Total hot run time: 54571 ms

doris-robot · 2024-01-30T04:55:57Z

TPC-DS: Total hot run time: 172528 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3e55df2966ca4e2d2da80d7af9564e2c75828a98, data reload: false

query1	931	337	328	328
query2	6546	1980	1866	1866
query3	6703	204	198	198
query4	27955	22056	22109	22056
query5	4448	349	371	349
query6	247	154	158	154
query7	4600	267	257	257
query8	242	170	170	170
query9	9009	2240	2230	2230
query10	414	217	195	195
query11	16867	15466	15361	15361
query12	117	67	59	59
query13	1683	367	382	367
query14	9177	6475	6544	6475
query15	214	178	179	178
query16	6339	257	245	245
query17	1902	482	465	465
query18	1840	246	256	246
query19	172	130	126	126
query20	72	66	61	61
query21	196	133	129	129
query22	4889	4700	4765	4700
query23	31198	30347	30366	30347
query24	12423	2746	2792	2746
query25	593	301	306	301
query26	2084	137	140	137
query27	3123	279	283	279
query28	7669	1827	1808	1808
query29	2232	658	615	615
query30	284	137	140	137
query31	921	710	721	710
query32	93	51	49	49
query33	717	209	205	205
query34	1196	465	465	465
query35	857	778	752	752
query36	1338	1206	1169	1169
query37	177	54	61	54
query38	3332	3175	3111	3111
query39	1323	1252	1245	1245
query40	329	83	81	81
query41	39	35	35	35
query42	94	85	83	83
query43	545	479	493	479
query44	1130	668	667	667
query45	189	182	176	176
query46	1037	629	615	615
query47	1571	1516	1440	1440
query48	397	303	304	303
query49	1272	282	279	279
query50	684	320	312	312
query51	5314	5149	5211	5149
query52	92	77	92	77
query53	322	260	263	260
query54	265	187	186	186
query55	79	76	79	76
query56	178	165	156	156
query57	997	880	907	880
query58	187	150	155	150
query59	2350	2352	2490	2352
query60	204	184	184	184
query61	83	89	82	82
query62	649	370	369	369
query63	272	272	269	269
query64	5600	1782	1765	1765
query65	3322	3234	3205	3205
query66	1491	318	311	311
query67	14323	14232	13995	13995
query68	4070	493	487	487
query69	418	321	312	312
query70	1583	1530	1522	1522
query71	296	217	235	217
query72	3719	2831	2853	2831
query73	689	314	323	314
query74	6629	6161	6246	6161
query75	2839	2346	2270	2270
query76	2940	972	945	945
query77	439	235	235	235
query78	8967	8881	8563	8563
query79	2435	489	488	488
query80	2039	313	304	304
query81	524	208	201	201
query82	713	85	79	79
query83	241	117	117	117
query84	280	79	75	75
query85	2505	330	338	330
query86	512	424	383	383
query87	3437	3225	3267	3225
query88	3888	2205	2209	2205
query89	419	371	353	353
query90	2096	183	183	183
query91	142	116	118	116
query92	53	45	45	45
query93	2226	444	399	399
query94	1536	155	153	153
query95	506	454	460	454
query96	611	321	320	320
query97	4250	4109	4147	4109
query98	204	194	181	181
query99	1048	670	679	670
Total cold run time: 279392 ms
Total hot run time: 172528 ms

doris-robot · 2024-01-30T05:01:22Z

ClickBench: Total hot run time: 31.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3e55df2966ca4e2d2da80d7af9564e2c75828a98, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.68	0.10	0.09
query5	0.53	0.51	0.51
query6	1.18	0.64	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.50	0.50
query10	0.55	0.55	0.54
query11	0.11	0.08	0.09
query12	0.11	0.09	0.10
query13	0.60	0.61	0.61
query14	0.76	0.80	0.78
query15	0.79	0.79	0.78
query16	0.38	0.38	0.37
query17	1.02	1.01	1.04
query18	0.22	0.27	0.25
query19	1.81	1.80	1.80
query20	0.02	0.01	0.01
query21	15.41	0.55	0.61
query22	2.31	2.38	2.06
query23	17.16	0.94	0.86
query24	2.32	1.11	0.92
query25	0.40	0.12	0.18
query26	0.54	0.15	0.13
query27	0.05	0.05	0.05
query28	11.41	0.84	0.86
query29	12.59	3.25	3.31
query30	0.60	0.48	0.53
query31	2.78	0.35	0.34
query32	3.36	0.48	0.48
query33	3.24	3.21	3.25
query34	15.82	4.34	4.28
query35	4.28	4.30	4.37
query36	1.13	1.06	1.08
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.49 s
Total hot run time: 31.37 s

doris-robot · 2024-01-30T05:04:26Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3e55df2966ca4e2d2da80d7af9564e2c75828a98 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

doris-robot · 2024-01-30T05:14:44Z

TeamCity be ut coverage result:
Function Coverage: 36.19% (8624/23827)
Line Coverage: 28.25% (70527/249620)
Region Coverage: 27.25% (36390/133542)
Branch Coverage: 24.04% (18648/77556)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3e55df2966ca4e2d2da80d7af9564e2c75828a98_3e55df2966ca4e2d2da80d7af9564e2c75828a98/report/index.html

github-actions · 2024-01-30T06:11:59Z

PR approved by anyone and no changes requested.

zy-kkk

LGTM

github-actions · 2024-01-30T07:35:56Z

PR approved by at least one committer and no changes requested.

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

yiguolei · 2024-01-30T13:47:45Z

run buildall

github-actions · 2024-01-30T13:58:00Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-30T14:14:27Z

TeamCity be ut coverage result:
Function Coverage: 36.19% (8624/23831)
Line Coverage: 28.25% (70507/249607)
Region Coverage: 27.24% (36381/133542)
Branch Coverage: 24.03% (18630/77540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/83b94defac46ac83366e8dd2cb13807bef8e095d_83b94defac46ac83366e8dd2cb13807bef8e095d/report/index.html

doris-robot · 2024-01-30T14:18:20Z

TPC-H: Total hot run time: 36776 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83b94defac46ac83366e8dd2cb13807bef8e095d, data reload: false

------ Round 1 ----------------------------------
q1	17639	4480	4491	4480
q2	2036	141	129	129
q3	10602	902	885	885
q4	4648	730	717	717
q5	7676	2816	2813	2813
q6	181	118	117	117
q7	1119	711	695	695
q8	9249	1985	2006	1985
q9	7228	6309	6321	6309
q10	8055	2450	2389	2389
q11	408	200	198	198
q12	739	278	278	278
q13	18000	3297	3302	3297
q14	276	256	253	253
q15	527	482	496	482
q16	456	434	421	421
q17	914	579	528	528
q18	6817	5893	6005	5893
q19	1567	1381	1366	1366
q20	584	335	319	319
q21	7062	3080	2938	2938
q22	789	311	284	284
Total cold run time: 106572 ms
Total hot run time: 36776 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4408	4311	4379	4311
q2	318	227	239	227
q3	3010	2876	2858	2858
q4	1795	1663	1603	1603
q5	5195	5202	5176	5176
q6	192	112	115	112
q7	2100	1792	1741	1741
q8	3104	3201	3231	3201
q9	8338	8359	8308	8308
q10	5765	3545	3535	3535
q11	540	457	454	454
q12	732	593	566	566
q13	13104	3120	3081	3081
q14	272	249	255	249
q15	529	493	484	484
q16	561	470	497	470
q17	1833	1709	1734	1709
q18	7941	7563	7471	7471
q19	9902	1556	1547	1547
q20	2143	1912	1878	1878
q21	4724	4643	4746	4643
q22	538	481	482	481
Total cold run time: 77044 ms
Total hot run time: 54105 ms

doris-robot · 2024-01-30T14:29:12Z

TPC-DS: Total hot run time: 172237 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83b94defac46ac83366e8dd2cb13807bef8e095d, data reload: false

query1	930	328	323	323
query2	6579	2147	1872	1872
query3	6695	200	195	195
query4	27890	22078	22025	22025
query5	4449	353	334	334
query6	245	155	151	151
query7	4979	273	265	265
query8	245	192	178	178
query9	9238	2255	2259	2255
query10	412	215	214	214
query11	17450	15320	15387	15320
query12	120	66	61	61
query13	1681	371	385	371
query14	9226	6486	6388	6388
query15	215	176	179	176
query16	6388	248	257	248
query17	1891	487	460	460
query18	1901	258	256	256
query19	172	137	126	126
query20	71	73	65	65
query21	202	129	127	127
query22	4882	4664	4555	4555
query23	31157	30155	30101	30101
query24	9583	2790	2773	2773
query25	535	311	298	298
query26	826	140	137	137
query27	2829	284	277	277
query28	6189	1838	1829	1829
query29	935	619	617	617
query30	278	134	134	134
query31	922	702	726	702
query32	93	52	46	46
query33	518	206	207	206
query34	835	440	454	440
query35	844	763	756	756
query36	1232	1235	1237	1235
query37	89	59	58	58
query38	3293	3140	3115	3115
query39	1309	1250	1250	1250
query40	193	90	80	80
query41	37	37	34	34
query42	85	82	80	80
query43	552	502	485	485
query44	1074	673	675	673
query45	189	172	170	170
query46	1050	629	621	621
query47	1631	1555	1526	1526
query48	397	308	316	308
query49	1107	276	275	275
query50	685	303	299	299
query51	5320	5120	5111	5111
query52	87	81	80	80
query53	318	257	251	251
query54	242	179	176	176
query55	88	72	72	72
query56	176	161	169	161
query57	966	930	922	922
query58	180	147	155	147
query59	2419	2452	2453	2452
query60	203	176	171	171
query61	80	80	81	80
query62	619	354	337	337
query63	273	266	265	265
query64	4033	1754	1717	1717
query65	3265	3205	3210	3205
query66	1146	309	305	305
query67	14365	14120	14313	14120
query68	4287	514	509	509
query69	448	317	309	309
query70	1566	1453	1510	1453
query71	277	210	224	210
query72	3818	2806	2812	2806
query73	685	320	316	316
query74	6627	6225	6257	6225
query75	2886	2254	2327	2254
query76	2996	950	902	902
query77	372	232	221	221
query78	9062	8732	8696	8696
query79	2328	489	481	481
query80	1440	315	317	315
query81	537	195	198	195
query82	874	86	75	75
query83	248	110	122	110
query84	259	70	73	70
query85	1548	336	321	321
query86	507	386	404	386
query87	3450	3320	3218	3218
query88	3548	2138	2148	2138
query89	421	335	349	335
query90	1900	185	175	175
query91	145	116	117	116
query92	49	42	42	42
query93	1851	454	454	454
query94	1309	161	153	153
query95	498	443	445	443
query96	601	307	317	307
query97	4253	4101	4132	4101
query98	197	196	189	189
query99	1086	650	673	650
Total cold run time: 268181 ms
Total hot run time: 172237 ms

doris-robot · 2024-01-30T14:34:38Z

ClickBench: Total hot run time: 30.98 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83b94defac46ac83366e8dd2cb13807bef8e095d, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.52	0.51
query6	1.18	0.63	0.66
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.49	0.52
query10	0.55	0.56	0.53
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.59	0.59	0.60
query14	0.78	0.78	0.78
query15	0.79	0.77	0.77
query16	0.38	0.38	0.38
query17	0.99	1.03	1.00
query18	0.22	0.26	0.24
query19	1.84	1.74	1.80
query20	0.02	0.01	0.01
query21	15.43	0.56	0.56
query22	2.77	2.26	1.82
query23	17.38	0.82	0.81
query24	2.51	1.17	1.31
query25	0.30	0.31	0.07
query26	0.52	0.13	0.14
query27	0.06	0.04	0.04
query28	11.07	0.88	0.85
query29	12.59	3.11	3.10
query30	0.64	0.55	0.57
query31	2.79	0.36	0.34
query32	3.33	0.48	0.48
query33	3.18	3.25	3.16
query34	15.82	4.30	4.24
query35	4.37	4.30	4.26
query36	1.10	1.05	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.15
query41	0.07	0.02	0.02
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.94 s
Total hot run time: 30.98 s

doris-robot · 2024-01-30T14:37:44Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 83b94defac46ac83366e8dd2cb13807bef8e095d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

ByteYue approved these changes Jan 30, 2024

View reviewed changes

github-actions bot added the reviewed label Jan 30, 2024

zy-kkk approved these changes Jan 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024

yiguolei approved these changes Jan 30, 2024

View reviewed changes

Refactor MemTableFlushExecutor::create_flush_token to improve

83b94de

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

yiguolei force-pushed the refactor/memtable_create_flush_token branch from 3e55df2 to 83b94de Compare January 30, 2024 13:47

yiguolei approved these changes Jan 31, 2024

View reviewed changes

yiguolei merged commit f6f8c12 into apache:master Jan 31, 2024
24 of 26 checks passed

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

Refactor MemTableFlushExecutor::create_flush_token to improve (#30554)

e9d5de5

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

Refactor MemTableFlushExecutor::create_flush_token to improve (#30554)

49ad81e

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

Refactor MemTableFlushExecutor::create_flush_token to improve (#30554)

9eeb7dc

readability && Add default type guard Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[refactor](readability) Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard #30554

[refactor](readability) Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard #30554

JackDrogon commented Jan 30, 2024

JackDrogon commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

zy-kkk left a comment

github-actions bot commented Jan 30, 2024

yiguolei commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

[refactor](readability) Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard #30554

[refactor](readability) Refactor MemTableFlushExecutor::create_flush_token to improve readability && Add default type guard #30554

Conversation

JackDrogon commented Jan 30, 2024

Proposed changes

Further comments

JackDrogon commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

zy-kkk left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 30, 2024

yiguolei commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024