Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction #48241

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Feb 24, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 force-pushed the config-prune-delete-sign-in-base-compaction branch from 728d234 to 4113ad7 Compare February 24, 2025 10:23
@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 24, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31510 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

------ Round 1 ----------------------------------
q1	17602	5393	5098	5098
q2	2050	303	163	163
q3	10480	2380	758	758
q4	10229	1045	519	519
q5	7585	2492	2301	2301
q6	195	169	135	135
q7	908	768	613	613
q8	9303	1384	1149	1149
q9	4893	4618	4541	4541
q10	6829	2323	1877	1877
q11	481	276	256	256
q12	345	357	218	218
q13	17761	3673	3079	3079
q14	229	233	211	211
q15	503	470	461	461
q16	624	610	582	582
q17	579	867	336	336
q18	6828	6260	6234	6234
q19	1208	970	543	543
q20	305	324	190	190
q21	2873	2141	1943	1943
q22	369	330	303	303
Total cold run time: 102179 ms
Total hot run time: 31510 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5136	5168	5150	5150
q2	230	335	227	227
q3	2184	2695	2279	2279
q4	1390	1807	1363	1363
q5	4208	4117	4158	4117
q6	216	164	123	123
q7	1912	1818	1800	1800
q8	2644	2637	2604	2604
q9	7269	7015	7129	7015
q10	2979	3205	2752	2752
q11	584	503	485	485
q12	669	776	647	647
q13	3492	3897	3233	3233
q14	271	310	271	271
q15	514	461	465	461
q16	644	689	651	651
q17	1169	1610	1341	1341
q18	7521	7328	7358	7328
q19	803	797	846	797
q20	2006	2074	1890	1890
q21	5372	4992	4848	4848
q22	649	573	537	537
Total cold run time: 51862 ms
Total hot run time: 49919 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183137 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

query1	960	391	369	369
query2	6524	1869	1836	1836
query3	6796	222	215	215
query4	26570	23728	23204	23204
query5	4337	659	501	501
query6	305	204	183	183
query7	4611	502	303	303
query8	291	242	234	234
query9	8642	2575	2570	2570
query10	454	306	258	258
query11	15446	15037	14881	14881
query12	159	106	101	101
query13	1680	548	403	403
query14	8976	6075	6206	6075
query15	214	197	178	178
query16	7230	627	453	453
query17	1240	683	555	555
query18	1938	394	287	287
query19	181	192	168	168
query20	122	145	115	115
query21	202	119	100	100
query22	4280	4341	4152	4152
query23	34014	32881	32866	32866
query24	8444	2398	2377	2377
query25	513	446	379	379
query26	1238	271	146	146
query27	2717	474	333	333
query28	4322	2416	2409	2409
query29	705	555	414	414
query30	227	183	153	153
query31	906	859	760	760
query32	72	63	61	61
query33	567	360	305	305
query34	766	846	506	506
query35	802	814	721	721
query36	965	949	887	887
query37	115	99	101	99
query38	4072	4195	4028	4028
query39	1503	1389	1390	1389
query40	208	117	97	97
query41	53	51	54	51
query42	125	111	106	106
query43	503	488	478	478
query44	1278	799	783	783
query45	174	193	161	161
query46	855	1021	640	640
query47	1722	1759	1672	1672
query48	389	407	303	303
query49	773	504	415	415
query50	695	719	418	418
query51	4162	4183	4223	4183
query52	108	104	92	92
query53	228	249	188	188
query54	476	485	399	399
query55	79	85	81	81
query56	278	248	270	248
query57	1109	1156	1042	1042
query58	242	240	232	232
query59	2602	2807	2517	2517
query60	289	297	258	258
query61	125	119	114	114
query62	780	722	657	657
query63	233	186	189	186
query64	4261	1075	753	753
query65	3228	3151	3148	3148
query66	1144	433	339	339
query67	16009	15457	15381	15381
query68	5575	790	514	514
query69	483	293	260	260
query70	1217	1151	1069	1069
query71	396	291	259	259
query72	5756	3604	3927	3604
query73	682	735	356	356
query74	9546	9133	9258	9133
query75	3147	3179	2736	2736
query76	3189	1161	738	738
query77	507	369	328	328
query78	9931	10004	9270	9270
query79	2240	843	600	600
query80	601	529	454	454
query81	502	290	242	242
query82	465	125	93	93
query83	162	173	154	154
query84	237	94	73	73
query85	783	339	297	297
query86	379	308	284	284
query87	4483	4512	4319	4319
query88	4214	2190	2198	2190
query89	395	321	282	282
query90	1855	200	189	189
query91	132	139	104	104
query92	75	58	54	54
query93	1867	1013	573	573
query94	700	414	304	304
query95	343	268	254	254
query96	506	543	275	275
query97	2776	2784	2732	2732
query98	238	207	196	196
query99	1470	1430	1285	1285
Total cold run time: 270086 ms
Total hot run time: 183137 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.41	0.41	0.40
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.53	0.52
query10	0.57	0.57	0.58
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.73	2.69	2.75
query15	0.93	0.84	0.85
query16	0.38	0.38	0.38
query17	1.01	1.04	1.04
query18	0.20	0.19	0.20
query19	1.93	1.77	1.98
query20	0.02	0.01	0.02
query21	15.35	0.91	0.55
query22	0.75	1.13	0.61
query23	15.04	1.36	0.65
query24	7.47	1.46	0.56
query25	0.54	0.11	0.23
query26	0.53	0.16	0.13
query27	0.05	0.05	0.05
query28	9.67	0.82	0.44
query29	12.53	4.01	3.28
query30	0.25	0.08	0.06
query31	2.85	0.57	0.38
query32	3.22	0.55	0.47
query33	2.98	3.03	3.05
query34	15.77	5.08	4.51
query35	4.56	4.53	4.52
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.55 s
Total hot run time: 30.3 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 44.51% (11753/26406)
Line Coverage: 34.01% (98270/288948)
Region Coverage: 33.09% (50325/152098)
Branch Coverage: 28.73% (25298/88042)
Coverage Report: http://coverage.selectdb-in.cc/coverage/24376acd8325eb8087eb5131569ac9513a604ccc_24376acd8325eb8087eb5131569ac9513a604ccc/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants