Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](index compaction)support compact multi segments in one index #28889

Merged
merged 4 commits into from
Dec 28, 2023
Merged

[fix](index compaction)support compact multi segments in one index #28889

merged 4 commits into from
Dec 28, 2023

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Dec 22, 2023

Proposed changes

pick from branch-2.0-var #28545

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@qidaye
Copy link
Contributor Author

qidaye commented Dec 22, 2023

run buildall

zzzxl1993
zzzxl1993 approved these changes Dec 22, 2023
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 91cd6e3bc3283b52e5cfd297171a2febecddf6f8, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4672	4400	4441	4400
q2	366	148	157	148
q3	1460	1295	1250	1250
q4	1115	930	918	918
q5	3158	3169	3174	3169
q6	246	131	130	130
q7	998	502	480	480
q8	2157	2221	2178	2178
q9	6666	6687	6693	6687
q10	3219	3273	3250	3250
q11	306	197	182	182
q12	354	212	213	212
q13	4542	3837	3816	3816
q14	242	213	214	213
q15	572	540	537	537
q16	439	388	386	386
q17	1009	661	572	572
q18	7056	6834	6801	6801
q19	1518	1423	1407	1407
q20	510	323	327	323
q21	3073	2644	2687	2644
q22	360	284	288	284
Total cold run time: 44038 ms
Total hot run time: 39987 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4373	4342	4326	4326
q2	270	166	173	166
q3	3507	3517	3507	3507
q4	2388	2375	2377	2375
q5	5732	5737	5716	5716
q6	239	124	123	123
q7	2367	1845	1830	1830
q8	3517	3514	3515	3514
q9	9078	9029	9006	9006
q10	3917	3994	4016	3994
q11	481	375	363	363
q12	771	601	601	601
q13	4271	3585	3559	3559
q14	289	268	256	256
q15	572	528	533	528
q16	511	461	482	461
q17	1882	1853	1836	1836
q18	8604	8063	8100	8063
q19	1731	1744	1740	1740
q20	2264	1950	1945	1945
q21	6523	6139	6172	6139
q22	505	430	419	419
Total cold run time: 63792 ms
Total hot run time: 60467 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.26 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183520649 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.52% (8538/23376)
Line Coverage: 28.61% (69419/242633)
Region Coverage: 27.64% (35928/129988)
Branch Coverage: 24.36% (18362/75364)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91cd6e3bc3283b52e5cfd297171a2febecddf6f8_91cd6e3bc3283b52e5cfd297171a2febecddf6f8/report/index.html

@xiaokang
Copy link
Contributor

add testcase

airborne12
airborne12 approved these changes Dec 23, 2023
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@qidaye qidaye force-pushed the index_compaction_core branch from affda27 to 6b6016e Compare December 28, 2023 09:38
@qidaye
Copy link
Contributor Author

qidaye commented Dec 28, 2023

run buildall

@qidaye
Copy link
Contributor Author

qidaye commented Dec 28, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.59% (8590/23477)
Line Coverage: 28.67% (69879/243730)
Region Coverage: 27.67% (36163/130686)
Branch Coverage: 24.38% (18474/75778)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d9a33cc21926f13778da01a336b4116c44126b59_d9a33cc21926f13778da01a336b4116c44126b59/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17183807683 Bytes

zzzxl1993
zzzxl1993 approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit d9a33cc21926f13778da01a336b4116c44126b59, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4963	4638	4630	4630
q2	359	142	140	140
q3	1479	1318	1199	1199
q4	1138	943	931	931
q5	3201	3158	3163	3158
q6	250	127	127	127
q7	1009	491	492	491
q8	2279	2264	2254	2254
q9	6739	6701	6717	6701
q10	3217	3271	3251	3251
q11	333	209	195	195
q12	350	209	209	209
q13	4159	3447	3393	3393
q14	242	213	214	213
q15	572	526	526	526
q16	441	385	374	374
q17	1053	771	572	572
q18	7026	6990	6769	6769
q19	1632	1648	1653	1648
q20	533	310	296	296
q21	3179	2721	2658	2658
q22	382	300	303	300
Total cold run time: 44536 ms
Total hot run time: 40035 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4565	4560	4550	4550
q2	270	166	165	165
q3	3372	3367	3346	3346
q4	2230	2214	2210	2210
q5	5753	5764	5755	5755
q6	241	116	119	116
q7	2387	1874	1888	1874
q8	3619	3631	3608	3608
q9	9016	8973	8970	8970
q10	3779	3899	3902	3899
q11	493	359	370	359
q12	774	608	602	602
q13	3912	3195	3194	3194
q14	292	253	255	253
q15	571	519	512	512
q16	497	450	447	447
q17	1972	1957	1976	1957
q18	8710	8266	8201	8201
q19	1782	1756	1770	1756
q20	2256	1927	1918	1918
q21	6158	5783	5760	5760
q22	548	448	441	441
Total cold run time: 63197 ms
Total hot run time: 59893 ms

xiaokang
xiaokang approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit e64c568 into apache:master Dec 28, 2023
17 of 18 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 28, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@qidaye qidaye deleted the index_compaction_core branch December 28, 2023 13:35
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Dec 29, 2023
@qidaye
[fix](index compaction)support compact multi segments in one index (a…
8c5c1e0 
…pache#28889)
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Dec 29, 2023
@qidaye
[fix](index compaction)support compact multi segments in one index (a…
20b121e 
…pache#28889)
@qidaye qidaye mentioned this pull request Dec 29, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 29, 2023
@qidaye
[fix](index compaction)support compact multi segments in one index #2…
6495eb2 
…8889 (#29276)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@qidaye @HappenLee
[fix](index compaction)support compact multi segments in one index (a…
41d174f 
…pache#28889)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@qidaye @doris-robot @xiaokang @airborne12 @zzzxl1993