Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](group_commit) group_commit is not support on table with property light_schema_change=false #29244

Merged
merged 2 commits into from
Dec 28, 2023
Merged

[fix](group_commit) group_commit is not support on table with property light_schema_change=false #29244

merged 2 commits into from
Dec 28, 2023

Conversation

hust-hhb
Copy link
Contributor

When light_schema_change is false, there is no column id in schema, now relaying wal is relying on column id, so group commit is not support on this scenario.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8589/23477)
Line Coverage: 28.67% (69870/243704)
Region Coverage: 27.66% (36149/130677)
Branch Coverage: 24.38% (18477/75772)
Coverage Report: http://coverage.selectdb-in.cc/coverage/faa32a4b8f583e98681d06f26327de7ce3db96ec_faa32a4b8f583e98681d06f26327de7ce3db96ec/report/index.html

@hust-hhb
Copy link
Contributor Author

run buildall

mymeiyi
mymeiyi approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@mymeiyi mymeiyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8588/23477)
Line Coverage: 28.67% (69858/243704)
Region Coverage: 27.66% (36148/130677)
Branch Coverage: 24.38% (18470/75772)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b58ccb553dcb8a6211cc342801ca8edcc34ca4b_8b58ccb553dcb8a6211cc342801ca8edcc34ca4b/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.21 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17183878659 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit faa32a4b8f583e98681d06f26327de7ce3db96ec, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4990	4657	4669	4657
q2	365	151	158	151
q3	1482	1336	1247	1247
q4	1128	947	917	917
q5	3206	3154	3182	3154
q6	255	127	127	127
q7	1048	496	512	496
q8	2268	2290	2263	2263
q9	6704	6690	6639	6639
q10	3206	3276	3265	3265
q11	336	215	208	208
q12	351	205	208	205
q13	4181	3417	3412	3412
q14	240	216	210	210
q15	567	526	521	521
q16	448	391	380	380
q17	1035	806	604	604
q18	7099	6822	7092	6822
q19	1639	1652	1636	1636
q20	521	307	315	307
q21	3197	2697	2739	2697
q22	368	309	299	299
Total cold run time: 44634 ms
Total hot run time: 40217 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4625	4600	4590	4590
q2	275	173	171	171
q3	3394	3370	3363	3363
q4	2233	2215	2218	2215
q5	5740	5750	5738	5738
q6	242	120	118	118
q7	2381	1853	1872	1853
q8	3627	3613	3618	3613
q9	8949	9022	8927	8927
q10	3826	3897	3913	3897
q11	498	378	367	367
q12	774	594	589	589
q13	3907	3195	3174	3174
q14	281	260	258	258
q15	570	514	522	514
q16	502	489	454	454
q17	1984	1953	1942	1942
q18	8592	8241	8248	8241
q19	1761	1768	1758	1758
q20	2268	1928	1932	1928
q21	6131	5787	5747	5747
q22	543	475	477	475
Total cold run time: 63103 ms
Total hot run time: 59932 ms

dataroaring
dataroaring approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 28, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 8b58ccb553dcb8a6211cc342801ca8edcc34ca4b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4996	4612	4665	4612
q2	360	154	158	154
q3	1478	1305	1220	1220
q4	1133	931	886	886
q5	3160	3156	3169	3156
q6	253	125	124	124
q7	1042	497	496	496
q8	2281	2263	2267	2263
q9	6722	6710	6687	6687
q10	3208	3295	3253	3253
q11	326	217	216	216
q12	354	199	203	199
q13	4198	3431	3389	3389
q14	247	211	213	211
q15	577	513	521	513
q16	438	383	382	382
q17	1041	757	548	548
q18	7191	6886	6936	6886
q19	1652	1650	1656	1650
q20	521	297	296	296
q21	3150	2718	2710	2710
q22	367	301	303	301
Total cold run time: 44695 ms
Total hot run time: 40152 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4580	4587	4560	4560
q2	272	164	173	164
q3	3381	3382	3378	3378
q4	2236	2210	2216	2210
q5	5724	5720	5725	5720
q6	241	119	117	117
q7	2363	1849	1856	1849
q8	3613	3626	3617	3617
q9	9055	9026	8959	8959
q10	3802	3884	3904	3884
q11	486	357	367	357
q12	768	591	597	591
q13	3878	3209	3185	3185
q14	283	258	254	254
q15	570	521	519	519
q16	496	455	453	453
q17	1986	1963	1956	1956
q18	8644	8109	8120	8109
q19	1748	1763	1748	1748
q20	2256	1935	1911	1911
q21	6118	5763	5747	5747
q22	565	460	472	460
Total cold run time: 63065 ms
Total hot run time: 59748 ms

@dataroaring dataroaring merged commit 99a1e06 into apache:master Dec 28, 2023
16 of 18 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@hust-hhb @HappenLee
[fix](group_commit) group_commit is not support on table with propert…
3a27bd6 
…y light_schema_change=false (apache#29244)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mymeiyi mymeiyi mymeiyi approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hust-hhb @doris-robot @mymeiyi @dataroaring