Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Pipeline](load) fix may oom in pipeline load #27714

Merged
merged 1 commit into from
Nov 29, 2023
Merged

Conversation

HappenLee
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@kaijchen kaijchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 11c73d5182ca35a9e179e4fc70b1bbf6e2225ac6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4909	4666	4660	4660
q2	358	166	158	158
q3	1517	1322	1274	1274
q4	1155	1001	972	972
q5	3263	3214	3230	3214
q6	258	137	133	133
q7	1016	521	546	521
q8	2238	2229	2228	2228
q9	6947	6923	6958	6923
q10	3295	3377	3346	3346
q11	344	214	209	209
q12	356	216	219	216
q13	4652	3860	3842	3842
q14	245	211	215	211
q15	592	541	524	524
q16	439	379	411	379
q17	1029	651	607	607
q18	7924	7546	7964	7546
q19	1559	1517	1530	1517
q20	594	330	319	319
q21	3403	2993	2996	2993
q22	370	295	302	295
Total cold run time: 46463 ms
Total hot run time: 42087 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4598	4590	4572	4572
q2	319	220	209	209
q3	3735	3740	3739	3739
q4	2537	2531	2520	2520
q5	6203	6186	6171	6171
q6	250	122	124	122
q7	2570	1980	1977	1977
q8	3758	3680	3697	3680
q9	9435	9356	9401	9356
q10	4069	4145	4147	4145
q11	643	529	503	503
q12	804	616	660	616
q13	4371	3636	3654	3636
q14	286	250	254	250
q15	581	513	532	513
q16	508	442	503	442
q17	2076	2075	2099	2075
q18	9441	8814	9018	8814
q19	1794	1768	1761	1761
q20	2327	1995	1954	1954
q21	7367	6940	7072	6940
q22	684	570	554	554
Total cold run time: 68356 ms
Total hot run time: 64549 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.85 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17101030398 Bytes

@HappenLee HappenLee merged commit 6371dba into apache:master Nov 29, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants