Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](bdbje) add free disk config #27578

Merged
merged 3 commits into from
Nov 27, 2023
Merged

Conversation

morningman
Copy link
Contributor

Proposed changes

Add new config bdbje_free_disk_bytes, if disk space less than this config, the bdb env will not start.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morningman
Copy link
Contributor Author

run buildall

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 94a63aada2b709ce4ac59812da9d61ea8c1320a1, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4876	4678	4637	4637
q2	361	155	158	155
q3	2034	1890	1901	1890
q4	1386	1242	1247	1242
q5	3965	3970	4042	3970
q6	250	129	131	129
q7	1443	896	869	869
q8	2774	2807	2789	2789
q9	9755	9508	9538	9508
q10	3463	3509	3517	3509
q11	388	242	246	242
q12	448	285	288	285
q13	4568	3818	3826	3818
q14	319	281	299	281
q15	581	544	531	531
q16	667	577	587	577
q17	1137	1009	956	956
q18	7784	7386	7412	7386
q19	1689	1665	1697	1665
q20	542	310	287	287
q21	4400	3963	3976	3963
q22	477	376	376	376
Total cold run time: 53307 ms
Total hot run time: 49065 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4580	4548	4566	4548
q2	354	233	275	233
q3	4011	3976	3974	3974
q4	2686	2673	2663	2663
q5	9761	9719	9745	9719
q6	246	123	119	119
q7	3012	2456	2445	2445
q8	4456	4466	4504	4466
q9	12948	12786	12834	12786
q10	4082	4182	4173	4173
q11	806	644	756	644
q12	975	824	806	806
q13	4322	3614	3594	3594
q14	380	349	363	349
q15	579	519	524	519
q16	724	699	683	683
q17	3833	3876	3833	3833
q18	9384	8710	8992	8710
q19	1808	1791	1795	1791
q20	2406	2057	2038	2038
q21	8710	8298	8596	8298
q22	944	822	812	812
Total cold run time: 81007 ms
Total hot run time: 77203 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.11 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 28 seconds loaded 2358488459 Bytes, about 80 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17098774879 Bytes

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Member

@liugddx liugddx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 7ac97c1 into apache:master Nov 27, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants