Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](executor) Fe publish topic info tcp leak #29739

Merged
merged 2 commits into from
Jan 9, 2024

Conversation

whutpencil
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

After publishing topic info to be, FE did not return the thrust client correctly, resulting in a TCP connection leak.

image

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

wangbo
wangbo previously approved these changes Jan 9, 2024
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo
Copy link
Contributor

wangbo commented Jan 9, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2024
Copy link
Contributor

github-actions bot commented Jan 9, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 9, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 9, 2024
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo
Copy link
Contributor

wangbo commented Jan 9, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2024
Copy link
Contributor

github-actions bot commented Jan 9, 2024

PR approved by at least one committer and no changes requested.

@morningman morningman added dev/2.1.0 usercase Important user case type label labels Jan 9, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 39272 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a22d958879b6465a9b1a8411f8ae503ae4d3cd0f, data reload: false

------ Round 1 ----------------------------------
q1	18350	5543	5499	5499
q2	2038	158	147	147
q3	10663	1133	1203	1133
q4	10245	811	815	811
q5	7814	2932	2868	2868
q6	205	129	135	129
q7	908	513	510	510
q8	9325	2041	2000	2000
q9	6477	6437	6340	6340
q10	8202	3031	2967	2967
q11	438	218	211	211
q12	387	226	228	226
q13	18269	3497	3455	3455
q14	240	213	208	208
q15	552	518	509	509
q16	475	437	414	414
q17	922	736	673	673
q18	7258	6626	6641	6626
q19	1592	1513	1506	1506
q20	712	317	305	305
q21	2825	2403	2402	2402
q22	382	333	340	333
Total cold run time: 108279 ms
Total hot run time: 39272 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5541	5555	5472	5472
q2	351	248	248	248
q3	3365	3305	3326	3305
q4	2121	2070	2067	2067
q5	5842	5868	5852	5852
q6	206	129	124	124
q7	2321	1885	1873	1873
q8	3412	3523	3548	3523
q9	8985	8860	8880	8860
q10	3820	3867	3873	3867
q11	594	508	503	503
q12	831	631	646	631
q13	5843	3234	3252	3234
q14	301	284	265	265
q15	552	513	509	509
q16	537	490	496	490
q17	2372	2375	2365	2365
q18	8661	8622	8374	8374
q19	1554	1606	1613	1606
q20	2201	1959	1945	1945
q21	5642	5368	5412	5368
q22	596	516	524	516
Total cold run time: 65648 ms
Total hot run time: 60997 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.38 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s
storage size: 17184249609 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 31.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a22d958879b6465a9b1a8411f8ae503ae4d3cd0f, data reload: false

query1	0.07	0.06	0.06
query2	0.06	0.03	0.03
query3	0.32	0.24	0.23
query4	1.72	0.21	0.20
query5	0.53	0.52	0.52
query6	1.55	0.68	0.68
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.57	0.57
query10	0.61	0.63	0.61
query11	0.18	0.12	0.12
query12	0.16	0.12	0.12
query13	0.60	0.61	0.60
query14	0.76	0.80	0.78
query15	0.81	0.81	0.80
query16	0.35	0.36	0.35
query17	1.00	0.99	1.00
query18	0.25	0.25	0.25
query19	1.89	1.73	1.78
query20	0.01	0.01	0.02
query21	15.42	0.55	0.52
query22	2.94	3.14	1.25
query23	17.38	1.19	1.04
query24	10.01	0.80	0.32
query25	1.81	0.17	0.17
query26	0.14	0.17	0.16
query27	0.17	0.17	0.18
query28	5.75	0.92	0.85
query29	12.99	3.64	3.57
query30	0.52	0.52	0.47
query31	2.91	0.37	0.38
query32	3.31	0.50	0.48
query33	3.20	3.23	3.22
query34	16.60	4.45	4.49
query35	4.44	4.60	4.53
query36	1.15	1.08	1.08
query37	0.07	0.05	0.04
query38	0.04	0.02	0.03
query39	0.02	0.02	0.01
query40	0.17	0.13	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 110.67 s
Total hot run time: 31.58 s

@yiguolei yiguolei merged commit 791b6a9 into apache:master Jan 9, 2024
18 of 19 checks passed
yiguolei pushed a commit that referenced this pull request Jan 12, 2024
* [fix](executor) Fe publish topic info tcp leak

* enhancement
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.0 dev/3.0.0-merged reviewed usercase Important user case type label workload-group workload-schedule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants