Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](log) fix group commit warn log #42403

Merged
merged 1 commit into from
Oct 28, 2024
Merged

[fix](log) fix group commit warn log #42403

merged 1 commit into from
Oct 28, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Oct 24, 2024 •
edited
Loading

if stream load in group commit mode connect to be directly, the fe does not select backend for it. the map used to record <table, be, dataSize> in fe is empty, so there are too many warn log:

[GroupCommitManager.updateLoadDataInternal():399] can not find backend id: 12229430
2024-10-21 08:58:28,975 WARN (thrift-server-pool-7|132)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9b14585e73951196265e3b726947e1f34d20a90, data reload: false

------ Round 1 ----------------------------------
q1	17607	7416	7263	7263
q2	2012	288	299	288
q3	12071	1081	1106	1081
q4	10578	816	880	816
q5	7772	3078	3049	3049
q6	238	153	153	153
q7	1022	616	595	595
q8	9352	1957	1872	1872
q9	6639	6445	6409	6409
q10	7050	2420	2482	2420
q11	440	241	247	241
q12	405	221	225	221
q13	17783	3003	3005	3003
q14	249	218	210	210
q15	582	522	513	513
q16	653	578	577	577
q17	977	517	567	517
q18	7315	6647	6792	6647
q19	1358	1007	989	989
q20	467	191	188	188
q21	4018	3076	2945	2945
q22	1106	1023	1005	1005
Total cold run time: 109694 ms
Total hot run time: 41002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7282	7279	7337	7279
q2	331	231	231	231
q3	3048	2982	2919	2919
q4	2101	1873	1772	1772
q5	5757	5718	5769	5718
q6	235	147	145	145
q7	2271	1850	1827	1827
q8	3401	3587	3421	3421
q9	8913	8901	8918	8901
q10	3618	3582	3605	3582
q11	591	478	498	478
q12	845	632	673	632
q13	10491	3197	3189	3189
q14	311	269	277	269
q15	590	534	526	526
q16	703	642	648	642
q17	1844	1611	1612	1611
q18	8319	7987	7552	7552
q19	1734	1489	1535	1489
q20	2141	1867	1899	1867
q21	5694	5489	5487	5487
q22	1129	1067	1052	1052
Total cold run time: 71349 ms
Total hot run time: 60589 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193121 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9b14585e73951196265e3b726947e1f34d20a90, data reload: false

query1	846	404	413	404
query2	6295	2087	2060	2060
query3	8684	194	211	194
query4	35312	23620	23704	23620
query5	4905	476	462	462
query6	286	183	194	183
query7	4194	305	301	301
query8	299	232	238	232
query9	9621	2732	2742	2732
query10	491	262	273	262
query11	18226	15344	15267	15267
query12	149	102	112	102
query13	1553	426	438	426
query14	9707	7432	7246	7246
query15	288	175	179	175
query16	8005	474	494	474
query17	1641	595	565	565
query18	2146	347	333	333
query19	371	167	156	156
query20	128	119	121	119
query21	213	111	117	111
query22	4980	4405	4610	4405
query23	35048	34366	34149	34149
query24	11042	2832	2776	2776
query25	592	406	408	406
query26	754	158	160	158
query27	2123	282	281	281
query28	6810	2455	2412	2412
query29	736	427	426	426
query30	274	166	167	166
query31	1029	802	810	802
query32	96	58	57	57
query33	775	306	298	298
query34	935	536	521	521
query35	1049	920	923	920
query36	1108	926	920	920
query37	149	87	87	87
query38	4381	4248	4286	4248
query39	1497	1522	1414	1414
query40	201	101	101	101
query41	49	47	51	47
query42	121	101	102	101
query43	527	493	492	492
query44	1280	815	817	815
query45	198	169	170	169
query46	1148	694	708	694
query47	1920	1839	1825	1825
query48	423	316	323	316
query49	896	438	418	418
query50	813	401	388	388
query51	7083	7084	7060	7060
query52	96	91	88	88
query53	249	177	175	175
query54	1148	420	419	419
query55	78	72	84	72
query56	275	266	260	260
query57	1316	1159	1163	1159
query58	234	228	243	228
query59	3111	3091	2932	2932
query60	292	271	282	271
query61	105	99	105	99
query62	847	665	666	665
query63	224	193	183	183
query64	3634	646	667	646
query65	3293	3187	3181	3181
query66	804	303	307	303
query67	15990	15887	15710	15710
query68	4374	568	548	548
query69	544	300	305	300
query70	1179	1155	1167	1155
query71	369	274	286	274
query72	7476	4036	4027	4027
query73	779	355	367	355
query74	9483	8985	8996	8985
query75	3563	2753	2735	2735
query76	2929	973	943	943
query77	647	293	312	293
query78	10671	9619	9638	9619
query79	3105	614	609	609
query80	1624	443	467	443
query81	595	237	235	235
query82	715	142	140	140
query83	313	138	134	134
query84	281	71	70	70
query85	1244	297	280	280
query86	390	295	301	295
query87	4736	4730	4664	4664
query88	3649	2233	2192	2192
query89	410	287	281	281
query90	1941	189	191	189
query91	137	105	101	101
query92	58	49	48	48
query93	1823	527	531	527
query94	997	298	298	298
query95	355	254	249	249
query96	617	284	292	284
query97	2893	2696	2680	2680
query98	212	204	192	192
query99	1766	1289	1312	1289
Total cold run time: 302968 ms
Total hot run time: 193121 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9b14585e73951196265e3b726947e1f34d20a90, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.11	0.10
query5	0.43	0.41	0.41
query6	1.14	0.66	0.66
query7	0.02	0.02	0.03
query8	0.04	0.03	0.04
query9	0.55	0.50	0.50
query10	0.55	0.54	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.62	0.60
query14	2.69	2.86	2.72
query15	0.90	0.85	0.83
query16	0.39	0.39	0.39
query17	1.08	0.98	1.03
query18	0.20	0.19	0.20
query19	1.94	1.90	1.92
query20	0.01	0.01	0.01
query21	15.37	0.58	0.58
query22	2.80	1.91	1.87
query23	17.03	1.10	0.78
query24	3.16	0.79	0.47
query25	0.14	0.12	0.17
query26	0.49	0.15	0.15
query27	0.04	0.04	0.04
query28	11.24	1.09	1.06
query29	12.54	3.30	3.30
query30	0.26	0.06	0.05
query31	2.88	0.38	0.39
query32	3.25	0.47	0.46
query33	3.03	2.99	3.09
query34	16.88	4.42	4.50
query35	4.51	4.54	4.48
query36	0.68	0.47	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 107.55 s
Total hot run time: 32.14 s

dataroaring
dataroaring approved these changes Oct 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@CalvinKirs CalvinKirs merged commit bb3712f into apache:master Oct 28, 2024
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 28, 2024
@mymeiyi
[fix](log) fix group commit warn log (#42403)
3727f6e 
if stream load in group commit mode connect to be directly, the fe does
not select backend for it. the map used to record <table, be, dataSize>
in fe is empty, so there are too many warn log:
```
[GroupCommitManager.updateLoadDataInternal():399] can not find backend id: 12229430
2024-10-21 08:58:28,975 WARN (thrift-server-pool-7|132)
```
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Oct 28, 2024
@mymeiyi
[fix](log) fix group commit warn log (apache#42403)
af5b628 
if stream load in group commit mode connect to be directly, the fe does
not select backend for it. the map used to record <table, be, dataSize>
in fe is empty, so there are too many warn log:
```
[GroupCommitManager.updateLoadDataInternal():399] can not find backend id: 12229430
2024-10-21 08:58:28,975 WARN (thrift-server-pool-7|132)
```
@mymeiyi mymeiyi mentioned this pull request Oct 28, 2024
Merged
dataroaring pushed a commit that referenced this pull request Oct 28, 2024
@mymeiyi
[fix](log) fix group commit warn log (#42403) (#42575)
73621c4 
pick #42403
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
@mymeiyi @dataroaring
[fix](log) fix group commit warn log (#42403)
f491436 
if stream load in group commit mode connect to be directly, the fe does
not select backend for it. the map used to record <table, be, dataSize>
in fe is empty, so there are too many warn log:
```
[GroupCommitManager.updateLoadDataInternal():399] can not find backend id: 12229430
2024-10-21 08:58:28,975 WARN (thrift-server-pool-7|132)
```
@gavinchou gavinchou added the cir label Nov 11, 2024
@mymeiyi mymeiyi mentioned this pull request Nov 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. cir dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mymeiyi @doris-robot @CalvinKirs @dataroaring @gavinchou