Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[log](mow) reduce log for data load on mow table #47647

Merged
merged 4 commits into from
Feb 10, 2025
Merged

[log](mow) reduce log for data load on mow table #47647

merged 4 commits into from
Feb 10, 2025

Conversation

zhannngchen
Copy link
Contributor

@zhannngchen zhannngchen commented Feb 8, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

reduce some useless logs according to the frequency stats from online user logs

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

dataroaring
dataroaring previously approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhannngchen
Copy link
Contributor Author

run buildall

1 similar comment
@dataroaring
Copy link
Contributor

run buildall

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@dataroaring
Copy link
Contributor

run buildall

@zhannngchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31390 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 158b49c3484ded7eb020f6587233a73ffecf9cfb, data reload: false

------ Round 1 ----------------------------------
q1	17601	5373	5052	5052
q2	2040	301	169	169
q3	10402	1279	688	688
q4	10224	1007	557	557
q5	7559	2394	2383	2383
q6	185	165	137	137
q7	911	745	614	614
q8	9290	1265	1006	1006
q9	5168	4772	4607	4607
q10	6832	2321	1893	1893
q11	500	287	251	251
q12	343	351	220	220
q13	17769	3714	3051	3051
q14	229	234	205	205
q15	508	465	469	465
q16	623	617	577	577
q17	569	870	332	332
q18	6857	6194	6297	6194
q19	1220	952	540	540
q20	318	328	205	205
q21	2862	2282	1935	1935
q22	360	334	309	309
Total cold run time: 102370 ms
Total hot run time: 31390 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5157	5143	5168	5143
q2	244	333	230	230
q3	2179	2732	2338	2338
q4	1515	1894	1454	1454
q5	4266	4159	4201	4159
q6	211	168	121	121
q7	1887	1829	1803	1803
q8	2636	2830	2635	2635
q9	7204	7090	7159	7090
q10	3051	3238	2758	2758
q11	590	518	499	499
q12	679	761	640	640
q13	3580	3973	3330	3330
q14	277	321	277	277
q15	496	472	473	472
q16	633	680	639	639
q17	1142	1648	1312	1312
q18	7592	7377	7231	7231
q19	781	798	846	798
q20	2037	2044	1885	1885
q21	5581	5052	4924	4924
q22	644	598	559	559
Total cold run time: 52382 ms
Total hot run time: 50297 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183472 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 158b49c3484ded7eb020f6587233a73ffecf9cfb, data reload: false

query1	973	375	375	375
query2	6515	1867	1804	1804
query3	6788	213	210	210
query4	26715	23771	23041	23041
query5	4349	652	472	472
query6	299	193	186	186
query7	4610	493	305	305
query8	298	246	231	231
query9	8638	2503	2496	2496
query10	443	328	271	271
query11	15868	15151	14938	14938
query12	156	106	104	104
query13	1656	522	387	387
query14	9197	6933	6176	6176
query15	214	197	189	189
query16	7258	607	430	430
query17	1214	721	561	561
query18	1993	408	307	307
query19	204	186	174	174
query20	127	123	119	119
query21	215	127	105	105
query22	4352	4333	4186	4186
query23	34028	33088	33087	33087
query24	7653	2399	2401	2399
query25	534	443	373	373
query26	1263	267	153	153
query27	2122	492	333	333
query28	3904	2403	2409	2403
query29	777	547	408	408
query30	253	184	166	166
query31	955	901	742	742
query32	74	64	64	64
query33	576	345	291	291
query34	792	860	507	507
query35	798	826	724	724
query36	940	1006	874	874
query37	122	122	81	81
query38	4097	4097	4107	4097
query39	1437	1393	1469	1393
query40	219	122	108	108
query41	54	54	50	50
query42	121	103	103	103
query43	489	500	454	454
query44	1275	792	793	792
query45	180	166	168	166
query46	862	1038	654	654
query47	1761	1771	1702	1702
query48	366	406	299	299
query49	813	505	436	436
query50	670	750	399	399
query51	4133	4178	4130	4130
query52	107	102	96	96
query53	222	249	191	191
query54	493	463	408	408
query55	79	80	80	80
query56	272	273	275	273
query57	1158	1131	1082	1082
query58	242	233	238	233
query59	2679	2675	2536	2536
query60	280	279	248	248
query61	121	126	119	119
query62	813	715	662	662
query63	217	189	188	188
query64	4434	987	735	735
query65	3245	3191	3133	3133
query66	1152	402	332	332
query67	15753	15727	15458	15458
query68	6888	778	505	505
query69	480	292	267	267
query70	1199	1128	1126	1126
query71	404	286	279	279
query72	6101	3603	3784	3603
query73	768	751	350	350
query74	9239	8942	9057	8942
query75	3140	3162	2697	2697
query76	3212	1164	735	735
query77	470	361	278	278
query78	9945	10149	9267	9267
query79	2364	794	597	597
query80	607	589	455	455
query81	482	273	241	241
query82	638	149	117	117
query83	186	173	155	155
query84	227	92	70	70
query85	754	340	305	305
query86	344	294	299	294
query87	4441	4453	4606	4453
query88	3600	2268	2221	2221
query89	394	315	287	287
query90	1927	198	189	189
query91	142	140	113	113
query92	70	58	55	55
query93	1742	1014	579	579
query94	697	418	297	297
query95	346	265	261	261
query96	484	548	274	274
query97	2792	2865	2725	2725
query98	241	214	204	204
query99	1347	1404	1302	1302
Total cold run time: 270288 ms
Total hot run time: 183472 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 158b49c3484ded7eb020f6587233a73ffecf9cfb, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.09
query5	0.40	0.41	0.40
query6	1.16	0.67	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.57	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.78	2.74	2.82
query15	0.93	0.85	0.85
query16	0.38	0.38	0.39
query17	1.00	1.02	1.04
query18	0.21	0.19	0.19
query19	1.94	1.78	2.01
query20	0.01	0.02	0.01
query21	15.38	0.87	0.53
query22	0.73	1.21	0.67
query23	14.92	1.35	0.61
query24	6.75	1.50	1.43
query25	0.54	0.16	0.08
query26	0.50	0.17	0.14
query27	0.05	0.05	0.05
query28	10.41	0.88	0.41
query29	12.58	3.95	3.28
query30	0.25	0.09	0.06
query31	2.83	0.58	0.39
query32	3.22	0.55	0.46
query33	2.95	3.02	2.99
query34	15.69	5.20	4.55
query35	4.59	4.58	4.55
query36	0.65	0.53	0.47
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.44 s
Total hot run time: 31.17 s

dataroaring
dataroaring approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

hust-hhb
hust-hhb approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@hust-hhb hust-hhb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1dcf4d1 into apache:master Feb 10, 2025
27 of 31 checks passed
zhannngchen added a commit to zhannngchen/incubator-doris that referenced this pull request Feb 10, 2025
@zhannngchen
[log](mow) reduce log for data load on mow table (apache#47647)
f1e1700 
reduce some useless logs according to the frequency stats from online
user logs.
@zhannngchen zhannngchen mentioned this pull request Feb 10, 2025
Merged
zhannngchen added a commit to zhannngchen/incubator-doris that referenced this pull request Feb 11, 2025
@zhannngchen
[log](mow) reduce log for data load on mow table (apache#47647)
e145ce7
dataroaring pushed a commit that referenced this pull request Feb 11, 2025
@zhannngchen
branch-3.0: [log](mow) reduce log for data load on mow table (#47647) (…
3a7d873 
…#47725)

cherry-pick #47647
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@zhannngchen @lzyy2024
[log](mow) reduce log for data load on mow table (apache#47647)
58919d6 
reduce some useless logs according to the frequency stats from online
user logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hust-hhb hust-hhb hust-hhb approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zhannngchen @hello-stephen @dataroaring @doris-robot @hust-hhb @wm1581066