Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](partial update)add more logs for partial update #35802

Merged
merged 3 commits into from
Jun 4, 2024
Merged

[improvement](partial update)add more logs for partial update #35802

merged 3 commits into from
Jun 4, 2024

Conversation

zhannngchen
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

To make us to pinpoint the issues of partial update easier

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen
Copy link
Contributor Author

run buildall

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zhannngchen zhannngchen mentioned this pull request Jun 3, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40529 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3248f8675dbfa6faf603a8bcf0dfbdd0d7b934d8, data reload: false

------ Round 1 ----------------------------------
q1	17710	5418	4271	4271
q2	2027	191	193	191
q3	10608	1173	1132	1132
q4	10542	764	834	764
q5	7561	2725	2735	2725
q6	222	140	141	140
q7	966	621	616	616
q8	9251	2121	2116	2116
q9	9519	6635	6615	6615
q10	9040	3796	3805	3796
q11	491	250	245	245
q12	482	232	228	228
q13	17779	2998	2995	2995
q14	275	225	224	224
q15	504	465	470	465
q16	515	387	371	371
q17	971	693	751	693
q18	8138	7415	7487	7415
q19	6080	1569	1495	1495
q20	657	352	307	307
q21	4901	3864	3383	3383
q22	398	346	342	342
Total cold run time: 118637 ms
Total hot run time: 40529 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4384	4261	4288	4261
q2	374	267	277	267
q3	2996	2742	2725	2725
q4	1899	1627	1668	1627
q5	5295	5313	5258	5258
q6	221	128	129	128
q7	2150	1767	1741	1741
q8	3177	3318	3295	3295
q9	8419	8404	8329	8329
q10	3899	3634	3693	3634
q11	621	487	518	487
q12	783	609	619	609
q13	17503	3014	3058	3014
q14	284	270	258	258
q15	516	475	492	475
q16	476	419	406	406
q17	1771	1491	1440	1440
q18	7684	7560	7587	7560
q19	1843	1616	1582	1582
q20	2012	1812	1802	1802
q21	4894	4787	4740	4740
q22	681	545	569	545
Total cold run time: 71882 ms
Total hot run time: 54183 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172613 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3248f8675dbfa6faf603a8bcf0dfbdd0d7b934d8, data reload: false

query1	935	381	364	364
query2	6463	2564	2468	2468
query3	6646	207	207	207
query4	19351	17278	17394	17278
query5	4108	468	473	468
query6	245	164	162	162
query7	4609	304	296	296
query8	318	297	284	284
query9	8554	2429	2384	2384
query10	450	300	272	272
query11	10485	9931	9905	9905
query12	135	87	88	87
query13	1642	369	357	357
query14	10330	7038	7849	7038
query15	229	192	189	189
query16	7824	285	257	257
query17	1823	526	539	526
query18	1966	264	275	264
query19	212	164	163	163
query20	98	85	86	85
query21	208	137	129	129
query22	4343	4080	4070	4070
query23	33944	32931	33049	32931
query24	11189	2883	2867	2867
query25	674	381	423	381
query26	1582	156	157	156
query27	3005	320	316	316
query28	7654	2082	2060	2060
query29	1022	636	597	597
query30	273	148	151	148
query31	964	751	746	746
query32	94	53	60	53
query33	764	289	275	275
query34	1007	477	524	477
query35	748	634	620	620
query36	1110	963	921	921
query37	163	69	75	69
query38	2874	2752	2726	2726
query39	859	799	781	781
query40	241	129	128	128
query41	59	52	52	52
query42	122	100	100	100
query43	613	555	561	555
query44	1209	749	773	749
query45	201	169	169	169
query46	1073	728	724	724
query47	1836	1748	1753	1748
query48	366	295	312	295
query49	1156	423	418	418
query50	795	426	400	400
query51	6954	6860	6780	6780
query52	110	91	96	91
query53	360	300	289	289
query54	1025	456	441	441
query55	75	74	72	72
query56	305	255	283	255
query57	1141	1054	1061	1054
query58	244	255	291	255
query59	3592	3568	3369	3369
query60	307	283	277	277
query61	89	89	88	88
query62	638	460	450	450
query63	328	293	302	293
query64	9925	2218	1777	1777
query65	3181	3094	3112	3094
query66	1368	326	324	324
query67	15320	14860	15065	14860
query68	4546	531	537	531
query69	449	312	316	312
query70	1133	1179	1142	1142
query71	419	291	297	291
query72	7202	5129	4689	4689
query73	736	325	324	324
query74	6012	5761	5491	5491
query75	3590	2693	2686	2686
query76	2870	917	947	917
query77	452	301	300	300
query78	10307	9859	9800	9800
query79	2288	523	518	518
query80	1176	479	467	467
query81	586	218	215	215
query82	744	105	110	105
query83	254	180	179	179
query84	252	85	87	85
query85	2063	279	360	279
query86	486	329	331	329
query87	3291	3097	3070	3070
query88	3957	2370	2343	2343
query89	491	395	386	386
query90	1830	194	191	191
query91	129	97	103	97
query92	60	50	52	50
query93	2355	513	496	496
query94	1218	187	186	186
query95	408	315	309	309
query96	596	265	262	262
query97	3189	3019	3051	3019
query98	249	226	219	219
query99	1235	846	841	841
Total cold run time: 276106 ms
Total hot run time: 172613 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3248f8675dbfa6faf603a8bcf0dfbdd0d7b934d8, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.04	0.05
query4	1.65	0.07	0.07
query5	0.49	0.46	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.53
query11	0.16	0.11	0.12
query12	0.15	0.13	0.13
query13	0.61	0.59	0.59
query14	0.79	0.78	0.76
query15	0.82	0.79	0.82
query16	0.36	0.37	0.37
query17	1.05	0.98	1.06
query18	0.23	0.23	0.25
query19	1.82	1.80	1.81
query20	0.02	0.00	0.01
query21	15.45	0.73	0.67
query22	4.30	7.71	1.51
query23	18.30	1.35	1.17
query24	1.99	0.23	0.20
query25	0.13	0.09	0.09
query26	0.28	0.18	0.18
query27	0.08	0.07	0.08
query28	13.27	1.02	1.00
query29	12.64	3.30	3.28
query30	0.24	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.48
query33	2.88	2.82	2.90
query34	17.22	4.41	4.38
query35	4.46	4.52	4.46
query36	0.66	0.47	0.45
query37	0.19	0.16	0.16
query38	0.16	0.16	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.15
query41	0.11	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.57 s
Total hot run time: 29.99 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.29% (9246/25479)
Line Coverage: 27.65% (75863/274376)
Region Coverage: 26.88% (39297/146221)
Branch Coverage: 23.60% (19909/84370)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3248f8675dbfa6faf603a8bcf0dfbdd0d7b934d8_3248f8675dbfa6faf603a8bcf0dfbdd0d7b934d8/report/index.html

dataroaring
dataroaring approved these changes Jun 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 0ff3955 into apache:master Jun 4, 2024
26 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@zhannngchen @dataroaring
[improvement](partial update)add more logs for partial update (#35802)
74e444a 
To make us to pinpoint the issues of partial update easier
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@zhannngchen
[branch-2.1](partial update)add logs for partial update (#35416)
5541fd1 
add logs for partial update

the master PR is #35802

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@zhannngchen @seawinde
[improvement](partial update)add more logs for partial update (apache…
0d11f04 
…#35802)

To make us to pinpoint the issues of partial update easier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zhannngchen @doris-robot @Yukang-Lian @dataroaring @wm1581066