Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](partial-update) print more log while get_rowset_by_ids missed some rowsets #33711

Merged
merged 1 commit into from
Apr 18, 2024
Merged

[enhancement](partial-update) print more log while get_rowset_by_ids missed some rowsets #33711

merged 1 commit into from
Apr 18, 2024

Conversation

zhannngchen
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

get_rowset_by_ids my return less rowsets than given rowset_ids, because compaction may delete some rowsets that refered by rowset_ids.

This might cause update fail due to not found key, we print more log here for debuging

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.50% (8908/25092)
Line Coverage: 27.22% (73149/268724)
Region Coverage: 26.35% (37817/143543)
Branch Coverage: 23.13% (19272/83330)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c2ad352b0c4194eaef17349094b8788322d82706_c2ad352b0c4194eaef17349094b8788322d82706/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38331 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c2ad352b0c4194eaef17349094b8788322d82706, data reload: false

------ Round 1 ----------------------------------
q1	17587	4261	4204	4204
q2	2006	192	185	185
q3	10459	1152	1144	1144
q4	10192	810	740	740
q5	7506	2672	2669	2669
q6	220	131	134	131
q7	1006	599	581	581
q8	9211	2042	2009	2009
q9	7359	6598	6513	6513
q10	8454	3518	3555	3518
q11	455	234	224	224
q12	443	221	214	214
q13	17759	2916	2945	2916
q14	265	228	231	228
q15	527	502	489	489
q16	520	392	382	382
q17	954	656	658	656
q18	7402	6821	6662	6662
q19	7030	1436	1507	1436
q20	655	315	303	303
q21	3463	2825	2835	2825
q22	354	312	302	302
Total cold run time: 113827 ms
Total hot run time: 38331 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4163	4172	4163
q2	361	266	262	262
q3	3001	2738	2709	2709
q4	1868	1604	1587	1587
q5	5264	5317	5327	5317
q6	208	122	123	122
q7	2223	1843	1874	1843
q8	3205	3332	3329	3329
q9	8594	8511	8565	8511
q10	4104	3881	4059	3881
q11	599	500	485	485
q12	790	622	635	622
q13	16177	3301	3185	3185
q14	315	294	292	292
q15	516	479	514	479
q16	500	445	432	432
q17	1834	1566	1511	1511
q18	8166	8109	7883	7883
q19	1649	1554	1603	1554
q20	2094	1887	1883	1883
q21	5319	5133	5115	5115
q22	551	479	468	468
Total cold run time: 71663 ms
Total hot run time: 55633 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2ad352b0c4194eaef17349094b8788322d82706, data reload: false

query1	901	367	365	365
query2	6180	2779	2389	2389
query3	6671	212	198	198
query4	26229	21299	21294	21294
query5	4116	406	405	405
query6	278	174	169	169
query7	4576	287	288	287
query8	225	163	170	163
query9	8440	2431	2418	2418
query10	410	237	253	237
query11	14750	14258	14222	14222
query12	134	92	84	84
query13	1671	366	366	366
query14	8995	8065	7549	7549
query15	268	184	179	179
query16	8249	259	266	259
query17	1987	567	564	564
query18	2097	271	265	265
query19	334	148	148	148
query20	88	85	81	81
query21	190	123	120	120
query22	5047	4877	4834	4834
query23	33870	33026	33500	33026
query24	10920	3110	2974	2974
query25	605	393	383	383
query26	725	162	154	154
query27	2321	373	372	372
query28	6247	2142	2102	2102
query29	866	653	603	603
query30	257	185	180	180
query31	987	764	798	764
query32	88	53	53	53
query33	734	259	281	259
query34	1122	510	506	506
query35	876	720	701	701
query36	1109	899	937	899
query37	106	72	68	68
query38	3466	3391	3404	3391
query39	1635	1619	1566	1566
query40	175	129	129	129
query41	47	110	44	44
query42	104	95	94	94
query43	603	543	554	543
query44	1245	744	751	744
query45	291	279	267	267
query46	1111	764	753	753
query47	2078	1924	1929	1924
query48	394	306	315	306
query49	821	402	380	380
query50	791	410	403	403
query51	6873	6819	6669	6669
query52	97	93	88	88
query53	348	277	275	275
query54	298	230	220	220
query55	72	69	69	69
query56	232	215	213	213
query57	1193	1151	1104	1104
query58	217	196	194	194
query59	3361	3355	3327	3327
query60	244	229	238	229
query61	92	85	84	84
query62	609	445	435	435
query63	302	275	282	275
query64	4810	4082	3933	3933
query65	3074	3010	3029	3010
query66	752	320	333	320
query67	15335	14980	14912	14912
query68	5443	533	542	533
query69	531	295	299	295
query70	1273	1165	1135	1135
query71	453	277	272	272
query72	6444	2610	2546	2546
query73	726	327	327	327
query74	6796	6459	6425	6425
query75	3668	2663	2603	2603
query76	4017	1031	926	926
query77	582	271	274	271
query78	11076	10225	10195	10195
query79	7987	515	522	515
query80	1950	532	424	424
query81	531	239	246	239
query82	1592	99	101	99
query83	327	162	163	162
query84	258	90	86	86
query85	1328	262	254	254
query86	492	311	310	310
query87	3440	3267	3303	3267
query88	5357	2445	2458	2445
query89	486	367	380	367
query90	1925	185	181	181
query91	125	98	97	97
query92	55	45	48	45
query93	6480	512	510	510
query94	1062	185	180	180
query95	388	304	304	304
query96	632	322	265	265
query97	3157	2952	2922	2922
query98	236	216	202	202
query99	1206	872	836	836
Total cold run time: 293362 ms
Total hot run time: 184779 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2ad352b0c4194eaef17349094b8788322d82706, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.49	0.46	0.50
query6	1.48	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.54	0.57	0.54
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.58	0.59
query14	0.76	0.78	0.77
query15	0.81	0.82	0.82
query16	0.36	0.36	0.36
query17	0.99	0.96	0.95
query18	0.22	0.24	0.24
query19	1.79	1.71	1.70
query20	0.02	0.01	0.01
query21	15.41	0.66	0.65
query22	4.27	6.66	2.37
query23	18.42	1.43	1.29
query24	1.35	0.42	0.20
query25	0.15	0.07	0.09
query26	0.26	0.17	0.17
query27	0.07	0.08	0.08
query28	13.35	0.99	0.98
query29	12.65	3.21	3.21
query30	0.25	0.06	0.05
query31	2.85	0.39	0.38
query32	3.28	0.47	0.47
query33	2.89	2.81	2.82
query34	17.10	4.40	4.38
query35	4.51	4.44	4.40
query36	0.64	0.45	0.45
query37	0.19	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.03	0.04	0.04
Total cold run time: 109.17 s
Total hot run time: 30.65 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c2ad352b0c4194eaef17349094b8788322d82706 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.1 seconds inserted 10000000 Rows, about 709K ops/s

liaoxin01
liaoxin01 approved these changes Apr 17, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Apr 18, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 9d9c080 into apache:master Apr 18, 2024
26 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Apr 19, 2024
@zhannngchen @Doris-Extras
[enhancement](partial-update) print more log while missed some rowsets (
56eb5ea 
#33711)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.10-merged dev/2.1.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zhannngchen @doris-robot @liaoxin01 @dataroaring @wm1581066