Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](snapshot) Link binlog files according to consistent rowsets #41319

Merged
merged 1 commit into from
Sep 26, 2024
Merged

[fix](snapshot) Link binlog files according to consistent rowsets #41319

merged 1 commit into from
Sep 26, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Sep 26, 2024

In previous implementations, binlog files would be linked according to making snapshot requests. However, sometimes not all requests can be executed directly. For example, when a certain version in missing_version does not exist, it will fallback to the full snapshot. Therefore, it is correct to link binlog files according to consistent rowsets.

@w41ter
[fix](snapshot) Link binlog files according to consistent rowsets
4254a77 
In previous implementations, binlog files would be linked according to
making snapshot request. However, sometimes not all requests can be
executed directly. For example, when a certain version in
missing_version does not exist, it will fallback to full snapshot.
Therefore, it is correct to link binlog files according to consistent
rowsets.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Sep 26, 2024

run buildall

dataroaring
dataroaring approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41077 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4254a77c40d6f3502099637c4a43c9ab9ed53990, data reload: false

------ Round 1 ----------------------------------
q1	17910	7477	7286	7286
q2	2049	281	292	281
q3	12211	1091	1202	1091
q4	10563	789	763	763
q5	7772	2954	2900	2900
q6	241	157	150	150
q7	1008	625	622	622
q8	9397	1984	1957	1957
q9	6604	6489	6434	6434
q10	7023	2251	2363	2251
q11	436	247	259	247
q12	447	226	224	224
q13	17772	3048	3006	3006
q14	238	213	211	211
q15	567	539	527	527
q16	668	610	606	606
q17	1006	619	616	616
q18	7483	6811	6615	6615
q19	1409	1032	994	994
q20	490	209	200	200
q21	4060	3235	3132	3132
q22	1101	964	990	964
Total cold run time: 110455 ms
Total hot run time: 41077 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7258	7493	7719	7493
q2	343	239	231	231
q3	3101	3003	2981	2981
q4	2166	1860	1818	1818
q5	5709	5670	5835	5670
q6	237	148	147	147
q7	2252	1809	1822	1809
q8	3380	3578	3587	3578
q9	9055	8944	8895	8895
q10	3622	3575	3558	3558
q11	587	505	529	505
q12	823	641	645	641
q13	9684	3190	3170	3170
q14	318	280	277	277
q15	580	525	515	515
q16	737	665	658	658
q17	1845	1630	1619	1619
q18	8402	7766	7751	7751
q19	1721	1481	1392	1392
q20	2133	1914	1897	1897
q21	5412	5506	5439	5439
q22	1171	1024	1035	1024
Total cold run time: 70536 ms
Total hot run time: 61068 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9625/25813)
Line Coverage: 28.71% (79691/277561)
Region Coverage: 28.12% (41189/146467)
Branch Coverage: 24.76% (20981/84752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4254a77c40d6f3502099637c4a43c9ab9ed53990_4254a77c40d6f3502099637c4a43c9ab9ed53990/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192346 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4254a77c40d6f3502099637c4a43c9ab9ed53990, data reload: false

query1	954	396	381	381
query2	6374	2088	2053	2053
query3	8685	199	202	199
query4	33932	23599	23441	23441
query5	3442	477	469	469
query6	284	182	184	182
query7	4200	319	329	319
query8	297	227	234	227
query9	9416	2674	2673	2673
query10	466	286	282	282
query11	17980	15298	15304	15298
query12	152	101	98	98
query13	1528	433	422	422
query14	8827	7086	7201	7086
query15	236	172	177	172
query16	7894	497	481	481
query17	1619	601	576	576
query18	2093	311	312	311
query19	255	153	148	148
query20	120	122	111	111
query21	205	108	106	106
query22	4814	4736	4452	4452
query23	35287	35158	34535	34535
query24	11179	2969	2810	2810
query25	590	411	416	411
query26	740	159	166	159
query27	2149	311	296	296
query28	6989	2489	2412	2412
query29	736	421	420	420
query30	275	150	149	149
query31	1012	801	766	766
query32	86	53	53	53
query33	732	283	293	283
query34	907	511	483	483
query35	877	734	734	734
query36	1099	964	958	958
query37	142	85	83	83
query38	4055	3966	3922	3922
query39	1514	1443	1469	1443
query40	205	94	94	94
query41	49	45	48	45
query42	110	98	93	93
query43	532	501	476	476
query44	1228	819	807	807
query45	197	164	165	164
query46	1143	705	732	705
query47	1930	1807	1863	1807
query48	468	353	377	353
query49	867	392	373	373
query50	832	415	412	412
query51	7095	6888	7004	6888
query52	101	89	87	87
query53	262	181	185	181
query54	1157	471	464	464
query55	77	75	78	75
query56	290	267	265	265
query57	1242	1097	1095	1095
query58	237	236	259	236
query59	3145	2917	2989	2917
query60	295	276	270	270
query61	104	125	108	108
query62	831	673	650	650
query63	216	188	188	188
query64	3612	653	619	619
query65	3270	3195	3260	3195
query66	709	305	316	305
query67	15815	15822	15554	15554
query68	3891	603	570	570
query69	450	295	297	295
query70	1194	1146	1147	1146
query71	341	279	284	279
query72	6140	3915	4015	3915
query73	769	353	360	353
query74	9852	8999	9128	8999
query75	3423	2707	2646	2646
query76	2468	844	955	844
query77	444	290	294	290
query78	10469	9705	9541	9541
query79	4419	591	614	591
query80	1712	449	446	446
query81	589	241	238	238
query82	684	145	146	145
query83	286	136	140	136
query84	270	80	73	73
query85	1286	323	283	283
query86	411	302	294	294
query87	4439	4414	4381	4381
query88	4379	2366	2342	2342
query89	412	293	296	293
query90	1813	190	192	190
query91	181	144	147	144
query92	64	51	48	48
query93	3109	552	552	552
query94	811	308	291	291
query95	354	265	262	262
query96	650	282	273	273
query97	3197	3147	3150	3147
query98	215	206	201	201
query99	1541	1308	1315	1308
Total cold run time: 298424 ms
Total hot run time: 192346 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4254a77c40d6f3502099637c4a43c9ab9ed53990, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.54	0.52	0.52
query6	1.13	0.72	0.73
query7	0.01	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.56	0.58	0.54
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	2.92	2.96	3.02
query15	0.91	0.83	0.84
query16	0.37	0.39	0.38
query17	1.02	1.04	1.04
query18	0.20	0.19	0.20
query19	1.96	1.83	2.05
query20	0.02	0.01	0.01
query21	15.38	0.58	0.58
query22	2.28	2.67	1.81
query23	17.08	0.88	0.72
query24	2.84	1.48	0.87
query25	0.32	0.18	0.09
query26	0.36	0.14	0.13
query27	0.04	0.04	0.03
query28	10.41	1.10	1.07
query29	12.60	3.25	3.29
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.30	0.46	0.46
query33	3.00	3.01	3.02
query34	16.67	4.43	4.43
query35	4.54	4.51	4.51
query36	0.66	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.02	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.12 s
Total hot run time: 32.78 s

@w41ter w41ter merged commit ae766d3 into master Sep 26, 2024
31 of 37 checks passed
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 26, 2024
@w41ter
[fix](snapshot) Link binlog files according to consistent rowsets (ap…
5536505 
…ache#41319)

In previous implementations, binlog files would be linked according to
making snapshot requests. However, sometimes not all requests can be
executed directly. For example, when a certain version in
missing_version does not exist, it will fallback to the full snapshot.
Therefore, it is correct to link binlog files according to consistent
rowsets.
This was referenced Sep 26, 2024
Merged
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 26, 2024
@w41ter
[fix](snapshot) Link binlog files according to consistent rowsets (ap…
f42cd09 
…ache#41319)

In previous implementations, binlog files would be linked according to
making snapshot requests. However, sometimes not all requests can be
executed directly. For example, when a certain version in
missing_version does not exist, it will fallback to the full snapshot.
Therefore, it is correct to link binlog files according to consistent
rowsets.
@w41ter w41ter deleted the fix_snapshot_binlog_files branch September 26, 2024 07:56
w41ter added a commit that referenced this pull request Sep 26, 2024
@w41ter
[fix](snapshot) Link binlog files according to consistent rowsets #41319
d875b02 
 (#41345)

cherry pick from #41319
w41ter added a commit that referenced this pull request Sep 30, 2024
@w41ter
[fix](snapshot) Link binlog files according to consistent rowsets #41319
4058860 
 (#41344)

cherry pick from #41319
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@w41ter @dataroaring
[fix](snapshot) Link binlog files according to consistent rowsets (#4…
e5865f0 
…1319)

In previous implementations, binlog files would be linked according to
making snapshot requests. However, sometimes not all requests can be
executed directly. For example, when a certain version in
missing_version does not exist, it will fallback to the full snapshot.
Therefore, it is correct to link binlog files according to consistent
rowsets.
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@w41ter @cjj2010
[fix](snapshot) Link binlog files according to consistent rowsets (ap…
7bb87e1 
…ache#41319)

In previous implementations, binlog files would be linked according to
making snapshot requests. However, sometimes not all requests can be
executed directly. For example, when a certain version in
missing_version does not exist, it will fallback to the full snapshot.
Therefore, it is correct to link binlog files according to consistent
rowsets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@w41ter @doris-robot @gavinchou @dataroaring @wm1581066