[fix](fe ut) fix unstable test DecommissionBackendTest #26891

yujun777 · 2023-11-13T09:57:22Z

fix unstable test DecommissionBackendTest.testDecommissionBackendWithDropTable by PR #25444
because it increase backend num. The new create table 'tb1''s tablets maynot create on the decommission backend.

Also fix "alter system add backend", because it's new create backend's state is alive = false;

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…DropTable

yujun777 · 2023-11-13T09:57:31Z

run buildall

dataroaring

LGTM

github-actions · 2023-11-13T10:14:13Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-11-13T10:14:16Z

PR approved by anyone and no changes requested.

doris-robot · 2023-11-13T12:01:49Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.11 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162337826 Bytes

doris-robot · 2023-11-13T12:06:32Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c7130191c99655b7ec5ad585ecdb712ebb5c8ac3, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5273	5242	5248	5242
q2	364	150	159	150
q3	2034	2016	2011	2011
q4	1392	1372	1344	1344
q5	3997	4011	4006	4006
q6	258	132	131	131
q7	1501	894	904	894
q8	2773	2780	2771	2771
q9	9743	9685	9406	9406
q10	3479	3548	3541	3541
q11	374	262	256	256
q12	435	287	286	286
q13	4598	4146	4156	4146
q14	322	305	308	305
q15	633	544	583	544
q16	689	590	591	590
q17	1135	1100	1101	1100
q18	8009	7654	7523	7523
q19	1685	1689	1665	1665
q20	524	309	317	309
q21	4693	4329	4378	4329
q22	541	421	414	414
Total cold run time: 54452 ms
Total hot run time: 50963 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5047	5004	5153	5004
q2	341	222	229	222
q3	4042	4011	4037	4011
q4	2838	2794	2771	2771
q5	9653	9630	9643	9630
q6	246	125	127	125
q7	3073	2529	2503	2503
q8	4896	4919	4970	4919
q9	13148	12983	13014	12983
q10	4095	4164	4223	4164
q11	799	674	661	661
q12	1016	840	874	840
q13	4261	3950	3839	3839
q14	380	356	372	356
q15	582	539	571	539
q16	778	721	725	721
q17	3866	3898	3871	3871
q18	9756	9531	9555	9531
q19	1868	1783	1776	1776
q20	2386	2082	2059	2059
q21	8963	8859	8772	8772
q22	956	851	845	845
Total cold run time: 82990 ms
Total hot run time: 80142 ms

yujun777 · 2023-11-14T15:37:26Z

run buildall

doris-robot · 2023-11-14T16:10:43Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit e061721753bdfc818f11ea3fe8cba534327f7a4b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5239	5075	5119	5075
q2	364	150	159	150
q3	2045	2007	2023	2007
q4	1407	1355	1356	1355
q5	3952	4079	4005	4005
q6	253	129	122	122
q7	1463	896	885	885
q8	2787	2778	2784	2778
q9	9844	9765	9577	9577
q10	3484	3541	3535	3535
q11	394	258	256	256
q12	430	279	286	279
q13	4557	4156	4121	4121
q14	316	283	294	283
q15	635	576	577	576
q16	679	587	580	580
q17	1146	1062	1087	1062
q18	8099	7607	7671	7607
q19	1695	1705	1690	1690
q20	532	329	279	279
q21	4672	4369	4394	4369
q22	515	404	404	404
Total cold run time: 54508 ms
Total hot run time: 50995 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5051	5071	4987	4987
q2	319	214	229	214
q3	4043	4023	3980	3980
q4	2773	2744	2739	2739
q5	9667	9547	9692	9547
q6	249	121	121	121
q7	3051	2519	2524	2519
q8	4824	4880	4839	4839
q9	13254	13108	13070	13070
q10	4093	4159	4198	4159
q11	720	667	664	664
q12	981	881	797	797
q13	4271	3921	3890	3890
q14	396	346	340	340
q15	580	553	544	544
q16	731	710	688	688
q17	3923	3832	3855	3832
q18	9584	9276	9315	9276
q19	1864	1806	1770	1770
q20	2430	2050	2074	2050
q21	8836	8714	8748	8714
q22	912	847	835	835
Total cold run time: 82552 ms
Total hot run time: 79575 ms

doris-robot · 2023-11-14T16:21:26Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.17 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17096119978 Bytes

dataroaring

LGTM

github-actions · 2023-11-15T00:22:20Z

PR approved by at least one committer and no changes requested.

wm1581066

LGTM

yujun777 · 2023-11-15T08:48:52Z

run buildall

yujun777 · 2023-11-15T08:49:17Z

run buildall

doris-robot · 2023-11-15T09:21:29Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.34 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17094269572 Bytes

doris-robot · 2023-11-15T09:21:50Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.6 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17094440795 Bytes

doris-robot · 2023-11-15T09:51:15Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c968d67d0029109e40205bc55f99572c050c2a63, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5286	5149	5133	5133
q2	357	179	159	159
q3	2026	1960	1925	1925
q4	1392	1261	1252	1252
q5	3992	3990	4006	3990
q6	250	132	132	132
q7	1417	887	895	887
q8	2772	2789	2789	2789
q9	16338	9761	9558	9558
q10	3456	3519	3521	3519
q11	384	245	262	245
q12	445	280	287	280
q13	4620	3828	3852	3828
q14	323	293	279	279
q15	660	565	560	560
q16	674	586	586	586
q17	1141	941	913	913
q18	7736	7445	7458	7445
q19	1709	1688	1671	1671
q20	562	330	296	296
q21	4489	4051	4020	4020
q22	529	414	437	414
Total cold run time: 60558 ms
Total hot run time: 49881 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5078	5095	5002	5002
q2	356	265	221	221
q3	4115	3953	3925	3925
q4	2796	2751	2770	2751
q5	9615	9647	9545	9545
q6	249	123	126	123
q7	2748	2279	2345	2279
q8	4807	4841	4860	4841
q9	13234	13123	13200	13123
q10	4081	4165	4171	4165
q11	765	645	641	641
q12	1019	838	896	838
q13	4311	3658	3624	3624
q14	384	358	339	339
q15	617	552	545	545
q16	756	660	726	660
q17	3855	3973	3840	3840
q18	9620	8971	9134	8971
q19	1839	1776	1778	1776
q20	2395	2062	2055	2055
q21	8709	8401	8230	8230
q22	942	820	897	820
Total cold run time: 82291 ms
Total hot run time: 78314 ms

dataroaring

LGTM

github-actions · 2023-11-15T12:52:04Z

PR approved by at least one committer and no changes requested.

fix unstable test DecommissionBackendTest.testDecommissionBackendWith…

c713019

…DropTable

dataroaring previously approved these changes Nov 13, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2023

github-actions bot added the reviewed label Nov 13, 2023

xiaokang added the dev/2.0.3 label Nov 14, 2023

fix fe ut

e061721

yujun777 dismissed dataroaring’s stale review via e061721 November 14, 2023 15:37

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 14, 2023

dataroaring previously approved these changes Nov 15, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 15, 2023

wm1581066 approved these changes Nov 15, 2023

View reviewed changes

yujun777 marked this pull request as draft November 15, 2023 08:27

fix ut

c968d67

yujun777 dismissed dataroaring’s stale review via c968d67 November 15, 2023 08:46

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 15, 2023

yujun777 marked this pull request as ready for review November 15, 2023 08:49

dataroaring approved these changes Nov 15, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 15, 2023

dataroaring merged commit 4dda5aa into apache:master Nov 15, 2023

yujun777 mentioned this pull request Nov 17, 2023

[branch-2.0](fe ut) fix unstable test DecommissionBackendTest #26891 #27173

Merged

dataroaring added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 17, 2023

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[fix](fe ut) fix unstable test DecommissionBackendTest (apache#26891)

1d212a0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](fe ut) fix unstable test DecommissionBackendTest #26891

[fix](fe ut) fix unstable test DecommissionBackendTest #26891

yujun777 commented Nov 13, 2023 •

edited

Loading

yujun777 commented Nov 13, 2023

dataroaring left a comment

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

doris-robot commented Nov 13, 2023

doris-robot commented Nov 13, 2023

yujun777 commented Nov 14, 2023

doris-robot commented Nov 14, 2023

doris-robot commented Nov 14, 2023

dataroaring left a comment

github-actions bot commented Nov 15, 2023

wm1581066 left a comment

yujun777 commented Nov 15, 2023

yujun777 commented Nov 15, 2023

doris-robot commented Nov 15, 2023

doris-robot commented Nov 15, 2023

doris-robot commented Nov 15, 2023

dataroaring left a comment

github-actions bot commented Nov 15, 2023

[fix](fe ut) fix unstable test DecommissionBackendTest #26891

[fix](fe ut) fix unstable test DecommissionBackendTest #26891

Conversation

yujun777 commented Nov 13, 2023 • edited Loading

Proposed changes

Further comments

yujun777 commented Nov 13, 2023

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

doris-robot commented Nov 13, 2023

doris-robot commented Nov 13, 2023

yujun777 commented Nov 14, 2023

doris-robot commented Nov 14, 2023

doris-robot commented Nov 14, 2023

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 15, 2023

wm1581066 left a comment

Choose a reason for hiding this comment

yujun777 commented Nov 15, 2023

yujun777 commented Nov 15, 2023

doris-robot commented Nov 15, 2023

doris-robot commented Nov 15, 2023

doris-robot commented Nov 15, 2023

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 15, 2023

yujun777 commented Nov 13, 2023 •

edited

Loading