Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](routine-load) optimize error msg when meet out of range #30118

Merged
merged 1 commit into from
Jan 22, 2024
Merged

[improve](routine-load) optimize error msg when meet out of range #30118

merged 1 commit into from
Jan 22, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Jan 19, 2024

Proposed changes

When meet out of range in routine load, it is hard for user to deal it correctly, so add some processing method in message.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui
Copy link
Contributor Author

sollhui commented Jan 19, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3a28d640f3a5847777abd90adac993ff3502d61, data reload: false

------ Round 1 ----------------------------------
q1	17674	5142	5305	5142
q2	2034	140	129	129
q3	10640	1159	1139	1139
q4	10218	765	737	737
q5	7761	3140	3121	3121
q6	197	120	118	118
q7	852	494	477	477
q8	9258	1949	1951	1949
q9	7281	6360	6302	6302
q10	8178	3066	3060	3060
q11	415	209	206	206
q12	355	190	187	187
q13	17992	3340	3346	3340
q14	246	216	218	216
q15	565	506	519	506
q16	450	378	397	378
q17	934	538	474	474
q18	7426	6943	6717	6717
q19	1555	1382	1395	1382
q20	603	295	291	291
q21	2749	2371	2394	2371
q22	358	317	300	300
Total cold run time: 107741 ms
Total hot run time: 38542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5380	5414	5290	5290
q2	322	216	214	214
q3	3307	3245	3183	3183
q4	2077	2049	2028	2028
q5	5980	5669	5909	5669
q6	195	113	117	113
q7	2360	1911	1869	1869
q8	3197	3333	3388	3333
q9	8907	8688	8876	8688
q10	3905	3753	3742	3742
q11	549	460	440	440
q12	786	610	599	599
q13	16908	3120	3117	3117
q14	286	258	276	258
q15	559	499	509	499
q16	509	472	458	458
q17	1895	1835	1883	1835
q18	9439	18429	9620	9620
q19	25679	1559	1508	1508
q20	4638	1932	1928	1928
q21	14447	5302	5330	5302
q22	995	529	540	529
Total cold run time: 112320 ms
Total hot run time: 60222 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175992 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3a28d640f3a5847777abd90adac993ff3502d61, data reload: false

query1	914	367	324	324
query2	6543	2095	1812	1812
query3	6696	200	198	198
query4	32113	22075	22131	22075
query5	6936	521	499	499
query6	254	148	150	148
query7	4601	260	259	259
query8	225	192	178	178
query9	8471	2576	2571	2571
query10	424	224	228	224
query11	16992	15650	15466	15466
query12	115	67	67	67
query13	1691	373	385	373
query14	10438	6923	6879	6879
query15	209	174	174	174
query16	4602	231	226	226
query17	997	473	471	471
query18	1783	259	256	256
query19	175	139	134	134
query20	76	68	62	62
query21	190	137	129	129
query22	4977	4783	4753	4753
query23	31337	30647	30598	30598
query24	11717	2786	2800	2786
query25	574	318	303	303
query26	1445	138	140	138
query27	3229	265	269	265
query28	7040	1853	1854	1853
query29	1264	621	628	621
query30	282	134	138	134
query31	948	733	736	733
query32	83	50	49	49
query33	700	210	206	206
query34	1131	451	452	451
query35	856	759	764	759
query36	1388	1176	1240	1176
query37	90	60	58	58
query38	3371	3237	3280	3237
query39	1311	1260	1258	1258
query40	207	89	82	82
query41	39	35	34	34
query42	91	85	82	82
query43	534	480	507	480
query44	1106	677	676	676
query45	192	178	173	173
query46	1060	646	650	646
query47	1602	1575	1586	1575
query48	392	340	321	321
query49	1145	280	290	280
query50	668	307	313	307
query51	5279	5273	5195	5195
query52	92	81	76	76
query53	331	257	249	249
query54	463	287	285	285
query55	78	76	78	76
query56	179	174	159	159
query57	988	965	914	914
query58	199	154	172	154
query59	2953	2612	2761	2612
query60	206	182	190	182
query61	90	87	87	87
query62	651	373	353	353
query63	282	262	265	262
query64	4975	1773	1735	1735
query65	3330	3245	3222	3222
query66	1223	331	307	307
query67	15718	14932	15338	14932
query68	11790	527	496	496
query69	609	310	304	304
query70	1694	1563	1530	1530
query71	556	215	205	205
query72	5072	2849	2858	2849
query73	2301	310	310	310
query74	7860	6400	6480	6400
query75	4791	2293	2295	2293
query76	6685	1069	987	987
query77	668	230	230	230
query78	9165	8804	8574	8574
query79	1003	502	486	486
query80	572	315	305	305
query81	458	203	204	203
query82	197	80	81	80
query83	138	117	117	117
query84	270	70	69	69
query85	1048	323	311	311
query86	385	389	361	361
query87	3584	3347	3343	3343
query88	2880	2162	2177	2162
query89	420	344	342	342
query90	1992	190	186	186
query91	173	131	133	131
query92	54	43	44	43
query93	1957	441	437	437
query94	1144	156	157	156
query95	496	460	447	447
query96	623	319	309	309
query97	4243	4128	4108	4108
query98	216	188	194	188
query99	991	671	718	671
Total cold run time: 294296 ms
Total hot run time: 175992 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3a28d640f3a5847777abd90adac993ff3502d61, data reload: false

query1	0.04	0.02	0.02
query2	0.05	0.02	0.02
query3	0.23	0.04	0.05
query4	1.71	0.06	0.06
query5	0.53	0.52	0.52
query6	1.22	0.62	0.63
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.54	0.50	0.49
query10	0.55	0.55	0.55
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.60	0.60	0.60
query14	0.80	0.83	0.80
query15	0.81	0.77	0.77
query16	0.35	0.37	0.36
query17	1.02	1.00	1.00
query18	0.24	0.24	0.24
query19	1.88	1.80	1.78
query20	0.01	0.01	0.01
query21	15.40	0.59	0.58
query22	3.21	2.35	2.04
query23	17.33	0.92	0.77
query24	2.37	1.18	0.91
query25	0.40	0.19	0.10
query26	0.54	0.12	0.12
query27	0.05	0.06	0.06
query28	12.15	0.76	0.76
query29	12.54	3.31	3.28
query30	0.50	0.45	0.44
query31	2.78	0.34	0.33
query32	3.39	0.47	0.47
query33	3.21	3.23	3.22
query34	15.86	4.41	4.31
query35	4.28	4.34	4.26
query36	1.11	1.07	1.08
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.15	0.13	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 106.35 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d3a28d640f3a5847777abd90adac993ff3502d61 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.7 seconds inserted 10000000 Rows, about 680K ops/s

dataroaring
dataroaring previously approved these changes Jan 19, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@sollhui
Copy link
Contributor Author

sollhui commented Jan 20, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 20, 2024
@sollhui
Copy link
Contributor Author

sollhui commented Jan 20, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Jan 20, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38521 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e547bcf76b8b7f73a161056b9c87dbc0df1949b3, data reload: false

------ Round 1 ----------------------------------
q1	17640	5208	5442	5208
q2	2048	146	132	132
q3	10638	1151	1153	1151
q4	10222	772	790	772
q5	7766	3151	3149	3149
q6	198	118	123	118
q7	868	499	485	485
q8	9235	1928	1949	1928
q9	7136	6379	6343	6343
q10	8210	3039	3030	3030
q11	418	222	215	215
q12	353	188	185	185
q13	17978	3361	3324	3324
q14	245	216	215	215
q15	544	500	518	500
q16	421	378	371	371
q17	926	524	490	490
q18	7515	6964	6590	6590
q19	1578	1328	1384	1328
q20	564	314	295	295
q21	2795	2379	2415	2379
q22	353	313	316	313
Total cold run time: 107651 ms
Total hot run time: 38521 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5311	5206	5233	5206
q2	322	218	212	212
q3	3328	3206	3177	3177
q4	2079	2078	1997	1997
q5	6026	5808	5965	5808
q6	198	114	118	114
q7	2312	1891	1928	1891
q8	3211	3358	3367	3358
q9	8822	8882	8758	8758
q10	3940	3765	3787	3765
q11	557	447	460	447
q12	799	607	626	607
q13	16919	3148	3097	3097
q14	283	255	261	255
q15	563	502	502	502
q16	521	465	462	462
q17	5890	1795	1837	1795
q18	9480	17189	9737	9737
q19	21875	1556	1524	1524
q20	4644	1930	1929	1929
q21	14518	5452	5361	5361
q22	1004	557	542	542
Total cold run time: 112602 ms
Total hot run time: 60544 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176036 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e547bcf76b8b7f73a161056b9c87dbc0df1949b3, data reload: false

query1	937	340	329	329
query2	6565	2021	1773	1773
query3	6695	204	192	192
query4	31006	22165	22196	22165
query5	6905	631	623	623
query6	254	165	168	165
query7	4617	261	255	255
query8	220	171	177	171
query9	8446	2545	2522	2522
query10	432	239	225	225
query11	16668	15772	15420	15420
query12	120	68	66	66
query13	1678	368	380	368
query14	10567	7045	6920	6920
query15	205	182	180	180
query16	4606	232	226	226
query17	998	480	480	480
query18	1786	257	262	257
query19	186	140	134	134
query20	68	68	65	65
query21	186	135	116	116
query22	4859	4527	4664	4527
query23	31554	30913	30841	30841
query24	11945	2766	2848	2766
query25	574	315	311	311
query26	1645	148	152	148
query27	3216	275	270	270
query28	7234	1842	1826	1826
query29	1669	651	641	641
query30	282	141	139	139
query31	940	738	745	738
query32	81	52	48	48
query33	698	216	228	216
query34	1130	474	469	469
query35	897	784	782	782
query36	1333	1182	1174	1174
query37	92	58	63	58
query38	3370	3266	3202	3202
query39	1317	1266	1251	1251
query40	333	87	83	83
query41	37	35	34	34
query42	93	83	86	83
query43	511	515	441	441
query44	1110	689	690	689
query45	192	175	176	175
query46	1046	658	663	658
query47	1598	1561	1542	1542
query48	396	307	324	307
query49	1158	295	289	289
query50	673	317	322	317
query51	5285	5213	5185	5185
query52	88	82	77	77
query53	329	255	257	255
query54	484	298	290	290
query55	83	75	75	75
query56	184	174	182	174
query57	956	921	868	868
query58	194	168	166	166
query59	2866	2579	2697	2579
query60	207	184	183	183
query61	87	86	93	86
query62	628	348	369	348
query63	276	264	254	254
query64	5838	1847	1771	1771
query65	3320	3271	3232	3232
query66	1256	327	316	316
query67	15493	14957	15137	14957
query68	10490	527	492	492
query69	598	309	300	300
query70	1761	1493	1386	1386
query71	541	210	210	210
query72	4652	2833	2871	2833
query73	1885	313	322	313
query74	6944	6456	6344	6344
query75	4725	2325	2289	2289
query76	6665	1043	1015	1015
query77	641	232	231	231
query78	9012	8778	8534	8534
query79	1382	507	506	506
query80	633	327	318	318
query81	445	207	207	207
query82	207	84	85	84
query83	145	118	125	118
query84	283	68	68	68
query85	1059	339	344	339
query86	386	384	386	384
query87	3611	3335	3372	3335
query88	3538	2191	2186	2186
query89	437	365	344	344
query90	2009	187	187	187
query91	156	122	128	122
query92	56	46	42	42
query93	3356	440	428	428
query94	1160	158	156	156
query95	504	464	455	455
query96	629	317	320	317
query97	4282	4195	4124	4124
query98	205	197	179	179
query99	999	702	697	697
Total cold run time: 294098 ms
Total hot run time: 176036 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e547bcf76b8b7f73a161056b9c87dbc0df1949b3, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.02
query3	0.23	0.05	0.05
query4	1.69	0.07	0.06
query5	0.53	0.52	0.52
query6	1.19	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.56	0.50	0.50
query10	0.56	0.55	0.56
query11	0.11	0.09	0.08
query12	0.11	0.10	0.09
query13	0.60	0.61	0.62
query14	0.78	0.80	0.80
query15	0.78	0.76	0.77
query16	0.40	0.39	0.39
query17	1.00	0.99	1.03
query18	0.24	0.24	0.25
query19	1.85	1.79	1.77
query20	0.01	0.00	0.01
query21	15.40	0.57	0.58
query22	2.42	2.25	1.64
query23	17.08	0.82	0.87
query24	2.41	1.45	0.12
query25	0.30	0.15	0.10
query26	0.56	0.14	0.13
query27	0.06	0.07	0.04
query28	12.03	0.77	0.76
query29	12.59	3.23	3.12
query30	0.54	0.48	0.53
query31	2.79	0.35	0.36
query32	3.36	0.48	0.49
query33	3.21	3.18	3.26
query34	15.78	4.35	4.29
query35	4.33	4.34	4.31
query36	1.12	1.07	1.10
query37	0.06	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.08	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.17 s
Total hot run time: 29.83 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e547bcf76b8b7f73a161056b9c87dbc0df1949b3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.4 seconds inserted 10000000 Rows, about 694K ops/s

dataroaring
dataroaring approved these changes Jan 22, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 27f64c5 into apache:master Jan 22, 2024
26 of 27 checks passed
@sollhui sollhui deleted the routine_load_out_off_range branch January 22, 2024 05:55
@sollhui sollhui mentioned this pull request Jan 22, 2024
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 22, 2024
@sollhui @seawinde
[fix](routine-load) optimize error msg when meet out of range (apache…
7c5be93 
…#30118)
yiguolei pushed a commit that referenced this pull request Jan 22, 2024
@sollhui
[fix](routine-load) optimize error msg when meet out of range (#30118) (
a014a37 
#30210)
yiguolei pushed a commit that referenced this pull request Jan 23, 2024
@sollhui @Doris-Extras
[fix](routine-load) optimize error msg when meet out of range (#30118)
d0dd090
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui
[fix](routine-load) optimize error msg when meet out of range (apache…
eb81fcb 
…#30118) (apache#30210)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sollhui @doris-robot @liaoxin01 @dataroaring @wm1581066