Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Opt](partial update) Add some cases for partial update #48161

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Feb 20, 2025

What problem does this PR solve?

add some p0 cases:

  • partial update using s3 load
  • partial update using stream load with hidden_columns params, with invalid delete param, with invalid where param

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 marked this pull request as ready for review February 21, 2025 07:54
@zhannngchen
Copy link
Contributor

plz add case descriptions

@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 24, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31629 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd7d83612bcee1c1236573618e4098bc41259800, data reload: false

------ Round 1 ----------------------------------
q1	18815	5366	5121	5121
q2	2045	303	162	162
q3	11916	1246	731	731
q4	10197	1030	531	531
q5	7529	2459	2271	2271
q6	187	174	136	136
q7	897	759	607	607
q8	9551	1336	1203	1203
q9	5024	4637	4859	4637
q10	6973	2370	1961	1961
q11	501	305	279	279
q12	372	359	224	224
q13	18674	3783	3113	3113
q14	242	263	227	227
q15	506	465	462	462
q16	717	625	596	596
q17	932	880	339	339
q18	8623	6373	6122	6122
q19	1704	945	524	524
q20	298	321	185	185
q21	2798	2102	1899	1899
q22	350	335	299	299
Total cold run time: 108851 ms
Total hot run time: 31629 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5159	5080	5110	5080
q2	233	325	231	231
q3	2284	2696	2310	2310
q4	1406	1814	1365	1365
q5	4222	4101	4236	4101
q6	210	168	127	127
q7	1969	1925	1753	1753
q8	2606	2559	2561	2559
q9	7237	7375	7170	7170
q10	2981	3255	2660	2660
q11	558	492	478	478
q12	753	795	630	630
q13	3496	3876	3272	3272
q14	273	310	272	272
q15	525	471	461	461
q16	627	682	643	643
q17	1128	1564	1340	1340
q18	7580	7370	7351	7351
q19	792	788	848	788
q20	1991	2013	1876	1876
q21	5447	4744	4879	4744
q22	601	580	550	550
Total cold run time: 52078 ms
Total hot run time: 49761 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189853 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd7d83612bcee1c1236573618e4098bc41259800, data reload: false

query1	1313	919	952	919
query2	6131	1908	1894	1894
query3	10986	4445	4547	4445
query4	53571	24571	23005	23005
query5	5208	565	493	493
query6	371	200	180	180
query7	5131	506	286	286
query8	306	226	234	226
query9	6226	2539	2552	2539
query10	406	309	245	245
query11	15179	15071	14809	14809
query12	162	112	110	110
query13	1140	524	397	397
query14	10537	6235	6611	6235
query15	210	201	186	186
query16	7075	630	460	460
query17	1106	755	584	584
query18	1532	425	336	336
query19	219	191	170	170
query20	145	132	128	128
query21	208	128	110	110
query22	4529	4673	4350	4350
query23	33886	33568	33323	33323
query24	5595	2371	2408	2371
query25	458	443	439	439
query26	700	281	165	165
query27	1873	500	330	330
query28	2965	2497	2444	2444
query29	555	578	411	411
query30	223	189	162	162
query31	916	873	837	837
query32	77	69	59	59
query33	475	355	305	305
query34	862	886	518	518
query35	794	857	792	792
query36	951	1013	908	908
query37	120	92	71	71
query38	4275	4330	4249	4249
query39	1489	1505	1411	1411
query40	201	109	99	99
query41	51	47	54	47
query42	126	118	106	106
query43	524	532	482	482
query44	1285	793	812	793
query45	177	173	163	163
query46	882	1054	658	658
query47	1862	1902	1768	1768
query48	400	442	315	315
query49	694	511	423	423
query50	712	834	410	410
query51	4317	4317	4281	4281
query52	112	100	90	90
query53	225	266	193	193
query54	475	483	417	417
query55	83	83	82	82
query56	281	267	251	251
query57	1205	1167	1125	1125
query58	263	273	234	234
query59	2829	2909	2867	2867
query60	281	287	263	263
query61	119	119	122	119
query62	735	749	701	701
query63	232	198	188	188
query64	1501	1011	721	721
query65	3206	3116	3110	3110
query66	730	391	295	295
query67	15707	15393	15556	15393
query68	5294	767	521	521
query69	509	297	265	265
query70	1207	1129	1106	1106
query71	420	296	246	246
query72	6355	3585	3752	3585
query73	1069	755	349	349
query74	8935	9150	9055	9055
query75	3235	3152	2703	2703
query76	3781	1160	714	714
query77	554	366	275	275
query78	9942	10073	9328	9328
query79	2435	824	603	603
query80	587	523	440	440
query81	504	275	240	240
query82	646	124	92	92
query83	172	167	151	151
query84	289	94	76	76
query85	831	342	305	305
query86	399	298	325	298
query87	4435	4530	4487	4487
query88	3798	2233	2189	2189
query89	428	308	284	284
query90	1793	190	197	190
query91	133	136	106	106
query92	75	62	57	57
query93	1964	1004	570	570
query94	685	413	300	300
query95	343	262	258	258
query96	498	538	274	274
query97	2824	2898	2784	2784
query98	218	201	197	197
query99	1320	1398	1309	1309
Total cold run time: 292798 ms
Total hot run time: 189853 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd7d83612bcee1c1236573618e4098bc41259800, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.42	0.40	0.41
query6	1.18	0.69	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.53	0.52
query10	0.58	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.61	0.61
query14	2.68	2.75	2.74
query15	0.91	0.85	0.84
query16	0.37	0.38	0.38
query17	1.03	1.03	1.06
query18	0.21	0.20	0.19
query19	1.95	1.86	1.98
query20	0.02	0.01	0.01
query21	15.42	0.88	0.54
query22	0.74	1.15	0.71
query23	14.90	1.40	0.62
query24	6.84	1.81	0.50
query25	0.51	0.18	0.07
query26	0.64	0.15	0.13
query27	0.05	0.04	0.04
query28	9.59	0.89	0.45
query29	12.59	4.00	3.27
query30	0.25	0.08	0.06
query31	2.83	0.57	0.38
query32	3.22	0.54	0.47
query33	3.06	3.10	3.02
query34	15.75	5.13	4.51
query35	4.54	4.53	4.55
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 104.97 s
Total hot run time: 30.36 s

dataroaring
dataroaring previously approved these changes Feb 24, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 24, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 24, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 24, 2025
@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 25, 2025

run cloud_p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants