Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](spill) fix spill data usage counter is not update correctly #37352

Merged
merged 1 commit into from
Jul 16, 2024
Merged

[fix](spill) fix spill data usage counter is not update correctly #37352

merged 1 commit into from
Jul 16, 2024

Conversation

jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

If QueryContext is destructed earlier than PipelineFragmentContext, spill_dir_ will be already moved to spill_gc directory, which result int spill data disk usage is not decreased.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39672 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e6e10eb9fbf894b90111f00e68ec0816fbbebf46, data reload: false

------ Round 1 ----------------------------------
q1	18025	4501	4403	4403
q2	2163	200	202	200
q3	10443	1201	1031	1031
q4	10189	831	730	730
q5	7496	2689	2568	2568
q6	220	136	137	136
q7	959	593	600	593
q8	9227	2058	2109	2058
q9	8743	6493	6476	6476
q10	8796	3689	3704	3689
q11	431	240	235	235
q12	402	235	225	225
q13	17841	2968	2983	2968
q14	264	223	225	223
q15	510	473	480	473
q16	470	371	373	371
q17	955	666	619	619
q18	8252	7584	7479	7479
q19	3583	1497	1466	1466
q20	673	334	328	328
q21	4988	3069	3227	3069
q22	389	332	336	332
Total cold run time: 115019 ms
Total hot run time: 39672 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4249	4224	4224
q2	375	261	266	261
q3	3027	2754	2724	2724
q4	1887	1592	1644	1592
q5	5250	5266	5253	5253
q6	218	134	132	132
q7	2107	1724	1745	1724
q8	3212	3332	3288	3288
q9	8345	8282	8340	8282
q10	3874	3620	3691	3620
q11	574	500	502	500
q12	779	597	594	594
q13	16570	2966	2972	2966
q14	288	269	254	254
q15	523	470	485	470
q16	467	403	424	403
q17	1752	1520	1475	1475
q18	7657	7546	7262	7262
q19	2168	1637	1593	1593
q20	1984	1767	1785	1767
q21	4741	4665	4778	4665
q22	633	534	520	520
Total cold run time: 70798 ms
Total hot run time: 53569 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e6e10eb9fbf894b90111f00e68ec0816fbbebf46, data reload: false

query1	916	371	366	366
query2	6468	2448	2304	2304
query3	6655	211	220	211
query4	27475	17634	17396	17396
query5	4189	500	491	491
query6	289	211	183	183
query7	4609	296	298	296
query8	324	283	287	283
query9	8492	2414	2403	2403
query10	607	316	282	282
query11	12994	10104	10074	10074
query12	130	89	82	82
query13	1644	371	374	371
query14	10363	6805	7658	6805
query15	248	183	183	183
query16	7804	298	313	298
query17	1827	557	521	521
query18	1858	275	270	270
query19	197	148	150	148
query20	89	82	83	82
query21	213	148	129	129
query22	4412	4051	3966	3966
query23	33863	33008	33224	33008
query24	11920	2847	2746	2746
query25	655	365	371	365
query26	1776	156	158	156
query27	2930	308	339	308
query28	7556	2086	2077	2077
query29	1115	630	599	599
query30	284	147	146	146
query31	981	748	740	740
query32	94	52	55	52
query33	777	293	303	293
query34	1010	494	473	473
query35	716	657	596	596
query36	1088	952	942	942
query37	277	76	79	76
query38	2867	2736	2716	2716
query39	881	814	808	808
query40	286	126	126	126
query41	55	50	53	50
query42	122	101	100	100
query43	584	560	568	560
query44	1194	746	734	734
query45	194	160	163	160
query46	1101	718	741	718
query47	1883	1794	1794	1794
query48	362	294	292	292
query49	1185	411	407	407
query50	772	384	397	384
query51	6926	6733	6755	6733
query52	103	92	97	92
query53	357	294	295	294
query54	1060	461	447	447
query55	76	72	75	72
query56	292	273	273	273
query57	1146	1062	1095	1062
query58	266	265	252	252
query59	3290	3124	3169	3124
query60	299	275	293	275
query61	96	129	92	92
query62	653	435	449	435
query63	313	293	285	285
query64	10461	2183	1660	1660
query65	3178	3082	3080	3080
query66	1378	348	334	334
query67	15502	15026	15042	15026
query68	4559	552	517	517
query69	531	406	367	367
query70	1160	1071	1148	1071
query71	371	285	274	274
query72	7137	5379	5462	5379
query73	751	330	319	319
query74	5895	5483	5433	5433
query75	3375	2615	2663	2615
query76	2838	954	1025	954
query77	519	367	305	305
query78	10784	8949	8764	8764
query79	2583	509	516	509
query80	1678	491	473	473
query81	572	218	217	217
query82	855	111	109	109
query83	311	171	170	170
query84	279	86	90	86
query85	1828	312	298	298
query86	493	330	292	292
query87	3226	3116	3187	3116
query88	4410	2363	2359	2359
query89	500	400	399	399
query90	1881	190	192	190
query91	138	101	104	101
query92	69	50	52	50
query93	2397	515	508	508
query94	1247	216	213	213
query95	410	321	379	321
query96	605	262	265	262
query97	3129	3009	3035	3009
query98	211	201	198	198
query99	1201	866	853	853
Total cold run time: 289670 ms
Total hot run time: 171837 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e6e10eb9fbf894b90111f00e68ec0816fbbebf46, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.07
query5	0.52	0.51	0.49
query6	1.13	0.73	0.73
query7	0.01	0.01	0.02
query8	0.06	0.05	0.05
query9	0.56	0.49	0.50
query10	0.55	0.54	0.56
query11	0.15	0.12	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.58
query14	0.76	0.79	0.75
query15	0.85	0.80	0.82
query16	0.37	0.37	0.36
query17	1.01	1.05	1.01
query18	0.22	0.25	0.24
query19	1.91	1.84	1.69
query20	0.02	0.01	0.01
query21	15.40	0.77	0.65
query22	5.03	7.13	1.59
query23	18.37	1.39	1.30
query24	2.13	0.24	0.22
query25	0.16	0.08	0.08
query26	0.31	0.22	0.22
query27	0.46	0.23	0.23
query28	13.27	1.02	1.00
query29	12.61	3.28	3.28
query30	0.25	0.05	0.06
query31	2.86	0.40	0.40
query32	3.28	0.48	0.45
query33	2.86	2.90	2.92
query34	16.95	4.37	4.36
query35	4.43	4.42	4.45
query36	0.65	0.48	0.47
query37	0.18	0.16	0.15
query38	0.14	0.14	0.14
query39	0.05	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.63 s
Total hot run time: 30.34 s

yiguolei
yiguolei approved these changes Jul 12, 2024
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

mrhhsg
mrhhsg approved these changes Jul 16, 2024
View reviewed changes
Copy link
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit a9b5cf2 into apache:master Jul 16, 2024
25 of 29 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@jacktengg @seawinde
[fix](spill) fix spill data usage counter is not update correctly (ap…
5f532ba 
…ache#37352)

## Proposed changes

Issue Number: close #xxx

If QueryContext is destructed earlier than PipelineFragmentContext,
spill_dir_ will be already moved to spill_gc directory, which result int
spill data disk usage is not decreased.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@jacktengg @dataroaring
[fix](spill) fix spill data usage counter is not update correctly (#3…
d00afc3 
…7352)

## Proposed changes

Issue Number: close #xxx

If QueryContext is destructed earlier than PipelineFragmentContext,
spill_dir_ will be already moved to spill_gc directory, which result int
spill data disk usage is not decreased.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrhhsg mrhhsg mrhhsg approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jacktengg @doris-robot @mrhhsg @yiguolei @dataroaring