Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](profile)add shuffle send rows/bytes #30456

Merged
merged 1 commit into from
Jan 27, 2024
Merged

[Feature](profile)add shuffle send rows/bytes #30456

merged 1 commit into from
Jan 27, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Jan 27, 2024 •
edited
Loading

Proposed changes

1 Add Metrics ShuffleSendBytes/ShuffleSendRows in fe.audit.log and tvf active_queries, it means rows and bytes after serialize which DataSinkNode send.
2

mysql [(none)]>desc function active_queries();
+------------------------+--------+------+-------+---------+-------+
| Field                  | Type   | Null | Key   | Default | Extra |
+------------------------+--------+------+-------+---------+-------+
| BeHost                 | TEXT   | No   | false | NULL    | NONE  |
| BePort                 | BIGINT | No   | false | NULL    | NONE  |
| QueryId                | TEXT   | No   | false | NULL    | NONE  |
| StartTime              | TEXT   | No   | false | NULL    | NONE  |
| QueryTimeMs            | BIGINT | No   | false | NULL    | NONE  |
| WorkloadGroupId        | BIGINT | No   | false | NULL    | NONE  |
| QueryCpuTimeMs         | BIGINT | No   | false | NULL    | NONE  |
| ScanRows               | BIGINT | No   | false | NULL    | NONE  |
| ScanBytes              | BIGINT | No   | false | NULL    | NONE  |
| BePeakMemoryBytes      | BIGINT | No   | false | NULL    | NONE  |
| CurrentUsedMemoryBytes | BIGINT | No   | false | NULL    | NONE  |
| ShuffleSendBytes       | BIGINT | No   | false | NULL    | NONE  |
| ShuffleSendRows        | BIGINT | No   | false | NULL    | NONE  |
| Database               | TEXT   | No   | false | NULL    | NONE  |
| FrontendInstance       | TEXT   | No   | false | NULL    | NONE  |
| Sql                    | TEXT   | No   | false | NULL    | NONE  |
+------------------------+--------+------+-------+---------+-------+
16 rows in set (0.01 sec)

mysql [(none)]>select QueryId,StartTime,QueryCpuTimeMs,ScanRows,ScanBytes,BePeakMemoryBytes,`Database`,FrontendInstance,ShuffleSendBytes,ShuffleSendRows from active_queries()\G;
*************************** 1. row ***************************
          QueryId: 774cc4b1cce448c7-b4c841b7b9d08923
        StartTime: 2024-01-27 13:53:27
   QueryCpuTimeMs: 28117
         ScanRows: 61352378
        ScanBytes: 21630189568
BePeakMemoryBytes: 1537534202
         Database: hits
 FrontendInstance: vm-10
 ShuffleSendBytes: 145682448
  ShuffleSendRows: 1175400
1 row in set (0.01 sec)

ERROR: 
No query specified

User=root|Ctl=internal|Db=hits|State=EOF|ErrorCode=0|ErrorMessage=|Time(ms)=30630|ScanBytes=27821867008|ScanRows=81032736|ReturnRows=11|StmtId=30|QueryId=774cc4b1cce448c7-b4c841b7b9d08923|IsQuery=true|isNereids=true|feIp=|Stmt=SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25|CpuTimeMS=38940|ShuffleSendBytes=333564448|ShuffleSendRows=2689024|SqlHash=b03d48a7e6849912003ad1cff9519957|peakMemoryBytes=2391511673|SqlDigest=d41d8cd98f00b204e9800998ecf8427e|TraceId=|WorkloadGroup=normal|FuzzyVariables=

@wangbo
Copy link
Contributor Author

wangbo commented Jan 27, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37917 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2635b043407809eced2daf8b0e5dd6f808663541, data reload: false

------ Round 1 ----------------------------------
q1	17651	5667	5068	5068
q2	2052	150	142	142
q3	10671	930	944	930
q4	4677	711	692	692
q5	7688	2863	2830	2830
q6	187	122	121	121
q7	1139	741	723	723
q8	9315	2008	2036	2008
q9	7237	6363	6343	6343
q10	8117	2443	2443	2443
q11	410	205	207	205
q12	771	293	279	279
q13	18004	3338	3328	3328
q14	270	255	246	246
q15	549	491	485	485
q16	462	413	404	404
q17	953	550	497	497
q18	6835	6248	6093	6093
q19	1582	1413	1384	1384
q20	588	339	341	339
q21	6469	3151	3069	3069
q22	794	295	288	288
Total cold run time: 106421 ms
Total hot run time: 37917 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5344	5046	4924	4924
q2	342	228	233	228
q3	3062	2789	2661	2661
q4	1818	1580	1534	1534
q5	5402	5309	5202	5202
q6	209	119	119	119
q7	2160	1726	1818	1726
q8	3172	3226	3317	3226
q9	8862	8732	12823	8732
q10	5521	3500	3410	3410
q11	552	477	474	474
q12	763	599	596	596
q13	17017	3254	3198	3198
q14	279	254	266	254
q15	538	489	487	487
q16	542	465	501	465
q17	2082	1660	1757	1660
q18	8977	9926	8336	8336
q19	24100	1570	1550	1550
q20	4736	1916	1907	1907
q21	15073	4801	5034	4801
q22	1054	444	453	444
Total cold run time: 111605 ms
Total hot run time: 55934 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173390 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2635b043407809eced2daf8b0e5dd6f808663541, data reload: false

query1	939	330	324	324
query2	6583	1956	1813	1813
query3	6698	197	196	196
query4	32581	22113	22022	22022
query5	4464	415	346	346
query6	245	157	159	157
query7	4621	261	252	252
query8	247	184	175	175
query9	8652	2288	2282	2282
query10	418	202	203	202
query11	17765	15432	15345	15345
query12	117	70	65	65
query13	1687	373	390	373
query14	9263	6552	6485	6485
query15	208	187	187	187
query16	5696	252	247	247
query17	956	474	473	473
query18	1815	259	260	259
query19	180	133	132	132
query20	67	73	69	69
query21	200	131	131	131
query22	4966	4766	4883	4766
query23	31174	30487	30301	30301
query24	11358	2761	2792	2761
query25	544	308	318	308
query26	894	139	147	139
query27	3028	284	289	284
query28	6370	1836	1819	1819
query29	946	613	601	601
query30	281	134	141	134
query31	904	718	706	706
query32	79	48	49	48
query33	593	208	214	208
query34	897	466	461	461
query35	841	750	767	750
query36	1329	1098	1194	1098
query37	90	57	60	57
query38	3297	3165	3201	3165
query39	1332	1262	1242	1242
query40	191	92	86	86
query41	44	35	35	35
query42	91	83	83	83
query43	520	497	492	492
query44	1010	677	685	677
query45	193	187	174	174
query46	1028	650	629	629
query47	1625	1534	1512	1512
query48	381	304	306	304
query49	1094	287	278	278
query50	676	314	300	300
query51	5217	5184	5222	5184
query52	91	90	72	72
query53	317	271	257	257
query54	259	182	191	182
query55	79	77	76	76
query56	175	165	163	163
query57	947	899	891	891
query58	180	168	154	154
query59	2689	2508	2589	2508
query60	206	178	174	174
query61	94	93	87	87
query62	590	366	362	362
query63	296	274	252	252
query64	4307	1765	1785	1765
query65	3343	3269	3217	3217
query66	1171	329	320	320
query67	14395	14431	14352	14352
query68	3669	489	482	482
query69	442	295	290	290
query70	1504	1507	1525	1507
query71	272	213	214	213
query72	3731	2873	2861	2861
query73	681	319	306	306
query74	6628	6344	6314	6314
query75	2790	2261	2287	2261
query76	2675	1027	912	912
query77	361	236	228	228
query78	9057	8925	8524	8524
query79	2685	498	494	494
query80	1977	330	324	324
query81	535	200	215	200
query82	709	85	82	82
query83	236	112	119	112
query84	294	84	80	80
query85	1939	352	342	342
query86	500	401	387	387
query87	3468	3347	3272	3272
query88	3769	2171	2165	2165
query89	412	363	355	355
query90	1915	182	179	179
query91	147	119	117	117
query92	56	43	43	43
query93	2896	458	439	439
query94	1193	158	158	158
query95	494	465	445	445
query96	625	310	314	310
query97	4247	4150	4137	4137
query98	214	185	187	185
query99	1044	702	697	697
Total cold run time: 274699 ms
Total hot run time: 173390 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2635b043407809eced2daf8b0e5dd6f808663541, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.05
query4	1.65	0.09	0.09
query5	0.52	0.52	0.51
query6	1.17	0.65	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.57	0.54	0.56
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.60	0.61
query14	0.78	0.82	0.81
query15	0.79	0.77	0.79
query16	0.38	0.39	0.38
query17	0.99	0.99	1.00
query18	0.22	0.27	0.24
query19	1.89	1.77	1.76
query20	0.01	0.01	0.01
query21	15.42	0.54	0.57
query22	2.78	2.06	2.14
query23	17.30	0.95	0.76
query24	2.42	1.07	1.17
query25	0.31	0.15	0.20
query26	0.58	0.14	0.14
query27	0.06	0.04	0.06
query28	11.87	0.86	0.84
query29	12.56	3.17	3.14
query30	0.59	0.56	0.51
query31	2.79	0.35	0.35
query32	3.33	0.49	0.48
query33	3.23	3.21	3.21
query34	15.91	4.39	4.33
query35	4.24	4.42	4.29
query36	1.11	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.68 s
Total hot run time: 31.38 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2635b043407809eced2daf8b0e5dd6f808663541 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.25% (8613/23763)
Line Coverage: 28.32% (70490/248925)
Region Coverage: 27.30% (36383/133258)
Branch Coverage: 24.10% (18646/77382)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2635b043407809eced2daf8b0e5dd6f808663541_2635b043407809eced2daf8b0e5dd6f808663541/report/index.html

@yiguolei yiguolei merged commit 6c988e7 into apache:master Jan 27, 2024
32 of 34 checks passed
yiguolei pushed a commit that referenced this pull request Jan 28, 2024
@wangbo @Doris-Extras
[Feature](profile)add shuffle send rows/bytes #30456
0433b87
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@jacktengg jacktengg jacktengg approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. meta-change reviewed workload-schedule
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wangbo @doris-robot @yiguolei @jacktengg