Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement](index tool) refine inverted index tool code #29717

Merged
merged 2 commits into from
Jan 9, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Copy link
Contributor

github-actions bot commented Jan 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

run buildall

Copy link
Contributor

github-actions bot commented Jan 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.64% (8640/23580)
Line Coverage: 28.65% (70208/245052)
Region Coverage: 27.60% (36316/131570)
Branch Coverage: 24.29% (18554/76396)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bac5553f10454f91625bc5577c6a91f302410a33_bac5553f10454f91625bc5577c6a91f302410a33/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39311 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bac5553f10454f91625bc5577c6a91f302410a33, data reload: false

------ Round 1 ----------------------------------
q1	18331	5538	5563	5538
q2	2594	167	156	156
q3	11289	1097	1097	1097
q4	10251	814	853	814
q5	7888	2976	2938	2938
q6	204	132	130	130
q7	917	519	511	511
q8	9311	1975	2004	1975
q9	6486	6387	6357	6357
q10	8228	3061	2965	2965
q11	432	234	217	217
q12	386	230	240	230
q13	18274	3467	3431	3431
q14	240	209	217	209
q15	554	518	502	502
q16	458	403	412	403
q17	934	765	696	696
q18	7337	6658	6649	6649
q19	1552	1515	1484	1484
q20	722	324	301	301
q21	2734	2414	2370	2370
q22	381	338	340	338
Total cold run time: 109503 ms
Total hot run time: 39311 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5521	5533	5509	5509
q2	345	262	269	262
q3	3322	3331	3346	3331
q4	2118	2072	2055	2055
q5	5838	5844	5858	5844
q6	203	123	124	123
q7	2289	1878	1894	1878
q8	3480	3574	3540	3540
q9	8887	8942	8885	8885
q10	3848	3870	3851	3851
q11	599	517	499	499
q12	801	628	651	628
q13	6379	3229	3230	3229
q14	302	258	257	257
q15	560	520	503	503
q16	555	487	493	487
q17	2416	2400	2390	2390
q18	8664	8576	8554	8554
q19	1570	1616	1613	1613
q20	2194	1986	1967	1967
q21	5576	5419	5340	5340
q22	574	549	516	516
Total cold run time: 66041 ms
Total hot run time: 61261 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187976 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bac5553f10454f91625bc5577c6a91f302410a33, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	984	374	363	363
query2	6880	2089	2075	2075
query3	6718	237	226	226
query4	25593	23016	22705	22705
query5	6146	628	620	620
query6	340	232	217	217
query7	4636	313	304	304
query8	257	241	235	235
query9	9732	3342	3305	3305
query10	703	359	332	332
query11	16564	15493	15755	15493
query12	174	128	117	117
query13	1651	345	342	342
query14	12425	7700	7657	7657
query15	256	245	249	245
query16	6381	353	351	351
query17	1741	529	550	529
query18	1990	349	338	338
query19	264	171	181	171
query20	126	113	123	113
query21	190	107	98	98
query22	4839	4564	4529	4529
query23	32720	31623	31785	31623
query24	12522	2969	3003	2969
query25	619	407	388	388
query26	1780	189	201	189
query27	2975	327	322	322
query28	7124	2131	2126	2126
query29	2142	427	435	427
query30	301	158	155	155
query31	1039	868	862	862
query32	135	108	100	100
query33	841	360	374	360
query34	916	499	507	499
query35	994	909	887	887
query36	1433	1320	1335	1320
query37	229	106	106	106
query38	3541	3383	3354	3354
query39	1542	1336	1325	1325
query40	310	104	96	96
query41	43	43	37	37
query42	123	111	117	111
query43	617	572	587	572
query44	1222	846	838	838
query45	218	208	205	205
query46	1110	743	722	722
query47	1799	1671	1684	1671
query48	376	310	293	293
query49	1225	340	341	340
query50	739	370	377	370
query51	5560	5404	5324	5324
query52	135	105	106	105
query53	237	177	177	177
query54	1409	720	747	720
query55	122	108	106	106
query56	280	310	307	307
query57	1060	1015	1011	1011
query58	367	350	310	310
query59	3436	3128	3156	3128
query60	383	321	305	305
query61	129	131	152	131
query62	589	526	488	488
query63	227	210	209	209
query64	6010	1824	1852	1824
query65	3611	3455	3462	3455
query66	1334	403	435	403
query67	16134	15886	15501	15501
query68	12511	565	571	565
query69	641	389	372	372
query70	1894	1762	1731	1731
query71	604	315	296	296
query72	5460	3443	3499	3443
query73	2893	338	339	338
query74	7096	6767	6604	6604
query75	5353	2398	2368	2368
query76	6434	999	1240	999
query77	763	373	388	373
query78	9552	9274	8859	8859
query79	1082	536	562	536
query80	567	380	366	366
query81	482	227	226	226
query82	243	134	124	124
query83	193	149	154	149
query84	256	58	58	58
query85	953	303	273	273
query86	447	450	461	450
query87	3651	3526	3445	3445
query88	3213	2542	2546	2542
query89	392	309	345	309
query90	1942	282	278	278
query91	134	98	100	98
query92	108	99	97	97
query93	1162	433	494	433
query94	890	257	262	257
query95	581	522	529	522
query96	641	345	348	345
query97	4487	4262	4273	4262
query98	259	238	225	225
query99	1193	897	919	897
Total cold run time: 304325 ms
Total hot run time: 187976 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.94 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.4 seconds inserted 10000000 Rows, about 364K ops/s
storage size: 17183830965 Bytes

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 9, 2024
Copy link
Contributor

github-actions bot commented Jan 9, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 9, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit b828219 into apache:master Jan 9, 2024
17 of 18 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4 dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants