[Improve](build-script)Add be-extension-ignore to ignore avro-scanner when building release packages

[DongLiang-0]

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[DongLiang-0]

run buildall

[github-actions]

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build.sh line 494:
    if [[ -n "$BE_EXTENSION_IGNORE" ]]; then
              ^------------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ -n "${BE_EXTENSION_IGNORE}" ]]; then


In build.sh line 495:
        IFS=',' read -r -a ignore_modules <<< "$BE_EXTENSION_IGNORE"
                                               ^------------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        IFS=',' read -r -a ignore_modules <<< "${BE_EXTENSION_IGNORE}"


In build.sh line 497:
            modules=("${modules[@]/be-java-extensions\/$module}")
                                                       ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            modules=("${modules[@]/be-java-extensions\/${module}}")


In build.sh line 781:
    if [[ -n "$BE_EXTENSION_IGNORE" ]]; then
              ^------------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ -n "${BE_EXTENSION_IGNORE}" ]]; then


In build.sh line 782:
        IFS=',' read -r -a ignore_modules <<< "$BE_EXTENSION_IGNORE"
                                               ^------------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        IFS=',' read -r -a ignore_modules <<< "${BE_EXTENSION_IGNORE}"


In build.sh line 786:
            if [[ -n "$module" ]]; then
                      ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            if [[ -n "${module}" ]]; then


In build.sh line 789:
                    if [[ "$module" == "$ignore_module" ]]; then
                           ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                        ^------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
                    if [[ "${module}" == "${ignore_module}" ]]; then


In build.sh line 794:
                if [[ "$ignore" -eq 0 ]]; then
                       ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
                if [[ "${ignore}" -eq 0 ]]; then


In build.sh line 795:
                    new_modules+=("$module")
                                   ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
                    new_modules+=("${module}")

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -492,9 +492,9 @@
 
     # If the BE_EXTENSION_IGNORE variable is not empty, remove the modules that need to be ignored from FE_MODULES
     if [[ -n "$BE_EXTENSION_IGNORE" ]]; then
-        IFS=',' read -r -a ignore_modules <<< "$BE_EXTENSION_IGNORE"
+        IFS=',' read -r -a ignore_modules <<<"$BE_EXTENSION_IGNORE"
         for module in "${ignore_modules[@]}"; do
-            modules=("${modules[@]/be-java-extensions\/$module}")
+            modules=("${modules[@]/be-java-extensions\/$module/}")
         done
     fi
 fi
@@ -779,7 +779,7 @@
     extensions_modules+=("preload-extensions")
 
     if [[ -n "$BE_EXTENSION_IGNORE" ]]; then
-        IFS=',' read -r -a ignore_modules <<< "$BE_EXTENSION_IGNORE"
+        IFS=',' read -r -a ignore_modules <<<"$BE_EXTENSION_IGNORE"
         new_modules=()
         for module in "${extensions_modules[@]}"; do
             module=${module// /}
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

[github-actions]

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -492,9 +492,9 @@
 
     # If the BE_EXTENSION_IGNORE variable is not empty, remove the modules that need to be ignored from FE_MODULES
     if [[ -n "${BE_EXTENSION_IGNORE}" ]]; then
-        IFS=',' read -r -a ignore_modules <<< "${BE_EXTENSION_IGNORE}"
+        IFS=',' read -r -a ignore_modules <<<"${BE_EXTENSION_IGNORE}"
         for module in "${ignore_modules[@]}"; do
-            modules=("${modules[@]/be-java-extensions\/${module}}")
+            modules=("${modules[@]/be-java-extensions\/${module}/}")
         done
     fi
 fi
@@ -779,7 +779,7 @@
     extensions_modules+=("preload-extensions")
 
     if [[ -n "${BE_EXTENSION_IGNORE}" ]]; then
-        IFS=',' read -r -a ignore_modules <<< "${BE_EXTENSION_IGNORE}"
+        IFS=',' read -r -a ignore_modules <<<"${BE_EXTENSION_IGNORE}"
         new_modules=()
         for module in "${extensions_modules[@]}"; do
             module=${module// /}
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

[DongLiang-0]

run buildall

[doris-robot]

TPC-H: Total hot run time: 38333 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67c040079a153ec16f93ac3b21b4415620b03ac8, data reload: false

------ Round 1 ----------------------------------
q1	5753	4207	4157	4157
q2	1177	161	155	155
q3	9485	1130	1161	1130
q4	6230	756	797	756
q5	3253	3194	3048	3048
q6	202	124	128	124
q7	1083	597	598	597
q8	1937	1985	2029	1985
q9	6631	6687	6612	6612
q10	3591	3575	3612	3575
q11	343	236	221	221
q12	364	208	204	204
q13	17513	2846	2846	2846
q14	238	199	217	199
q15	512	458	459	458
q16	467	377	377	377
q17	961	531	549	531
q18	7213	6529	6449	6449
q19	1516	1435	1454	1435
q20	465	253	271	253
q21	3533	2923	3020	2923
q22	352	298	310	298
Total cold run time: 72819 ms
Total hot run time: 38333 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4093	4079	4079
q2	330	233	234	233
q3	3014	2871	2880	2871
q4	1902	1521	1591	1521
q5	5311	5383	5369	5369
q6	201	118	119	118
q7	2276	1861	1854	1854
q8	3184	3321	3296	3296
q9	8752	8698	8701	8698
q10	3836	3782	3753	3753
q11	554	444	443	443
q12	727	513	562	513
q13	16885	2859	2855	2855
q14	271	245	250	245
q15	492	465	462	462
q16	485	441	415	415
q17	1734	1517	1477	1477
q18	7564	7356	7274	7274
q19	1602	1538	1521	1521
q20	1915	1745	1731	1731
q21	4831	4701	4657	4657
q22	531	455	478	455
Total cold run time: 70488 ms
Total hot run time: 53840 ms

[doris-robot]

TPC-DS: Total hot run time: 187129 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67c040079a153ec16f93ac3b21b4415620b03ac8, data reload: false

query1	958	361	354	354
query2	7390	2033	1959	1959
query3	6705	215	219	215
query4	31897	21244	21295	21244
query5	4335	495	419	419
query6	300	190	182	182
query7	4619	291	281	281
query8	228	175	183	175
query9	9293	2364	2332	2332
query10	572	237	257	237
query11	17360	14575	14315	14315
query12	142	99	88	88
query13	1633	419	429	419
query14	14203	11590	11251	11251
query15	268	201	194	194
query16	8226	275	269	269
query17	1958	601	548	548
query18	2082	292	287	287
query19	347	152	162	152
query20	97	89	91	89
query21	203	128	131	128
query22	4987	4869	4845	4845
query23	33570	32805	32858	32805
query24	12116	2911	2855	2855
query25	667	412	385	385
query26	1761	160	166	160
query27	2968	350	370	350
query28	7486	1938	1930	1930
query29	1037	672	632	632
query30	311	149	150	149
query31	953	730	746	730
query32	100	59	54	54
query33	776	261	257	257
query34	978	493	485	485
query35	855	610	618	610
query36	1051	911	923	911
query37	270	76	82	76
query38	3521	3446	3423	3423
query39	1484	1469	1429	1429
query40	289	114	111	111
query41	52	49	48	48
query42	105	97	93	93
query43	486	445	449	445
query44	1250	744	728	728
query45	289	259	264	259
query46	1108	727	707	707
query47	1937	1862	1849	1849
query48	463	356	378	356
query49	1234	336	342	336
query50	759	374	382	374
query51	6702	6764	6591	6591
query52	114	93	87	87
query53	350	267	274	267
query54	330	247	244	244
query55	85	80	80	80
query56	252	236	235	235
query57	1240	1150	1161	1150
query58	241	229	212	212
query59	2746	2610	2545	2545
query60	266	252	259	252
query61	130	130	112	112
query62	654	458	460	458
query63	315	281	279	279
query64	6934	4085	4023	4023
query65	3150	3025	3040	3025
query66	1424	370	358	358
query67	15560	14828	15149	14828
query68	9005	548	544	544
query69	701	384	388	384
query70	1421	1163	1193	1163
query71	521	282	269	269
query72	6820	2714	2561	2561
query73	1596	316	311	311
query74	7397	6708	6613	6613
query75	4463	2836	2835	2835
query76	5661	894	909	894
query77	641	269	254	254
query78	11066	10252	9996	9996
query79	9651	521	539	521
query80	1597	396	390	390
query81	521	209	220	209
query82	367	206	207	206
query83	216	149	148	148
query84	285	76	82	76
query85	1083	312	307	307
query86	352	276	297	276
query87	3711	3586	3525	3525
query88	4820	2357	2367	2357
query89	489	374	369	369
query90	2063	173	172	172
query91	171	135	152	135
query92	61	48	49	48
query93	6486	525	495	495
query94	1344	180	178	178
query95	436	341	338	338
query96	610	267	275	267
query97	3037	2892	2887	2887
query98	232	216	213	213
query99	1150	912	912	912
Total cold run time: 324146 ms
Total hot run time: 187129 ms

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 67c040079a153ec16f93ac3b21b4415620b03ac8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.