[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost #15972

HackToday · 2023-01-16T08:21:39Z

Signed-off-by: Wu, Kaiqiang kaiqiang.wu@intel.com
Co-authored-by: vesslanjin jun.i.jin@intel.com

Proposed changes

Issue Number: close #xxx

Problem summary

As AVX512 is available in most modern processors, it is good to use them if have performance boost.
In latest bitshuffle, AVX512 have been added. We could make it integrated in doris for AVX512 case.

Tested with master branch, queries(SSB query q1.1.sql~q4.3.sql total 13 queries) can be boost from 1.4%~3.2%. (use run-ssb-queries.sh 5 times, each time with 100 iterations.)

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-01-16T08:22:33Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1055:
	if [ "$arch" == "avx512" ]; then
           ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
	if [[ "${arch}" == "avx512" ]]; then


In thirdparty/build-thirdparty.sh line 1083:
        elif [ "$arch" == "avx512" ]; then
             ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        elif [[ "${arch}" == "avx512" ]]; then


In thirdparty/build-thirdparty.sh line 1085:
            "${DORIS_BIN_UTILS}/nm" --defined-only --extern-only $tmp_obj | while read -r addr type sym ; do
                                                                 ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                 ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            "${DORIS_BIN_UTILS}/nm" --defined-only --extern-only "$tmp_obj" | while read -r addr type sym ; do


In thirdparty/build-thirdparty.sh line 1086:
                echo ${sym} ${sym}_${arch}
                     ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                            ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                   ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
                echo "${sym}" "${sym}"_"${arch}"


In thirdparty/build-thirdparty.sh line 1088:
            "${DORIS_BIN_UTILS}/objcopy" --redefine-syms=renames.txt $tmp_obj $dst_obj
                                                                     ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                     ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                              ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                              ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            "${DORIS_BIN_UTILS}/objcopy" --redefine-syms=renames.txt "$tmp_obj" $dst_obj

For more information:
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
  https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- thirdparty/build-thirdparty.sh.orig
+++ thirdparty/build-thirdparty.sh
@@ -1052,8 +1052,8 @@
         if [[ "${arch}" == "avx2" ]]; then
             arch_flag="-mavx2"
         fi
-	if [ "$arch" == "avx512" ]; then
-           arch_flag="-mavx512bw -mavx512f"
+        if [ "$arch" == "avx512" ]; then
+            arch_flag="-mavx512bw -mavx512f"
         fi
         tmp_obj="bitshuffle_${arch}_tmp.o"
         dst_obj="bitshuffle_${arch}.o"
@@ -1082,9 +1082,9 @@
             "${objcopy}" --redefine-syms=renames.txt "${tmp_obj}" "${dst_obj}"
         elif [ "$arch" == "avx512" ]; then
             # Create a mapping file with '<old_sym> <suffixed_sym>' on each line.
-            "${DORIS_BIN_UTILS}/nm" --defined-only --extern-only $tmp_obj | while read -r addr type sym ; do
+            "${DORIS_BIN_UTILS}/nm" --defined-only --extern-only $tmp_obj | while read -r addr type sym; do
                 echo ${sym} ${sym}_${arch}
-            done > renames.txt
+            done >renames.txt
             "${DORIS_BIN_UTILS}/objcopy" --redefine-syms=renames.txt $tmp_obj $dst_obj
         else
             mv "${tmp_obj}" "${dst_obj}"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions

clang-tidy made some suggestions

be/src/gutil/cpu.cc

github-actions · 2023-01-16T08:41:02Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1055:
	if [ "$arch" == "avx512" ]; then
           ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
	if [[ "${arch}" == "avx512" ]]; then

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
  https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- thirdparty/build-thirdparty.sh.orig
+++ thirdparty/build-thirdparty.sh
@@ -1052,8 +1052,8 @@
         if [[ "${arch}" == "avx2" ]]; then
             arch_flag="-mavx2"
         fi
-	if [ "$arch" == "avx512" ]; then
-           arch_flag="-mavx512bw -mavx512f"
+        if [ "$arch" == "avx512" ]; then
+            arch_flag="-mavx512bw -mavx512f"
         fi
         tmp_obj="bitshuffle_${arch}_tmp.o"
         dst_obj="bitshuffle_${arch}.o"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2023-01-16T08:51:29Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1055:
        if [ "$arch" == "avx512" ]; then
           ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ "${arch}" == "avx512" ]]; then

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
  https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-01-16T08:53:43Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/build-thirdparty.sh line 1055:
        if [[ "$arch" == "avx512" ]]; then
               ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ "${arch}" == "avx512" ]]; then

For more information:
  https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

HackToday · 2023-01-16T09:06:57Z

@morningman Could you help make the workflow run ? It seems need maintainers' approval

be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp

adonis0147 · 2023-01-16T09:11:04Z

@morningman Could you help make the workflow run ? It seems need maintainers' approval

Done.

adonis0147 · 2023-01-16T09:22:04Z

Hi @HackToday , thanks for your contribution. Some workflows failed because we didn't build the third-party libraries in workflow. We fetched the docker image of pre-built third-party libraries and built the project against the libraries.

In order to fix these checks, you should split this pull request into two pieces. One is for upgrading the package bitshuffle and another is for the modification of BE.

HackToday · 2023-01-16T14:29:57Z

Hi @adonis0147 do you mean two separate PRs ? One PR for upgrading bitshuffle, and another for changing BE (until first PR merged)?
Thanks

adonis0147 · 2023-01-16T14:31:56Z

Hi @adonis0147 do you mean two separate PRs ? One PR for upgrading bitshuffle, and another for changing BE (until first PR merged)? Thanks

Yes.

HackToday · 2023-01-17T01:22:55Z

@adonis0147 The first split PR is here: #15993

In order to use AVX512 instructions. See #15972

HackToday · 2023-01-18T03:53:24Z

@adonis0147 hi, I am not sure the difference in Doris github run vs host run.

In my local env, I can build and run ut successfully.

The attached file is ut run in my local host.

run-ut-test.txt

but doris github in teamcity would complain with error

http://43.132.222.7:8111/viewLog.html?buildId=82706&buildTypeId=Doris_DorisBeUt_BeUt&tab=buildLog&_focus=260

adonis0147 · 2023-01-18T04:10:05Z

@adonis0147 hi, I am not sure the difference in Doris github run vs host run.

In my local env, I can build and run ut successfully.

The attached file is ut run in my local host.

run-ut-test.txt

but doris github in teamcity would complain with error

http://43.132.222.7:8111/viewLog.html?buildId=82706&buildTypeId=Doris_DorisBeUt_BeUt&tab=buildLog&_focus=260

Hi @HackToday , sorry to confuse you. The source of the pre-built third-party libraries which GitHub workflows use differs from the one TeamCity workflows use.

GitHub workflows use the pre-built third-party libraries from Apache Doris Third Party Prebuilt and these libraries are up to date. However, TeamCity workflows use ones from the Docker image apache/doris:build-env-ldb-toolchain-latest and they are obsolete now. That is the reason why your checks failed.

In short, just wait patiently and I will let you re-trigger all these checks when everything is ready.

By the way, would you like resolve the previous comments?

HackToday · 2023-01-18T05:46:00Z

hi @adonis0147 Have resolved. Please help check. Thanks.

adonis0147 · 2023-01-28T02:11:33Z

Hi @HackToday , the Docker image has been updated. Please rebase the branch on master to re-trigger the checks.

hello-stephen · 2023-01-29T01:04:23Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.9 seconds
load time: 524 seconds
storage size: 17122602409 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230129092056_clickbench_pr_86118.html

HackToday · 2023-01-29T02:19:02Z

hi @adonis0147 Have rebased. Not quite sure the Po regression failed reason:
http://43.132.222.7:8111/viewLog.html?buildId=85876&tab=buildResultsDiv&buildTypeId=Doris_DorisRegression_P0Regression#testNameId-4553081828222454766

Caused by: java.lang.AssertionError: Expect exception msg contains 'null', but meet 'java.sql.SQLException: errCode = 2, detailMessage = There is no scanNode Backend available.[10003: not alive]'

It seems not related. How to rerun the P0 case or something else ?

adonis0147 · 2023-01-29T02:30:39Z

hi @adonis0147 Have rebased. Not quite sure the Po regression failed reason: http://43.132.222.7:8111/viewLog.html?buildId=85876&tab=buildResultsDiv&buildTypeId=Doris_DorisRegression_P0Regression#testNameId-4553081828222454766

Caused by: java.lang.AssertionError: Expect exception msg contains 'null', but meet 'java.sql.SQLException: errCode = 2, detailMessage = There is no scanNode Backend available.[10003: not alive]'

It seems not related. How to rerun the P0 case or something else ?

Hi @HackToday , these errors indicated that the workflow built the codebase successfully, but some cases failed. They are irrelevant to this PR and we are resolving them now.

adonis0147 · 2023-01-29T06:23:04Z

Hi @HackToday , it seems that we have resolved the issues with the workflows. Please rebase the branch on the latest master to trigger the checks again.

…ormance boost Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com> Co-authored-by: vesslanjin <jun.i.jin@intel.com>

adonis0147

LGTM

github-actions · 2023-01-29T08:43:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-01-29T08:43:33Z

PR approved by anyone and no changes requested.

In order to use AVX512 instructions. See apache#15972

…ormance boost (apache#15972) As AVX512 is available in most modern processors, it is good to use them if have performance boost. In latest bitshuffle, AVX512 have been added. We could make it integrated in doris for AVX512 case. Tested with master branch, queries(SSB query q1.1.sql~q4.3.sql total 13 queries) can be boost from 1.4%~3.2%. (use run-ssb-queries.sh 5 times, each time with 100 iterations.) Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com> Co-authored-by: vesslanjin <jun.i.jin@intel.com>

xiaokang · 2023-09-10T10:06:20Z

@HackToday A avx512 related bug #24145 is reported. Would you analyse and fix this problem?

…for performance boost (#15972)" This reverts commit 28fcc09.

xiaokang · 2023-09-10T10:10:56Z

Revert this PR and wait for further fix.

…for performance boost (#15972)" (#24146) This reverts commit 28fcc09.

HackToday · 2023-09-11T01:35:27Z

@xiaokang sure, I have leave comment on the issue. I would double check if arch special issue or something else. Thanks

HackToday · 2023-09-11T05:05:39Z

@xiaokang please help leave your comment on the issue (#24145) for some questions. Thanks

…ormance boost (apache#15972) As AVX512 is available in most modern processors, it is good to use them if have performance boost. In latest bitshuffle, AVX512 have been added. We could make it integrated in doris for AVX512 case. Tested with master branch, queries(SSB query q1.1.sql~q4.3.sql total 13 queries) can be boost from 1.4%~3.2%. (use run-ssb-queries.sh 5 times, each time with 100 iterations.) Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com> Co-authored-by: vesslanjin <jun.i.jin@intel.com> Change-Id: Iec7a7e6640ef9c3873b3f966e9c7f42d25d2e73e

github-actions bot reviewed Jan 16, 2023

View reviewed changes

be/src/gutil/cpu.cc Show resolved Hide resolved

HackToday force-pushed the bitshuffle-avx512 branch from 901867e to 9c823b3 Compare January 16, 2023 08:40

HackToday force-pushed the bitshuffle-avx512 branch from 9c823b3 to 82aeafc Compare January 16, 2023 08:50

HackToday force-pushed the bitshuffle-avx512 branch from 82aeafc to 3fb4135 Compare January 16, 2023 08:52

HackToday force-pushed the bitshuffle-avx512 branch from 3fb4135 to ce4c8a7 Compare January 16, 2023 08:54

adonis0147 reviewed Jan 16, 2023

View reviewed changes

be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp Show resolved Hide resolved

HackToday mentioned this pull request Jan 17, 2023

[chore](thirdparty) update bitshuffle from 0.3.5 to 0.5.1 #15993

Merged

13 tasks

adonis0147 pushed a commit that referenced this pull request Jan 17, 2023

[chore](thirdparty) Update bitshuffle from 0.3.5 to 0.5.1 (#15993)

3810727

In order to use AVX512 instructions. See #15972

HackToday force-pushed the bitshuffle-avx512 branch from ce4c8a7 to 7bb62f3 Compare January 18, 2023 01:02

HackToday force-pushed the bitshuffle-avx512 branch from 7bb62f3 to e2cc3d5 Compare January 29, 2023 00:43

HackToday force-pushed the bitshuffle-avx512 branch from e2cc3d5 to faf42d0 Compare January 29, 2023 06:47

[improvement](bitshuffle)Enable avx512 support in bitshuffle for perf…

faf42d0

…ormance boost Signed-off-by: Wu, Kaiqiang <kaiqiang.wu@intel.com> Co-authored-by: vesslanjin <jun.i.jin@intel.com>

adonis0147 approved these changes Jan 29, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2023

github-actions bot added the reviewed label Jan 29, 2023

adonis0147 merged commit 28fcc09 into apache:master Jan 30, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Feb 1, 2023

[chore](thirdparty) Update bitshuffle from 0.3.5 to 0.5.1 (apache#15993)

846d72e

In order to use AVX512 instructions. See apache#15972

xiaokang mentioned this pull request Sep 10, 2023

[Bug] avx512 is misjudge and cause be coredump #24145

Open

3 tasks

xiaokang added a commit that referenced this pull request Sep 10, 2023

Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle …

d2945cb

…for performance boost (#15972)" This reverts commit 28fcc09.

xiaokang mentioned this pull request Sep 10, 2023

Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost" #24146

Merged

zy-kkk pushed a commit that referenced this pull request Sep 10, 2023

Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle …

ebac816

…for performance boost (#15972)" (#24146) This reverts commit 28fcc09.

xiaokang added a commit that referenced this pull request Sep 10, 2023

Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle …

e5dac51

…for performance boost (#15972)" (#24146) This reverts commit 28fcc09.

HackToday mentioned this pull request Oct 30, 2023

[Fix](bitshuffle)Enable avx512 support in bitshuffle for performance boost #25819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost #15972

[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost #15972

HackToday commented Jan 16, 2023 •

edited

Loading

github-actions bot commented Jan 16, 2023

github-actions bot left a comment

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

HackToday commented Jan 16, 2023

adonis0147 commented Jan 16, 2023

adonis0147 commented Jan 16, 2023 •

edited

Loading

HackToday commented Jan 16, 2023 •

edited

Loading

adonis0147 commented Jan 16, 2023

HackToday commented Jan 17, 2023

HackToday commented Jan 18, 2023

adonis0147 commented Jan 18, 2023

HackToday commented Jan 18, 2023

adonis0147 commented Jan 28, 2023

hello-stephen commented Jan 29, 2023 •

edited

Loading

HackToday commented Jan 29, 2023

adonis0147 commented Jan 29, 2023 •

edited

Loading

adonis0147 commented Jan 29, 2023

adonis0147 left a comment

github-actions bot commented Jan 29, 2023

github-actions bot commented Jan 29, 2023

xiaokang commented Sep 10, 2023

xiaokang commented Sep 10, 2023

HackToday commented Sep 11, 2023

HackToday commented Sep 11, 2023

[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost #15972

[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost #15972

Conversation

HackToday commented Jan 16, 2023 • edited Loading

Proposed changes

Problem summary

Checklist(Required)

Further comments

github-actions bot commented Jan 16, 2023

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 16, 2023

sh-checker report

github-actions bot commented Jan 16, 2023

sh-checker report

github-actions bot commented Jan 16, 2023

sh-checker report

HackToday commented Jan 16, 2023

adonis0147 commented Jan 16, 2023

adonis0147 commented Jan 16, 2023 • edited Loading

HackToday commented Jan 16, 2023 • edited Loading

adonis0147 commented Jan 16, 2023

HackToday commented Jan 17, 2023

HackToday commented Jan 18, 2023

adonis0147 commented Jan 18, 2023

HackToday commented Jan 18, 2023

adonis0147 commented Jan 28, 2023

hello-stephen commented Jan 29, 2023 • edited Loading

HackToday commented Jan 29, 2023

adonis0147 commented Jan 29, 2023 • edited Loading

adonis0147 commented Jan 29, 2023

adonis0147 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 29, 2023

github-actions bot commented Jan 29, 2023

xiaokang commented Sep 10, 2023

xiaokang commented Sep 10, 2023

HackToday commented Sep 11, 2023

HackToday commented Sep 11, 2023

HackToday commented Jan 16, 2023 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

adonis0147 commented Jan 16, 2023 •

edited

Loading

HackToday commented Jan 16, 2023 •

edited

Loading

hello-stephen commented Jan 29, 2023 •

edited

Loading

adonis0147 commented Jan 29, 2023 •

edited

Loading