Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement](java-udf) java-udf module split to sub modules #20185

Merged
merged 1 commit into from
Jun 13, 2023

Conversation

lexluo09
Copy link
Contributor

Proposed changes

Issue Number: close #19977

The java-udf module has become increasingly large and difficult to manage, making it inconvenient to package and use as needed. It needs to be split into multiple sub-modules, such as : java-commom、java-udf、jdbc-scanner、hudi-scanner、 paimon-scanner.

Further comments

the java-commom project Includes jni and common function.

@github-actions github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label May 29, 2023
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/doris/udf/JniUtil
Caused by: java.lang.ClassNotFoundException: org.apache.doris.udf.JniUtil
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/doris/common/jni/utils/JniUtil
Caused by: java.lang.ClassNotFoundException: org.apache.doris.common.jni.utils.JniUtil
```
需要从官网下载 `apache-doris-java-udf-jar-with-dependencies-1.2.0` 的 Java UDF 函数依赖包,放到 BE 安装目录下的 lib 目录,然后重新启动 BE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this jar need to be renamed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

en ,yes

@DongLiang-0
Copy link
Contributor

You need to copy the entire be-java-extensions jar package after successful construction to the lib directory of be in build.sh.

@lexluo09
Copy link
Contributor Author

You need to copy the entire be-java-extensions jar package after successful construction to the lib directory of be in build.sh.
Yes, thank you for your suggestion.

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@lexluo09 lexluo09 force-pushed the java_udf_split branch 2 times, most recently from 0eaa56d to 633487f Compare May 30, 2023 13:38
@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors
'shellcheck ' found no issues.

shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -580,7 +580,6 @@
     cd "${DORIS_HOME}"
 fi
 
-
 # Clean and prepare output dir
 DORIS_OUTPUT=${DORIS_OUTPUT:="${DORIS_HOME}/output/"}
 echo "OUTPUT DIR=${DORIS_OUTPUT}"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors
'shellcheck ' found no issues.

shfmt errors

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -579,7 +579,6 @@
     cd "${DORIS_HOME}"
 fi
 
-
 # Clean and prepare output dir
 DORIS_OUTPUT=${DORIS_OUTPUT:="${DORIS_HOME}/output/"}
 echo "OUTPUT DIR=${DORIS_OUTPUT}"
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename


@lexluo09 lexluo09 force-pushed the java_udf_split branch 2 times, most recently from ceaa772 to 5fcb003 Compare May 30, 2023 14:38
@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build.sh line 106:
    pushd "${DORIS_HOME}/fe/be-java-extensions"
    ^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).


In build.sh line 107:
    "${MVN_CMD}" clean
    ^----------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).


In build.sh line 108:
    popd
    ^--^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).

For more information:
  https://www.shellcheck.net/wiki/SC2317 -- Command appears to be unreachable...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

@lexluo09 lexluo09 force-pushed the java_udf_split branch 2 times, most recently from 321ebd6 to 066a4ef Compare May 30, 2023 14:45
@github-actions
Copy link
Contributor

sh-checker report

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build.sh line 106:
    pushd "${DORIS_HOME}/fe/be-java-extensions"
    ^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).


In build.sh line 107:
    "${MVN_CMD}" clean
    ^----------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).


In build.sh line 108:
    popd
    ^--^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).

For more information:
  https://www.shellcheck.net/wiki/SC2317 -- Command appears to be unreachable...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.



shfmt errors
'shfmt ' found no issues.

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@@ -38,7 +38,7 @@
#include "vec/core/block.h"
#include "vec/data_types/data_type_nullable.h"

const char* EXECUTOR_CLASS = "org/apache/doris/udf/UdfExecutor";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here seems error, udaf is under be/src/vec/aggregate_functions/aggregate_function_java_udaf.h

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks. I will fix it.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2023

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2023

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@morningman
Copy link
Contributor

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@lexluo09
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@morningman
Copy link
Contributor

run buildall

morningman
morningman previously approved these changes Jun 12, 2023
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 12, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 12, 2023
@morningman
Copy link
Contributor

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 57656b2 into apache:master Jun 13, 2023
morningman added a commit that referenced this pull request Jun 13, 2023
#20740)

Follow #20185
Move all BE java extension jars to `be/lib/java_extensions/` dir.
Also remove `udf` dir, used for BE native udf, which is deprecated since v1.2

The final output is:

```
output
├── be
│   ├── bin
│   ├── conf
│   ├── dict
│   ├── lib
|   ├── java_extensions
│       ├── hudi-scanner-jar-with-dependencies.jar
│       ├── java-udf-jar-with-dependencies.jar
│       ├── jdbc-scanner-jar-with-dependencies.jar
│       ├── max-compute-scanner-jar-with-dependencies.jar
│       └── paimon-scanner-jar-with-dependencies.jar
│   ├── LICENSE-dist.txt
│   ├── licenses
│   ├── log
│   ├── NOTICE.txt
│   ├── storage
│   └── www
└── fe
    ├── bin
    ├── conf
    ├── doris-meta
    ├── lib
    ├── LICENSE-dist.txt
    ├── licenses
    ├── log
    ├── mysql_ssl_default_certificate
    ├── NOTICE.txt
    ├── spark-dpp
    └── webroot
```
airborne12 pushed a commit to airborne12/apache-doris that referenced this pull request Jun 13, 2023
apache#20740)

Follow apache#20185
Move all BE java extension jars to `be/lib/java_extensions/` dir.
Also remove `udf` dir, used for BE native udf, which is deprecated since v1.2

The final output is:

```
output
├── be
│   ├── bin
│   ├── conf
│   ├── dict
│   ├── lib
|   ├── java_extensions
│       ├── hudi-scanner-jar-with-dependencies.jar
│       ├── java-udf-jar-with-dependencies.jar
│       ├── jdbc-scanner-jar-with-dependencies.jar
│       ├── max-compute-scanner-jar-with-dependencies.jar
│       └── paimon-scanner-jar-with-dependencies.jar
│   ├── LICENSE-dist.txt
│   ├── licenses
│   ├── log
│   ├── NOTICE.txt
│   ├── storage
│   └── www
└── fe
    ├── bin
    ├── conf
    ├── doris-meta
    ├── lib
    ├── LICENSE-dist.txt
    ├── licenses
    ├── log
    ├── mysql_ssl_default_certificate
    ├── NOTICE.txt
    ├── spark-dpp
    └── webroot
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev/2.0-beta-merged kind/docs Categorizes issue or PR as related to documentation. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] java-udf module split to sub modules
6 participants