Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvemnt](script) support custom lib dir to save custom libs #23887

Merged
merged 1 commit into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions bin/start_be.sh
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,13 @@ if [[ -d "${DORIS_HOME}/lib/hadoop_hdfs/" ]]; then
done
fi

# add custome_libs to CLASSPATH
if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
for f in "${DORIS_HOME}/custom_lib"/*.jar; do
DORIS_CLASSPATH="${DORIS_CLASSPATH}:${f}"
done
fi

if [[ -n "${HADOOP_CONF_DIR}" ]]; then
export DORIS_CLASSPATH="${DORIS_CLASSPATH}:${HADOOP_CONF_DIR}"
fi
Expand Down
7 changes: 7 additions & 0 deletions bin/start_fe.sh
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,13 @@ for f in "${DORIS_HOME}/lib"/*.jar; do
CLASSPATH="${f}:${CLASSPATH}"
done

# add custome_libs to CLASSPATH
if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
for f in "${DORIS_HOME}/custom_lib"/*.jar; do
CLASSPATH="${f}:${CLASSPATH}"
done
fi

# make sure the doris-fe.jar is at first order, so that some classed
# with same qualified name can be loaded priority from doris-fe.jar
CLASSPATH="${DORIS_FE_JAR}:${CLASSPATH}"
Expand Down
2 changes: 2 additions & 0 deletions docs/en/docs/lakehouse/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ under the License.

Search in the hadoop environment hadoop-lzo-*.jar, and put it under "${DORIS_HOME}/fe/lib/",then restart fe.

Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.

## HDFS

1. What to do with the`java.lang.VerifyError: xxx` error when accessing HDFS 3.x?
Expand Down
4 changes: 3 additions & 1 deletion docs/en/docs/lakehouse/multi-catalog/jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -791,7 +791,9 @@ DROP CATALOG <catalog_name>;

12. `Non supported character set (add orai18n.jar in your classpath): ZHS16GBK` exception occurs when reading Oracle

Download [orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html) and put it in the lib directory of Doris FE and the lib/java_extensions directory of BE (Doris versions before 2.0 need to be placed in the lib directory of BE).
Download [orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html) and put it in the lib directory of Doris FE and the `lib/java_extensions/` directory of BE (Doris versions before 2.0 need to be placed in the lib directory of BE).

Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.

13. `NoClassDefFoundError: net/jpountz/lz4/LZ4Factory` exception occurs when reading Clickhouse data via jdbc catalog.

Expand Down
5 changes: 4 additions & 1 deletion docs/en/docs/lakehouse/multi-catalog/paimon.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (

#### S3

>Note that.
> Note that.
>
> user need download [paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
>
> Place it in directory ${DORIS_HOME}/be/lib/java_extensions/preload-extensions and restart be
>
> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/` directory (if it does not exist, just create it manually) to prevent the file from being lost due to the replacement of the lib directory when upgrading the cluster.

```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
Expand Down
2 changes: 2 additions & 0 deletions docs/zh-CN/docs/lakehouse/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,8 @@ under the License.

去hadoop环境搜索`hadoop-lzo-*.jar`放在`"${DORIS_HOME}/fe/lib/"`目录下并重启fe。

从 2.0.2 版本起,可以将这个文件放置在BE的 `custom_lib/` 目录下(如不存在,手动创建即可),以防止升级集群时因为 lib 目录被替换而导致文件丢失。

## HDFS

1. 访问 HDFS 3.x 时报错:`java.lang.VerifyError: xxx`
Expand Down
2 changes: 2 additions & 0 deletions docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -788,6 +788,8 @@ DROP CATALOG <catalog_name>;

下载 [orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html) 并放到 Doris FE 的 lib 目录以及 BE 的 lib/java_extensions 目录 (Doris 2.0 之前的版本需放到 BE 的 lib 目录下) 下即可。

从 2.0.2 版本起,可以将这个文件放置在BE的 `custom_lib/` 目录下(如不存在,手动创建即可),以防止升级集群时因为 lib 目录被替换而导致文件丢失。

13. 通过jdbc catalog 读取Clickhouse数据出现`NoClassDefFoundError: net/jpountz/lz4/LZ4Factory` 错误信息

可以先下载[lz4-1.3.0.jar](https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar)包,然后放到DorisFE lib 目录以及BE 的 `lib/lib/java_extensions`目录中(Doris 2.0 之前的版本需放到 BE 的 lib 目录下)。
7 changes: 5 additions & 2 deletions docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (

#### S3

>注意:
> 注意:
>
> 用户需要手动下载[paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be

> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be。
>
> 从 2.0.2 版本起,可以将这个文件放置在BE的 `custom_lib/` 目录下(如不存在,手动创建即可),以防止升级集群时因为 lib 目录被替换而导致文件丢失。

```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
Expand Down