[Feature]Support Alibaba DLF metastore for hive external table #6403

mxdzs0612 · 2022-05-23T07:18:28Z

What type of PR is this：

Which issues of this PR fixes ：

Fixes #

Problem Summary(Required) ：

Data Lake Formation (DLF) is a key component of the cloud-native data lake framework which is widely used on Alibaba Cloud just like AWS Glue, details see https://www.alibabacloud.com/en/product/datalake-formation. This PR allows users to use DLF as metastore in Hive external table.

Usage:

CREATE EXTERNAL CATALOG dlf_hive_catalog
properties
(
    "type" = "hive",
    "hive.metastore.type" = "DLF"
);

Add a config file hive-site.xml to {FE Home DIR}/conf , with following configs:

<?xml version="1.0"?>
<configuration>
    <!--Set to use dlf client-->
    <property>
        <name>hive.metastore.type</name>
        <value>dlf</value>
    </property>
    <!--DLF endpoint, see https://www.alibabacloud.com/help/en/doc-detail/197608.html-->
    <property>
        <name>dlf.catalog.endpoint</name>
        <value>dlf-vpc.cn-xxx.aliyuncs.com</value>
    </property>
    <!--DLF region, see https://www.alibabacloud.com/help/en/doc-detail/197608.html-->
    <property>
        <name>dlf.catalog.region</name>
        <value>cn-beijing</value>
    </property>
    <!--Proxy mode of DLF-->
    <property>
        <name>dlf.catalog.proxyMode</name>
        <value>DLF_ONLY</value>
    </property>
    <!--Access Key mode of DLF-->
    <property>
        <name>dlf.catalog.akMode</name>
        <value>MANUAL</value>
    </property>
    <!--User id of the alibaba cloud account-->
    <property>
        <name>dlf.catalog.uid</name>
        <value>xxxxxx</value>
    </property>
    <!--Access Key ID of DLF, can be omitted if the cluster is created with the same Alibaba Cloud account of DLF-->
    <property>
        <name>dlf.catalog.accessKeyId</name>
        <value>xxxxxx</value>
    </property>
    <!--Access Key secret of DLF, can be omitted if the cluster is created with the same Alibaba Cloud account of DLF-->
    <property>
        <name>dlf.catalog.accessKeySecret</name>
        <value>xxxxxx</value>
    </property>
</configuration>

stephen-shelby · 2022-05-23T16:08:29Z

fe/fe-core/pom.xml

@@ -465,10 +465,30 @@ under the License.

        <!-- https://mvnrepository.com/artifact/com.facebook.presto.hive/hive-apache -->
        <dependency>
-            <groupId>com.facebook.presto.hive</groupId>
+            <groupId>io.trino.hive</groupId>


Please run the regression test.

stephen-shelby · 2022-05-23T16:14:36Z

fe/fe-core/src/main/java/com/starrocks/external/hive/HiveMetaClient.java

@@ -100,6 +101,7 @@ public class HiveMetaClient {

    public HiveMetaClient(String uris) throws DdlException {
        HiveConf conf = new HiveConf();
+        conf.addResource(new Path("file:///" + StarRocksFE.STARROCKS_HOME_DIR + "/conf/hive-site.xml"));


$FE/conf will be used as the classpath of fe. I think it is no necessary to add this?

stephen-shelby · 2022-05-23T16:17:55Z

fe/fe-core/src/main/java/com/starrocks/external/hive/HiveMetaClient.java

@@ -119,8 +121,13 @@ public class AutoCloseClient implements AutoCloseable {
        private final IMetaStoreClient hiveClient;

        private AutoCloseClient(HiveConf conf) throws MetaException {
-            hiveClient = RetryingMetaStoreClient.getProxy(conf, dummyHookLoader,
-                    HiveMetaStoreThriftClient.class.getName());
+            if ("dlf".equalsIgnoreCase(conf.get("hive.metastore"))) {


Where is the 'hive.metastore' configuration defined ? Could the name be modified? I think the 'hive.metastore' means too much

stephen-shelby · 2022-05-23T16:20:27Z

fe/pom.xml

+            <dependency>
+                <groupId>com.aliyun.datalake</groupId>
+                <artifactId>metastore-client-hive3</artifactId>
+                <version>0.2.14</version>


plz define the ${metastore-client-hive3.version} in $STARROCKS/pom.xml

stephen-shelby · 2022-05-23T16:22:21Z

fe/fe-core/src/main/java/com/starrocks/external/hive/HiveMetaStoreThriftClient.java

@@ -723,6 +723,12 @@ public void createCatalog(Catalog catalog)
        throw new TException("method not implemented");
    }

+    @Override
+    public void alterCatalog(String s, Catalog catalog) throws NoSuchObjectException, InvalidObjectException,


Why add this interface

stephen-shelby · 2022-05-23T16:25:35Z

fe/fe-core/src/main/java/com/starrocks/external/hive/ProxyMetaStoreClient.java

+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class ProxyMetaStoreClient implements IMetaStoreClient {


Is this only used by DLF?

stephen-shelby · 2022-05-23T16:27:03Z

fe/fe-core/src/main/java/com/starrocks/external/hive/ProxyMetaStoreClient.java

+
+public class ProxyMetaStoreClient implements IMetaStoreClient {
+    private static final Logger logger =
+            LoggerFactory.getLogger(com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient.class);


pom use metastore-client-hive3 , but here we use hive2. is it ok?

stephen-shelby · 2022-05-24T11:22:16Z

fe/fe-core/src/main/java/com/starrocks/external/hive/ProxyMetaStoreClient.java

@@ -0,0 +1,2478 @@
+// This file is licensed under the Elastic License 2.0. Copyright 2021-present, StarRocks Limited.
+// This file is based on code available under the Apache license here:
+//   https://github.com/aliyun/datalake-catalog-metastore-client/blob/master/metastore-client-hive/metastore-client-hive3/src/main/java/com/aliyun/datalake/metastore/hive2/ProxyMetaStoreClient.java


@imay is license ok ?

mxdzs0612 · 2022-05-27T06:44:26Z

run starrocks_fe_unittest

wanpengfei-git · 2022-05-27T06:50:04Z

[FE PR Coverage check]

😞 fail : 3 / 6 (50.00%)

file detail

	path	covered line	new line	coverage
🔵	com/starrocks/external/hive/HiveMetaStoreThriftClient.java	0	1	00.00%
🔵	com/starrocks/external/hive/HiveMetaClient.java	3	5	60.00%

stephen-shelby

LGTM

…ocks#6403) Data Lake Formation (DLF) is a key component of the cloud-native data lake framework which is widely used on Alibaba Cloud just like AWS Glue, details see https://www.alibabacloud.com/en/product/datalake-formation. This PR allows users to use DLF as metastore in Hive external table. Usage: Add a config file hive-site.xml to {FE Home DIR}/conf , with following configs: <?xml version="1.0"?> <configuration>  <property> <name>hive.metastore.type</name> <value>dlf</value> </property>  <property> <name>dlf.catalog.endpoint</name> <value>dlf-vpc.cn-beijing.aliyuncs.com</value> </property>  <property> <name>dlf.catalog.region</name> <value>cn-beijing</value> </property>  <property> <name>dlf.catalog.proxyMode</name> <value>DLF_ONLY</value> </property>  <property> <name>dlf.catalog.akMode</name> <value>EMR_AUTO</value> </property>  <property> <name>dlf.catalog.uid</name> <value>xxxxxx</value> </property>  <property> <name>dlf.catalog.accessKeyId</name> <value>xxxxxx</value> </property>  <property> <name>dlf.catalog.accessKeySecret</name> <value>xxxxxx</value> </property> </configuration>

Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com>

Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com> (cherry picked from commit 40daf76) Co-authored-by: 絵空事スピリット <wanglichen@starrocks.com>

mxdzs0612 marked this pull request as ready for review May 23, 2022 07:40

mxdzs0612 changed the title ~~Support Alibaba DLF for hive external table.~~ Support Alibaba DLF metastore for hive external table. May 23, 2022

mxdzs0612 changed the title ~~Support Alibaba DLF metastore for hive external table.~~ Support Alibaba DLF metastore for hive external table May 23, 2022

stephen-shelby reviewed May 23, 2022

View reviewed changes

stephen-shelby reviewed May 24, 2022

View reviewed changes

mxdzs0612 added 3 commits May 27, 2022 11:12

Support Alibaba DLF for hive external table.

8e74872

Fix code style.

87471b3

Optimize the code.

c6648a0

mxdzs0612 force-pushed the dlf branch from 72f23ff to c6648a0 Compare May 27, 2022 03:29

stephen-shelby reviewed May 27, 2022

View reviewed changes

stephen-shelby approved these changes May 27, 2022

View reviewed changes

gengjun-git approved these changes May 30, 2022

View reviewed changes

caneGuy approved these changes May 30, 2022

View reviewed changes

caneGuy changed the title ~~Support Alibaba DLF metastore for hive external table~~ [Feature]Support Alibaba DLF metastore for hive external table May 30, 2022

caneGuy merged commit 03e6ed0 into StarRocks:main May 30, 2022

mxdzs0612 deleted the dlf branch May 30, 2022 08:14

jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023

[Doc] Replace an outer link (StarRocks#6403)

40daf76

Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]Support Alibaba DLF metastore for hive external table #6403

[Feature]Support Alibaba DLF metastore for hive external table #6403

mxdzs0612 commented May 23, 2022 •

edited by miomiocat

Loading

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 23, 2022

stephen-shelby May 24, 2022

mxdzs0612 commented May 27, 2022

wanpengfei-git commented May 27, 2022

stephen-shelby left a comment

[Feature]Support Alibaba DLF metastore for hive external table #6403

[Feature]Support Alibaba DLF metastore for hive external table #6403

Conversation

mxdzs0612 commented May 23, 2022 • edited by miomiocat Loading

What type of PR is this：

Which issues of this PR fixes ：

Problem Summary(Required) ：

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxdzs0612 commented May 27, 2022

wanpengfei-git commented May 27, 2022

[FE PR Coverage check]

file detail

stephen-shelby left a comment

Choose a reason for hiding this comment

mxdzs0612 commented May 23, 2022 •

edited by miomiocat

Loading