Skip to content

Commit

Permalink
[Feature]Support Alibaba DLF metastore for hive external table (StarR…
Browse files Browse the repository at this point in the history
…ocks#6403)

Data Lake Formation (DLF) is a key component of the cloud-native data lake framework which is widely used on Alibaba Cloud just like AWS Glue, details see https://www.alibabacloud.com/en/product/datalake-formation. This PR allows users to use DLF as metastore in Hive external table.

Usage:
Add a config file hive-site.xml to {FE Home DIR}/conf , with following configs:

<?xml version="1.0"?>
<configuration>
    <!--Set to use dlf client-->
    <property>
        <name>hive.metastore.type</name>
        <value>dlf</value>
    </property>
    <!--DLF endpoint, see https://www.alibabacloud.com/help/en/doc-detail/197608.html-->
    <property>
        <name>dlf.catalog.endpoint</name>
        <value>dlf-vpc.cn-beijing.aliyuncs.com</value>
    </property>
    <!--DLF region, see https://www.alibabacloud.com/help/en/doc-detail/197608.html-->
    <property>
        <name>dlf.catalog.region</name>
        <value>cn-beijing</value>
    </property>
    <!--Proxy mode of DLF-->
    <property>
        <name>dlf.catalog.proxyMode</name>
        <value>DLF_ONLY</value>
    </property>
    <!--Access Key mode of DLF-->
    <property>
        <name>dlf.catalog.akMode</name>
        <value>EMR_AUTO</value>
    </property>
    <!--User id of the alibaba cloud account-->
    <property>
        <name>dlf.catalog.uid</name>
        <value>xxxxxx</value>
    </property>
    <!--Access Key ID of DLF, can be omitted if the cluster is created with the same Alibaba Cloud account of DLF-->
    <property>
        <name>dlf.catalog.accessKeyId</name>
        <value>xxxxxx</value>
    </property>
    <!--Access Key secret of DLF, can be omitted if the cluster is created with the same Alibaba Cloud account of DLF-->
    <property>
        <name>dlf.catalog.accessKeySecret</name>
        <value>xxxxxx</value>
    </property>
</configuration>
  • Loading branch information
mxdzs0612 authored and abc982627271 committed Jun 22, 2022
1 parent 5a02f04 commit 4e31c63
Show file tree
Hide file tree
Showing 5 changed files with 2,530 additions and 11 deletions.
22 changes: 21 additions & 1 deletion fe/fe-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -465,10 +465,30 @@ under the License.

<!-- https://mvnrepository.com/artifact/com.facebook.presto.hive/hive-apache -->
<dependency>
<groupId>com.facebook.presto.hive</groupId>
<groupId>io.trino.hive</groupId>
<artifactId>hive-apache</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/com.aliyun.datalake/metastore-client-hive3 -->
<dependency>
<groupId>com.aliyun.datalake</groupId>
<artifactId>metastore-client-hive3</artifactId>
<exclusions>
<exclusion>
<groupId>com.aliyun</groupId>
<artifactId>tea</artifactId>
</exclusion>
<exclusion>
<groupId>com.aliyun</groupId>
<artifactId>tea-openapi</artifactId>
</exclusion>
<exclusion>
<groupId>com.aliyun</groupId>
<artifactId>tea-util</artifactId>
</exclusion>
</exclusions>
</dependency>

<!-- https://mvnrepository.com/artifact/com.github.ben-manes.caffeine/caffeine -->
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
Expand Down
Loading

0 comments on commit 4e31c63

Please sign in to comment.