Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#5220] improvment(hadoop-catalog): Optimize the name properties keys for Hadoop catalog. #5244

Merged
merged 174 commits into from
Oct 30, 2024
Merged
Show file tree
Hide file tree
Changes from 168 commits
Commits
Show all changes
174 commits
Select commit Hold shift + click to select a range
ce3b19f
Fix minor in docker image.
yuqi1129 Aug 26, 2024
cd69c33
Merge branch 'apache:main' into main
yuqi1129 Sep 3, 2024
d2447a2
Add a framework to support multi-storage in a pluginized manner for …
yuqi1129 Sep 25, 2024
7e5a8b5
Fix compile distribution error.
yuqi1129 Sep 25, 2024
f53c5ef
fix
yuqi1129 Sep 25, 2024
36fedcd
fix
yuqi1129 Sep 25, 2024
e93fba5
fix
yuqi1129 Sep 25, 2024
b1e04b6
fix
yuqi1129 Sep 26, 2024
db00e65
Changed according to comments.
yuqi1129 Sep 29, 2024
c793582
fix
yuqi1129 Sep 29, 2024
013f5cb
fix
yuqi1129 Sep 30, 2024
dba5753
Merge branch 'main' of github.com:datastrato/graviton into issue_5019
yuqi1129 Oct 8, 2024
16dfc73
resolve comments.
yuqi1129 Oct 8, 2024
278fcd8
Polish code.
yuqi1129 Oct 9, 2024
3fb55ad
fix
yuqi1129 Oct 9, 2024
cd04666
fix
yuqi1129 Oct 9, 2024
d0bf13e
Support GCS fileset.
yuqi1129 Oct 9, 2024
ffaa064
Change gvfs accordingly.
yuqi1129 Oct 11, 2024
32d7f3d
Merge remote-tracking branch 'me/issue_5019' into issue_5019
yuqi1129 Oct 11, 2024
81a36ba
Merge branch 'apache:main' into main
yuqi1129 Oct 11, 2024
d82bf76
Update Java doc for FileSystemProvider
yuqi1129 Oct 11, 2024
dfdb772
Fix
yuqi1129 Oct 11, 2024
8708a8a
Fix
yuqi1129 Oct 11, 2024
ba9f8fa
Fix test error.
yuqi1129 Oct 11, 2024
dae99f7
Polish.
yuqi1129 Oct 11, 2024
e22053b
Polish
yuqi1129 Oct 12, 2024
4fb89e0
Polish
yuqi1129 Oct 12, 2024
e5746c0
Rename `AbstractIT` to `BaseIT`
yuqi1129 Oct 12, 2024
b2d7bed
Fix
yuqi1129 Oct 14, 2024
380717b
Merge branch 'apache:main' into issue_5019
yuqi1129 Oct 14, 2024
f4041ec
Fix python ut error again.
yuqi1129 Oct 14, 2024
66247ab
Merge branch 'issue_5019' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 14, 2024
3cfb94f
Fix test error again.
yuqi1129 Oct 14, 2024
7d1150f
Fix minor.
yuqi1129 Oct 14, 2024
608081b
fix
yuqi1129 Oct 14, 2024
9edfe82
Fix
yuqi1129 Oct 14, 2024
3079bf0
Fix
yuqi1129 Oct 14, 2024
da49e60
Fix
yuqi1129 Oct 14, 2024
b621d89
Fix
yuqi1129 Oct 14, 2024
05dd006
Merge branch 'issue_5019' into issue_5074
yuqi1129 Oct 15, 2024
9d5b8dc
rebase issue_5019
yuqi1129 Oct 15, 2024
e58f9a0
Fix
yuqi1129 Oct 15, 2024
c521daf
resolve comments
yuqi1129 Oct 15, 2024
46e996a
Resolve comments again.
yuqi1129 Oct 15, 2024
da0b7ca
Polish again.
yuqi1129 Oct 15, 2024
e9ccda4
Merge branch 'issue_5019' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 15, 2024
ba1fe5f
Rebase branch issue_5019
yuqi1129 Oct 15, 2024
4ffe389
Merge branch 'main' of github.com:datastrato/graviton into issue_5139
yuqi1129 Oct 15, 2024
7c44a57
Support python gvfs
yuqi1129 Oct 15, 2024
992ba0a
fix
yuqi1129 Oct 15, 2024
5dbca5f
fix
yuqi1129 Oct 15, 2024
f27520a
Update code.
yuqi1129 Oct 15, 2024
e29e47b
Merge branch 'issue_5019' into issue_5074
yuqi1129 Oct 15, 2024
bc1e76f
Rebase branch issue_5019
yuqi1129 Oct 15, 2024
8a9d3bf
Merge branch 'issue_5074' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 15, 2024
2115e31
fix
yuqi1129 Oct 15, 2024
c2e55d4
fix
yuqi1129 Oct 15, 2024
557aa02
fix
yuqi1129 Oct 15, 2024
5c3fa5c
fix
yuqi1129 Oct 15, 2024
408eca7
fix
yuqi1129 Oct 15, 2024
a02065d
fix
yuqi1129 Oct 15, 2024
8762bae
fix
yuqi1129 Oct 15, 2024
dc7a915
fix
yuqi1129 Oct 15, 2024
c230991
fix
yuqi1129 Oct 15, 2024
dc54880
skip some test.
yuqi1129 Oct 15, 2024
7ecc040
fix
yuqi1129 Oct 15, 2024
da46321
fix
yuqi1129 Oct 15, 2024
27bc2ab
Fix
yuqi1129 Oct 16, 2024
017c42e
Merge branch 'issue_5019' into issue_5074
yuqi1129 Oct 16, 2024
9dc0f5a
Fix
yuqi1129 Oct 16, 2024
41ff00d
Merge branch 'issue_5019' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 16, 2024
1fee1e4
Fix
yuqi1129 Oct 16, 2024
1789bd2
fix
yuqi1129 Oct 16, 2024
2ee1709
Fix
yuqi1129 Oct 16, 2024
05e5d20
Fix
yuqi1129 Oct 16, 2024
8f28211
Fix
yuqi1129 Oct 16, 2024
35cba1e
Fix
yuqi1129 Oct 16, 2024
bcf2f12
Fix
yuqi1129 Oct 16, 2024
f25a37d
Fix a problem
yuqi1129 Oct 16, 2024
a3da011
Merge branch 'issue_5019' into issue_5074
yuqi1129 Oct 16, 2024
3517996
Merge branch 'main' of github.com:apache/gravitino into issue_5074
yuqi1129 Oct 16, 2024
27a911a
fix
yuqi1129 Oct 16, 2024
e34dbea
Merge branch 'main' of github.com:apache/gravitino into issue_5074
yuqi1129 Oct 16, 2024
6bae7e5
Fix a problem
yuqi1129 Oct 16, 2024
fe13f5e
Fix a problem
yuqi1129 Oct 16, 2024
0181632
Fix a problem
yuqi1129 Oct 16, 2024
3ff9eef
Fix
yuqi1129 Oct 16, 2024
2ce660c
Fix
yuqi1129 Oct 16, 2024
f0fa87b
Fix
yuqi1129 Oct 16, 2024
d2921a8
Fix
yuqi1129 Oct 16, 2024
dc68dd1
Fix
yuqi1129 Oct 16, 2024
6431e2f
Fix
yuqi1129 Oct 16, 2024
242888f
Fix
yuqi1129 Oct 16, 2024
3ec2dcc
Merge branch 'issue_5074' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 16, 2024
f754997
Fix
yuqi1129 Oct 16, 2024
2cdfb35
Fix
yuqi1129 Oct 16, 2024
67dbc3a
Resolve comments.
yuqi1129 Oct 17, 2024
70a545e
Fix the java doc problem.
yuqi1129 Oct 17, 2024
4d54acf
Merge branch 'issue_5074' into issue_5139
yuqi1129 Oct 17, 2024
15bbf99
rebase issue_5074
yuqi1129 Oct 17, 2024
cfcc544
Optimize code.
yuqi1129 Oct 17, 2024
acf51e1
Merge branch 'issue_5074' of github.com:yuqi1129/gravitino into issue…
yuqi1129 Oct 17, 2024
11f9992
Merge branch 'main' of github.com:datastrato/graviton into issue_5139
yuqi1129 Oct 17, 2024
4f00a2f
Remove s3 related code.
yuqi1129 Oct 17, 2024
b9ef8f0
fix
yuqi1129 Oct 17, 2024
9f65fb5
try to import lazily.
yuqi1129 Oct 18, 2024
4defcc6
format code.
yuqi1129 Oct 18, 2024
3a907f4
fix
yuqi1129 Oct 18, 2024
76912b7
fix
yuqi1129 Oct 18, 2024
4478673
fix
yuqi1129 Oct 18, 2024
5a194df
Merge branch 'main' of github.com:datastrato/graviton into issue_5139
yuqi1129 Oct 18, 2024
c8b5c7c
Merge branch 'main' of github.com:datastrato/graviton into issue_5139
yuqi1129 Oct 21, 2024
6ff9353
Resolve comments
yuqi1129 Oct 21, 2024
1dac0f0
Merge branch 'main' of github.com:datastrato/graviton into issue_5188
yuqi1129 Oct 21, 2024
f592289
Support python client for S3.
yuqi1129 Oct 21, 2024
7069b6b
fix
yuqi1129 Oct 21, 2024
1281e72
fix
yuqi1129 Oct 21, 2024
f022d6e
fix
yuqi1129 Oct 21, 2024
0d2ccab
fix
yuqi1129 Oct 22, 2024
55633e8
Merge branch 'main' of github.com:datastrato/graviton into issue_5188
yuqi1129 Oct 22, 2024
217cc5f
fix
yuqi1129 Oct 22, 2024
6958aa8
fix
yuqi1129 Oct 22, 2024
b6837bb
fix
yuqi1129 Oct 22, 2024
92961ea
fix
yuqi1129 Oct 22, 2024
8ed6298
fix
yuqi1129 Oct 22, 2024
e256c8d
fix
yuqi1129 Oct 22, 2024
73e00f0
fix
yuqi1129 Oct 23, 2024
d56cf30
fix
yuqi1129 Oct 23, 2024
8b9b8d7
Replace pyarrow gvfs with gcsfs.
yuqi1129 Oct 23, 2024
b4d5728
fix
yuqi1129 Oct 23, 2024
0306ac5
fix
yuqi1129 Oct 23, 2024
a35a40d
fix
yuqi1129 Oct 23, 2024
cfd8a89
fix
yuqi1129 Oct 23, 2024
a131564
fix
yuqi1129 Oct 23, 2024
34e3ff4
Replace ArrowFSWrapper with s3fs.S3FileSystem.
yuqi1129 Oct 23, 2024
2cc234a
fix
yuqi1129 Oct 23, 2024
ba0237e
fix
yuqi1129 Oct 23, 2024
4df5ea1
fix
yuqi1129 Oct 23, 2024
4e49e6e
fix
yuqi1129 Oct 23, 2024
354418f
Merge branch 'issue_5188' into issue_5211
yuqi1129 Oct 23, 2024
d22c2f6
fix
yuqi1129 Oct 23, 2024
fc3c40a
fix
yuqi1129 Oct 23, 2024
015b788
fix comments.
yuqi1129 Oct 23, 2024
c414b4d
fix test error.
yuqi1129 Oct 23, 2024
895937c
Merge branch 'issue_5188' into issue_5211
yuqi1129 Oct 23, 2024
d261b70
fix test error.
yuqi1129 Oct 23, 2024
798b4d1
fix
yuqi1129 Oct 23, 2024
aa96f63
fix
yuqi1129 Oct 23, 2024
456ba31
Merge branch 'issue_5188' into issue_5211
yuqi1129 Oct 23, 2024
8d04cd4
fix
yuqi1129 Oct 23, 2024
804622c
fix
yuqi1129 Oct 23, 2024
b399365
fix
yuqi1129 Oct 24, 2024
120adc3
Merge branch 'issue_5188' into issue_5211
yuqi1129 Oct 24, 2024
72e8822
rebase main.
yuqi1129 Oct 24, 2024
e7b4da7
minor polish
yuqi1129 Oct 24, 2024
20360b0
Merge branch 'main' of github.com:datastrato/graviton into issue_5220
yuqi1129 Oct 24, 2024
87dbb60
Improvement: Optimize the key names of Hadoop catalog properties.
yuqi1129 Oct 24, 2024
d5ac7d4
Merge branch 'main' of github.com:datastrato/graviton into issue_5220
yuqi1129 Oct 24, 2024
a187080
Fix format.
yuqi1129 Oct 24, 2024
ca57d3f
Merge branch 'main' of github.com:datastrato/graviton into issue_5220
yuqi1129 Oct 25, 2024
10b6267
fix
yuqi1129 Oct 25, 2024
03bc296
change python keys to make it align with java
yuqi1129 Oct 25, 2024
32d6553
Update.
yuqi1129 Oct 29, 2024
11c9f41
Merge branch 'main' of github.com:yuqi1129/gravitino into issue_5220
yuqi1129 Oct 29, 2024
6192586
Merge branch 'main' of github.com:apache/gravitino into issue_5220
yuqi1129 Oct 29, 2024
67ac175
Merge branch 'main' of github.com:apache/gravitino into issue_5220
yuqi1129 Oct 29, 2024
268831d
fix
yuqi1129 Oct 29, 2024
27393e5
Optimize.
yuqi1129 Oct 29, 2024
b0d45f1
Optimize method `toHadoopConfigMap` and other fix.
yuqi1129 Oct 29, 2024
11a3146
Add UT for `FileSystemUtils.toHadoopConfigMap`
yuqi1129 Oct 29, 2024
218acae
Add UT for `FileSystemUtils.toHadoopConfigMap`
yuqi1129 Oct 29, 2024
8ff278f
Add VisibleForTesting annotation for public fields in xxxxFileSystemP…
yuqi1129 Oct 29, 2024
4f90525
Fix java docs problem.
yuqi1129 Oct 29, 2024
bf65415
Fix
yuqi1129 Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions bundles/aliyun-bundle/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ dependencies {
// org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:323)
// org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
implementation(libs.commons.lang)
implementation(project(":catalogs:catalog-common")) {
exclude("*")
}
}

tasks.withType(ShadowJar::class.java) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,40 @@
*/
package org.apache.gravitino.oss.fs;

import com.google.common.collect.ImmutableMap;
import java.io.IOException;
import java.util.Map;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
import org.apache.gravitino.storage.OSSProperties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem;
import org.apache.hadoop.fs.aliyun.oss.Constants;

public class OSSFileSystemProvider implements FileSystemProvider {

private static final String OSS_FILESYSTEM_IMPL = "fs.oss.impl";

private static final Map<String, String> GRAVITINO_KEY_TO_OSS_HADOOP_KEY =
ImmutableMap.of(
OSSProperties.GRAVITINO_OSS_ENDPOINT, Constants.ENDPOINT_KEY,
OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID, Constants.ACCESS_KEY_ID,
OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET, Constants.ACCESS_KEY_SECRET);

@Override
public FileSystem getFileSystem(Path path, Map<String, String> config) throws IOException {
Configuration configuration = new Configuration();
config.forEach(
(k, v) -> {
configuration.set(k.replace("gravitino.bypass.", ""), v);
});

Map<String, String> hadoopConfMap =
FileSystemUtils.toHadoopConfigMap(config, GRAVITINO_KEY_TO_OSS_HADOOP_KEY);
// OSS do not use service loader to load the file system, so we need to set the impl class
if (!hadoopConfMap.containsKey(OSS_FILESYSTEM_IMPL)) {
hadoopConfMap.put(OSS_FILESYSTEM_IMPL, AliyunOSSFileSystem.class.getCanonicalName());
}

hadoopConfMap.forEach(configuration::set);
return AliyunOSSFileSystem.newInstance(path.toUri(), configuration);
}

Expand Down
3 changes: 3 additions & 0 deletions bundles/aws-bundle/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ dependencies {
compileOnly(project(":catalogs:catalog-hadoop"))
compileOnly(libs.hadoop3.common)
implementation(libs.hadoop3.aws)
implementation(project(":catalogs:catalog-common")) {
exclude("*")
}
}

tasks.withType(ShadowJar::class.java) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,37 @@

package org.apache.gravitino.s3.fs;

import com.google.common.collect.ImmutableMap;
import java.io.IOException;
import java.util.Map;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
import org.apache.gravitino.storage.S3Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.s3a.Constants;
import org.apache.hadoop.fs.s3a.S3AFileSystem;

public class S3FileSystemProvider implements FileSystemProvider {

private static final Map<String, String> GRAVITINO_KEY_TO_S3_HADOOP_KEY =
ImmutableMap.of(
S3Properties.GRAVITINO_S3_ENDPOINT, Constants.ENDPOINT,
S3Properties.GRAVITINO_S3_ACCESS_KEY_ID, Constants.ACCESS_KEY,
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY, Constants.SECRET_KEY);

@Override
public FileSystem getFileSystem(Path path, Map<String, String> config) throws IOException {
Configuration configuration = new Configuration();
config.forEach(
(k, v) -> {
configuration.set(k.replace("gravitino.bypass.", ""), v);
});
Map<String, String> hadoopConfMap =
FileSystemUtils.toHadoopConfigMap(config, GRAVITINO_KEY_TO_S3_HADOOP_KEY);

if (!hadoopConfMap.containsKey(Constants.AWS_CREDENTIALS_PROVIDER)) {
configuration.set(
Constants.AWS_CREDENTIALS_PROVIDER, Constants.ASSUMED_ROLE_CREDENTIALS_DEFAULT);
}
hadoopConfMap.forEach(configuration::set);
return S3AFileSystem.newInstance(path.toUri(), configuration);
}

Expand Down
3 changes: 3 additions & 0 deletions bundles/gcp-bundle/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ dependencies {

implementation(libs.commons.lang3)
implementation(libs.hadoop3.gcs)
implementation(project(":catalogs:catalog-common")) {
exclude("*")
}
implementation(libs.google.auth.http)
implementation(libs.google.auth.credentials)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@
package org.apache.gravitino.gcs.fs;

import com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem;
import com.google.common.collect.ImmutableMap;
import java.io.IOException;
import java.util.Map;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemUtils;
import org.apache.gravitino.storage.GCSProperties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
Expand All @@ -30,15 +33,17 @@

public class GCSFileSystemProvider implements FileSystemProvider {
private static final Logger LOGGER = LoggerFactory.getLogger(GCSFileSystemProvider.class);
private static final String GCS_SERVICE_ACCOUNT_JSON_FILE =
"fs.gs.auth.service.account.json.keyfile";

private static final Map<String, String> GRAVITINO_KEY_TO_GCS_HADOOP_KEY =
ImmutableMap.of(GCSProperties.GCS_SERVICE_ACCOUNT_JSON_PATH, GCS_SERVICE_ACCOUNT_JSON_FILE);

@Override
public FileSystem getFileSystem(Path path, Map<String, String> config) throws IOException {
Configuration configuration = new Configuration();
config.forEach(
(k, v) -> {
configuration.set(k.replace("gravitino.bypass.", ""), v);
});

FileSystemUtils.toHadoopConfigMap(config, GRAVITINO_KEY_TO_GCS_HADOOP_KEY)
.forEach(configuration::set);
LOGGER.info("Creating GCS file system with config: {}", config);
return GoogleHadoopFileSystem.newInstance(path.toUri(), configuration);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.gravitino.storage;

public class GCSProperties {

// The path of service account JSON file of Google Cloud Storage.
public static final String GCS_SERVICE_ACCOUNT_JSON_PATH = "gcs-service-account-file";

private GCSProperties() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ public class OSSProperties {
// The static access key ID used to access OSS data.
public static final String GRAVITINO_OSS_ACCESS_KEY_ID = "oss-access-key-id";
// The static access key secret used to access OSS data.
public static final String GRAVITINO_OSS_ACCESS_KEY_SECRET = "oss-access-key-secret";
public static final String GRAVITINO_OSS_ACCESS_KEY_SECRET = "oss-secret-access-key";

private OSSProperties() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,8 @@ public class S3Properties {
// The region of the S3 service.
public static final String GRAVITINO_S3_REGION = "s3-region";

// The S3 credentials provider class name.
public static final String GRAVITINO_S3_CREDS_PROVIDER = "s3-creds-provider";

private S3Properties() {}
}
yuqi1129 marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions catalogs/catalog-hadoop/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ dependencies {
exclude(group = "*")
}

implementation(project(":catalogs:catalog-common")) {
exclude(group = "*")
}

compileOnly(libs.guava)

implementation(libs.hadoop3.common) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,20 @@
*/
public interface FileSystemProvider {

/**
* The prefix of the configuration key that should be bypassed when setting the configuration to
* the FileSystem instance.
*
* <p>For example, if the configuration key passed to {@link
* FileSystemProvider#getFileSystem(Path, Map)} 'gravitino.bypass.fs.s3a.endpoint', the prefix
* 'gravitino.bypass.' should be removed when setting the configuration to the FileSystem
* instance.
*
* <p>User can use this prefix to pass the configuration item that has not been defined in
* Gravitino.
*/
String GRAVITINO_BYPASS = "gravitino.bypass.";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add some comments to this variable, to let users know how to leverage this variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added


/**
* Get the FileSystem instance according to the configuration map and file path.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

import static org.apache.gravitino.catalog.hadoop.HadoopCatalogPropertiesMetadata.BUILTIN_HDFS_FS_PROVIDER;
import static org.apache.gravitino.catalog.hadoop.HadoopCatalogPropertiesMetadata.BUILTIN_LOCAL_FS_PROVIDER;
import static org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider.GRAVITINO_BYPASS;

import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
Expand Down Expand Up @@ -101,4 +102,45 @@ public static FileSystemProvider getFileSystemProviderByName(
"File system provider with name '%s' not found in the file system provider list.",
fileSystemProviderName)));
}

/**
* Convert the Gravitino configuration to Hadoop configuration.
*
* <p>Predefined keys have the highest priority. If the key does not exist in the predefined keys,
* it will be set to the configuration. Keys with prefixes 'gravitino.bypass' has the lowest
* priority.
*
* @param config Gravitino configuration
* @return Hadoop configuration Map
*/
public static Map<String, String> toHadoopConfigMap(
Map<String, String> config, Map<String, String> predefinedKeys) {
Map<String, String> result = Maps.newHashMap();
Set<String> highestPriorityKeys = Sets.newHashSet();
config.forEach(
(k, v) -> {
if (predefinedKeys.containsKey(k)) {
String key = predefinedKeys.get(k);
highestPriorityKeys.add(key);
result.put(key, v);
}

if (!k.startsWith(GRAVITINO_BYPASS)) {
// If the key does not start with 'gravitino.bypass' and is not in the highest priority
// keys, set the value to the configuration.
if (!highestPriorityKeys.contains(k)) {
result.put(k, v);
}
} else {
// If the key starts with 'gravitino.bypass', remove the prefix and set the value to the
// configuration if the key does not exist in the configuration.
String key = k.replace(GRAVITINO_BYPASS, "");
if (!result.containsKey(key)) {
result.put(key, v);
}
}
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simplify this code? It's a bit hard to understand, I think it is OK to iterate for several rounds, just make it easy to read.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a UT for this?


return result;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,26 @@
package org.apache.gravitino.catalog.hadoop.integration.test;

import static org.apache.gravitino.catalog.hadoop.HadoopCatalogPropertiesMetadata.FILESYSTEM_PROVIDERS;
import static org.apache.gravitino.storage.GCSProperties.GCS_SERVICE_ACCOUNT_JSON_PATH;

import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
import java.io.IOException;
import java.net.URI;
import java.util.Map;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Schema;
import org.apache.gravitino.file.Fileset;
import org.apache.gravitino.integration.test.util.GravitinoITUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;

@Tag("gravitino-docker-test")
@Disabled(
Expand Down Expand Up @@ -95,10 +102,8 @@ protected String defaultBaseLocation() {

protected void createCatalog() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move these ITs to each bundle?

Copy link
Contributor Author

@yuqi1129 yuqi1129 Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I can try it but I will not do it in this PR, It will make this PR large and hard to test the changes introduced by this PR, so I created #5325 to follow it.

Map<String, String> map = Maps.newHashMap();
map.put("gravitino.bypass.fs.gs.auth.service.account.enable", "true");
map.put("gravitino.bypass.fs.gs.auth.service.account.json.keyfile", SERVICE_ACCOUNT_FILE);
map.put(GCS_SERVICE_ACCOUNT_JSON_PATH, SERVICE_ACCOUNT_FILE);
map.put(FILESYSTEM_PROVIDERS, "gcs");

metalake.createCatalog(catalogName, Catalog.Type.FILESET, provider, "comment", map);

catalog = metalake.loadCatalog(catalogName);
Expand All @@ -107,4 +112,64 @@ protected void createCatalog() {
protected String generateLocation(String filesetName) {
return String.format("%s/%s", defaultBaseLocation, filesetName);
}

@Test
public void testCreateSchemaAndFilesetWithSpecialLocation() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to fix the bug introduced by #5296, we need to consider other Hadoop catalog like S3, OSS and GCS.

String localCatalogName = GravitinoITUtils.genRandomName("local_catalog");

String ossLocation = String.format("gs://%s", BUCKET_NAME);
Map<String, String> catalogProps = Maps.newHashMap();
catalogProps.put("location", ossLocation);
catalogProps.put(GCS_SERVICE_ACCOUNT_JSON_PATH, SERVICE_ACCOUNT_FILE);
catalogProps.put(FILESYSTEM_PROVIDERS, "gcs");

Catalog localCatalog =
metalake.createCatalog(
localCatalogName, Catalog.Type.FILESET, provider, "comment", catalogProps);
Assertions.assertEquals(ossLocation, localCatalog.properties().get("location"));

// Create schema without specifying location.
Schema localSchema =
localCatalog
.asSchemas()
.createSchema("local_schema", "comment", ImmutableMap.of("key1", "val1"));

Fileset localFileset =
localCatalog
.asFilesetCatalog()
.createFileset(
NameIdentifier.of(localSchema.name(), "local_fileset"),
"fileset comment",
Fileset.Type.MANAGED,
null,
ImmutableMap.of("k1", "v1"));
Assertions.assertEquals(
ossLocation + "/local_schema/local_fileset", localFileset.storageLocation());

// Delete schema
localCatalog.asSchemas().dropSchema(localSchema.name(), true);

// Create schema with specifying location.
Map<String, String> schemaProps = ImmutableMap.of("location", ossLocation);
Schema localSchema2 =
localCatalog.asSchemas().createSchema("local_schema2", "comment", schemaProps);
Assertions.assertEquals(ossLocation, localSchema2.properties().get("location"));

Fileset localFileset2 =
localCatalog
.asFilesetCatalog()
.createFileset(
NameIdentifier.of(localSchema2.name(), "local_fileset2"),
"fileset comment",
Fileset.Type.MANAGED,
null,
ImmutableMap.of("k1", "v1"));
Assertions.assertEquals(ossLocation + "/local_fileset2", localFileset2.storageLocation());

// Delete schema
localCatalog.asSchemas().dropSchema(localSchema2.name(), true);

// Delete catalog
metalake.dropCatalog(localCatalogName, true);
}
}
Loading
Loading