Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Subtask] [spark-connector] support passing catalog properties to spark connector #3292

Closed
Tracked by #1227
FANNG1 opened this issue May 7, 2024 · 2 comments · Fixed by #3270
Closed
Tracked by #1227

[Subtask] [spark-connector] support passing catalog properties to spark connector #3292

FANNG1 opened this issue May 7, 2024 · 2 comments · Fixed by #3270
Assignees
Labels
0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 subtask Subtasks of umbrella issue

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented May 7, 2024

Describe the subtask

To fix #3181 , we should allow user to pass catalog properties to spark connector, like spark.sql.hive.metastore.jars

Parent issue

#1227

@FANNG1 FANNG1 added the subtask Subtasks of umbrella issue label May 7, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented May 7, 2024

Which properties should passed to the spark connector?

  1. properties with prefix spark.bypass.. Pros: keep consistent with gravitino.bypass, and trino.bypass, Cons: complex, especially for the properties shared by multi engines.
  2. all catalog properties. Pros: simple and clear. Cons: lack of control.

@jerryshao @qqqttt123 WDYT?

@FANNG1 FANNG1 self-assigned this May 10, 2024
jerryshao pushed a commit that referenced this issue May 13, 2024
… spark connector (#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
github-actions bot pushed a commit that referenced this issue May 13, 2024
… spark connector (#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
@jerryshao jerryshao added 0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 labels May 13, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented May 13, 2024

using spark.bypass to keep consistent

FANNG1 added a commit that referenced this issue May 13, 2024
… spark connector (#3358)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: #3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment

Co-authored-by: FANNG <xiaojing@datastrato.com>
diqiu50 pushed a commit to diqiu50/gravitino that referenced this issue Jun 13, 2024
…ies to spark connector (apache#3270)

### What changes were proposed in this pull request?
1. passing Gravitino catalog properties with `spark.bypass.` prefix to
spark connector
2. refactor the catalog properties transform logic.

### Why are the changes needed?

Fix: apache#3292

### Does this PR introduce _any_ user-facing change?
yes, add document

### How was this patch tested?
1. add UT
3. test in local envriment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.5.1 Release v0.5.1 0.6.0 Release v0.6.0 subtask Subtasks of umbrella issue
Projects
None yet
2 participants