Support creating Delta tables with column mapping mode #17565
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10 tasks
ebyhr
force-pushed
the
ebi/delta-create-table-column-mapping-mode
branch
3 times, most recently
from
23d28c4 to
bacd0ba
Compare
ebyhr
force-pushed
the
ebi/delta-create-table-column-mapping-mode
branch
2 times, most recently
from
9de4af1 to
a36356e
Compare
findinpath
reviewed
Jun 7, 2023
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeTableProperties.java
Outdated
Show resolved
Hide resolved
findinpath
reviewed
Jun 7, 2023
findinpath
reviewed
Jun 7, 2023
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java
Show resolved
Hide resolved
findinpath
reviewed
Jun 7, 2023
plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeColumnMapping.java
Show resolved
Hide resolved
ebyhr
force-pushed
the
ebi/delta-create-table-column-mapping-mode
branch
from
a36356e to
c55b7a4
Compare
findinpath
approved these changes
Jun 9, 2023
findinpath
reviewed
Jun 9, 2023
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/MetadataEntry.java
Outdated
Show resolved
Hide resolved
findepi
reviewed
Jul 4, 2023
...no-product-tests/src/main/java/io/trino/tests/product/deltalake/util/DeltaLakeTestUtils.java
Show resolved
Hide resolved
findepi
approved these changes
Jul 4, 2023
core/trino-spi/src/main/java/io/trino/spi/session/PropertyMetadata.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/session/PropertyMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeTableProperties.java
Outdated
Show resolved
Hide resolved
Comment on lines
+971
to
+972
| Object serializedType = serializeColumnType(columnMappingMode, fieldId, column.getType()); | ||
| Type physicalType = deserializeType(typeManager, serializedType, usePhysicalName); |
There was a problem hiding this comment.
it looks weird to serializeColumnType only to deserializeType in the next line.
can this be simplified?
also, why serializeColumnType doesn't need usePhysicalName?
the input, column.getType(), doesn't have physical and logical names, so how are .../metadata/delta.columnMapping.physicalName and .../metadata/name populated?
There was a problem hiding this comment.
can this be simplified?
That's possible, but not so simple. Let me handle in follow-up.
also, why serializeColumnType doesn't need usePhysicalName?
The method uses ColumnMappingMode argument instead of the flag. I passed the mode because it affects not only delta.columnMapping.physicalName but also delta.columnMapping.id.
how are .../metadata/delta.columnMapping.physicalName and .../metadata/name populated?
delta.columnMapping.physicalNameis populated in serializeStructType → generateColumnMetadata in case of id and name modenameis always populated in serializeStructType → serializeStructField
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/MetadataEntry.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeTableProperties.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java
Show resolved
Hide resolved
This method tried to use 'hive' catalog by default.
|
Ran CI with the internal environment. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support creating Delta tables with column mapping mode
Fixes #12638
Release notes
(x) Release notes are required, with the following suggested text: