Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#4743]fix(trino-connector): Fix the default precision of time and timestamp in the Iceberg catalog. #4893

Merged
merged 1 commit into from
Sep 13, 2024

Conversation

diqiu50
Copy link
Contributor

@diqiu50 diqiu50 commented Sep 9, 2024

What changes were proposed in this pull request?

Fix the default precision of time and timestamp in the Iceberg catalog.
It causes Trino to be unable to read Iceberg tables with data of time and timestamp

Why are the changes needed?

Fix: #4743

Does this PR introduce any user-facing change?

NO

How was this patch tested?

New UT, IT

@diqiu50 diqiu50 self-assigned this Sep 9, 2024
@diqiu50 diqiu50 added the 0.7.0 Release v0.7.0 label Sep 9, 2024
@diqiu50 diqiu50 added this to the Gravitino 0.7.0 milestone Sep 9, 2024
@diqiu50 diqiu50 removed the 0.7.0 Release v0.7.0 label Sep 9, 2024
@yuqi1129
Copy link
Contributor

yuqi1129 commented Sep 9, 2024

Does this need to be backported to branch-0.6?

@jerryshao jerryshao removed this from the Gravitino 0.7.0 milestone Sep 9, 2024
@@ -20,6 +20,9 @@
package org.apache.gravitino.trino.connector.catalog.iceberg;

import io.trino.spi.TrinoException;
import io.trino.spi.type.TimeType;
import io.trino.spi.type.TimestampType;
import io.trino.spi.type.TimestampWithTimeZoneType;
import io.trino.spi.type.VarbinaryType;
import io.trino.spi.type.VarcharType;
import org.apache.gravitino.rel.types.Type;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems you shoud add corresponding logic to convert MICROS time to Iceberg time, and forbidden other time precision.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@diqiu50 diqiu50 Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trino iceberg connector can automatically convert other time precision to microsecond

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what will happen if the user creates an Iceberg table with timestamp(3) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It automatically converts to timestamp(6), The behavior is the same as in the Trino Iceberg catalog.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the code https://github.com/trinodb/trino/blob/457/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TypeConverter.java#L148-L177, it will throw an exception in origin Trino Iceberg catalog, Yes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I don't think it's the right behavior to convert timestamp(3) to timestamp(6) in Gravitino Iceberg connector, it conflict with original behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trino Iceberg connector can automatically converts timestamp and timestamp(n) to timestamp(6),
The Gravitino connector Iceberg catalog's behavior is the same as Trino Iceberg connector

@diqiu50
Copy link
Contributor Author

diqiu50 commented Sep 11, 2024

Does this need to be backported to branch-0.6?

Yes

@diqiu50 diqiu50 requested a review from FANNG1 September 12, 2024 03:52
@FANNG1
Copy link
Contributor

FANNG1 commented Sep 13, 2024

LGTM

@FANNG1
Copy link
Contributor

FANNG1 commented Sep 13, 2024

@yuqi1129 @mchades , do you have time to review again?

@FANNG1 FANNG1 added the branch-0.6 Automatically cherry-pick commit to branch-0.6 label Sep 13, 2024
@FANNG1 FANNG1 merged commit b0c4b11 into apache:main Sep 13, 2024
23 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 13, 2024
…mestamp in the Iceberg catalog. (#4893)

### What changes were proposed in this pull request?

Fix the default precision of time and timestamp in the Iceberg catalog. 
It causes Trino to be unable to read Iceberg tables with data of time
and timestamp

### Why are the changes needed?

Fix: #4743

### Does this PR introduce _any_ user-facing change?

NO

### How was this patch tested?

New UT, IT
@diqiu50 diqiu50 deleted the iceberg-time branch September 26, 2024 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-0.6 Automatically cherry-pick commit to branch-0.6
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug report] trino couldn't read Iceberg table with timestamp column created by spark
4 participants