Skip to content

Add support for timestamptz/timestamp with PostgresqlDialect #1480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mdedetrich
Copy link

Problem

This PR is an attempt to solve #921, specifically the issue at #921 (comment) where even if the database column is timestamp with timezone/timestamp, the JDBC Sink Connector still cannot insert data into the database because you get the error

Caused by: org.postgresql.util.PSQLException: ERROR: column "df_created" is of type timestamp with time zone but expression is of type character varying
  Hint: You will need to rewrite or cast the expression.

Note that this PR doesn't update the create table statements to use timestamp with timezone/timestamp as I don't think its possible to do this comprehensively. I am not sure about avro, but I use json with json schema and from what I can tell the connect API doesn't have access to the format field in the JSON RFC Schema spec which would tell it to create a column with the correct types, i.e.

"meta_ingested_at" : {
    "type" : "string",
    "format" : "datetime"
}

(in this case the datetime specifically tells us that you would need to create a timestamp with timezone column, where as if its datetime-local it would be timestamp).

Solution

The solution to this issue is to cast the expressions to timestamptz/timestamp using Postgres's JDBC drivers native support with OffsetDateTime/LocalDateTime (see https://tada.github.io/pljava/use/datetime.html).

Note that there is a workaround, i.e. using ?stringtype=unspecified from #921 (comment) however this has the following issues

  • It doesn't work with supported connectors on the confluent platform as there is no way to configure the JDBC connection string
  • It may have unintended consequences for other column types or in other words the reason why this is a workaround is because your telling the postgres server to assume the type and automatically cast it to what it thinks is correct. The more principled solution is the statements being sent to postgres having the correct type in the first place.
Does this solution apply anywhere else?
  • yes
  • no
If yes, where?

Test Strategy

Testing done:
  • Unit tests
  • Integration tests
  • System tests
  • Manual tests

Release Plan

This feature should be safe to release as it only fixes currently broken behaviour and doesn't have an impact otherwise. If you happen to be using the ?stringtype=unspecified workaround then it will stop applying after this fix as the workaround only effects types with no casts (and the point of this PR is to add a cast).

I can handle back-porting to older versions of connectors if necessary

@mdedetrich mdedetrich requested a review from a team as a code owner March 4, 2025 14:13
@confluent-cla-assistant
Copy link

confluent-cla-assistant bot commented Mar 4, 2025

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ mdedetrich
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@@ -280,6 +312,13 @@ protected ColumnConverter columnConverterFor(
if (UUID.class.getName().equals(columnDefn.classNameForType())) {
return rs -> rs.getString(col);
}

if (OffsetDateTime.class.getName().equals(columnDefn.classNameForType())) {
return rs -> rs.getObject(col, OffsetDateTime.class).format(ISO_OFFSET_DATE_TIME);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using format here with explicit formatters is strictly not necessary as you could use toString instead (the value output is the same), but the usage of an explicit formatter is better as there is no risk that it would change in the future (where as .toString representations can)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant