Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic to limit DateTime max #373

Closed
subkanthi opened this issue Nov 12, 2023 · 2 comments · Fixed by #379
Closed

Add logic to limit DateTime max #373

subkanthi opened this issue Nov 12, 2023 · 2 comments · Fixed by #379
Assignees

Comments

@subkanthi
Copy link
Collaborator

debezium-embedded_1 | java.lang.IllegalArgumentException: DateTime(-1420092000) should be between -1420070400 and 9904550399 inclusive of both values

Currently only DateTime64 limits are checked in the sink connector, if the DateTime limits exceeds UInt64, clickhouse-jdbc throws an error.

Solution: Limit the value before inserting into CH.

@subkanthi subkanthi self-assigned this Nov 12, 2023
@aadant
Copy link
Collaborator

aadant commented Nov 12, 2023

@subkanthi it makes sense to do it on Date and Date32 too. same potential problem affecting the sink connector.

@subkanthi
Copy link
Collaborator Author

subkanthi commented Nov 13, 2023

Changing the logic to write Date as String still throws the same exception

        } else if (isFieldTypeInt) {
            if (schemaName != null && schemaName.equalsIgnoreCase(Date.SCHEMA_NAME)) {
                // Date field arrives as INT32 with schema name set to io.debezium.time.Date
                ps.setString(index, DebeziumConverter.DateConverter.convertToString(value));
                //ps.setDate(index, DebeziumConverter.DateConverter.convert(value));
42043 [pool-143-thread-1] INFO  com.altinity.clickhouse.sink.connector.db.DbWriter  - *** QUERY***insert into temporal_types_DATE(`Type`,`Minimum_Value`,`Mid_Value`,`Maximum_Value`,`Null_Value`,`_version`,`is_deleted`) select `Type`,`Minimum_Value`,`Mid_Value`,`Maximum_Value`,`Null_Value`,`_version`,`is_deleted` from input('`Type` String,`Minimum_Value` Date32,`Mid_Value` Date32,`Maximum_Value` Date32,`Null_Value` Nullable(Date32),`_version` UInt64,`is_deleted` UInt8')
42043 [pool-143-thread-1] INFO com.altinity.clickhouse.sink.connector.db.DbWriter  - *** QUERY***insert into temporal_types_DATE(`Type`,`Minimum_Value`,`Mid_Value`,`Maximum_Value`,`Null_Value`,`_version`,`is_deleted`) select `Type`,`Minimum_Value`,`Mid_Value`,`Maximum_Value`,`Null_Value`,`_version`,`is_deleted` from input('`Type` String,`Minimum_Value` Date32,`Mid_Value` Date32,`Maximum_Value` Date32,`Null_Value` Nullable(Date32),`_version` UInt64,`is_deleted` UInt8')
42045 [pool-143-thread-1] ERROR com.altinity.clickhouse.sink.connector.db.DbWriter  - ******* ERROR inserting Batch *****************
java.lang.IllegalArgumentException: Date(-16437) should be between -16436 and 114635 inclusive of both values
	at com.clickhouse.data.ClickHouseChecker.newException(ClickHouseChecker.java:17)
	at com.clickhouse.data.ClickHouseChecker.between(ClickHouseChecker.java:88)
	at com.clickhouse.data.format.BinaryDataProcessor$Date32SerDe.serialize(BinaryDataProcessor.java:206)
	at com.clickhouse.data.ClickHouseDataProcessor.write(ClickHouseDataProcessor.java:534)
	at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.addBatch(InputBasedPreparedStatement.java:345)
	at com.altinity.clickhouse.sink.connector.db.DbWriter.addToPreparedStatementBatch(DbWriter.java:457)
	at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.flushRecordsToClickHouse(ClickHouseBatchRunnable.java:199)
	at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.processRecordsByTopic(ClickHouseBatchRunnable.java:169)
	at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.run(ClickHouseBatchRunnable.java:101)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants