Description
Search before asking
- I had searched in the issues and found no similar optimization requirement.
Description
When I use CDCSOURCE, I have a problem with timezone:
I found that during MySQLCDC['server-time-zone' = 'Asia/Shanghai'] synchronization , the timestamp obtained by flinkcdc's streamSource will be increased by 8 hours based on Asia/Shanghai.
I try to adjust io.debezium.time.ZonedTimestamp['source.server-time-zone' = 'UTC'], but it doesn't work, the flinkcdc version I use is 2.2.1.
So I refer to the issues and solutions of the flinkcdc community and the debezium documentation:
apache/flink-cdc#317
apache/flink-cdc#1366
https://debezium.io/documentation/reference/1.5/connectors/mysql.html#mysql-connector-properties
The conclusion I got is: the processing logic of flinkcdc's timestamp is:
mysql-debezium[time with timezone] -> flinkcdc-Source[time with UTC] -> sink-end[time with UTC],
Refer to RowDataDebeziumDeserializeSchema used in flinkCDC-sql (default serverTimezone = UTC).
But after I tried to overwrite JsonDebeziumDeserializationSchema, the packaging caused me troubles. I couldn't make it work properly after maven install, and it would increase parsing and cause performance loss.
Therefore, after consulting @aiwenmo, you can process the timestamp by modifying SQLSinkBuilder.convertValue and replace ZoneId.systemDefault with ZoneId.of("UTC").
当我使用CDCSOURCE时,我遇到了timezone的问题,我发现在进行MySQLCDC[server-time-zone = 'Asia/Shanghai']同步时,flinkcdc的streamSource获取到的时间戳会基于Asia/Shanghai再次往后增加8个小时。
我尝试调整io.debezium.time.ZonedTimestamp['source.server-time-zone' = 'UTC'],但是它不生效,我使用的flinkcdc版本是2.2.1.
因此我参考了flinkcdc社区的issue和解决方案以及debezium文档:
apache/flink-cdc#317
apache/flink-cdc#1366
https://debezium.io/documentation/reference/1.5/connectors/mysql.html#mysql-connector-properties
我得到的结论是:在flinkcdc的处理逻辑下,timestamp的转换逻辑是:
mysql-debezium[time with timezone] -> flinkcdc-Source[time with UTC] -> sink-end[time with UTC],
参考flinkCDC-sql中使用的RowDataDebeziumDeserializeSchema(默认serverTimezone = UTC).
但是我尝试重写JsonDebeziumDeserializationSchema后,打包造成了我的困扰,我并不能使其在maven install之后正常工作,并且会增加一次解析,造成性能损失,因此在咨询了@aiwenmo大佬后,可以通过修改SQLSinkBuilder.convertValue对timestamp进行处理,替换ZoneId.systemDefault为ZoneId.of("UTC")。
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Assignees
Labels
Type
Projects
Status
Done