-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose serverTimezone debezium option via MySQL Source spec for CDC #6573
Comments
@digital-thinking this is a problem between mysql/debezium. I found this Stack Overflow discussion: https://stackoverflow.com/questions/67340500/kafka-connect-mysql-server-time-zone-value-edt-is-unrecognized can you give it a try? |
@marcosmarxm seems like the same issue, do you know, where I can change that config in airbyte? |
Hi, while trying to setup CDC on a MariaDB database I encountered the same issue with Airbyte 0.33.5. |
@dcoueron No, and we abandoned airbyte for performance reasons. |
Facing the same issue on 0.39.16-alpha with CDT timezone. JDBC URL Params is set to
|
Also encountering this issue as well. @marcosmarxm can this issue be reopened? This appears to still be an issue with CDC not taking JDBC pramas. Or should we open a new Issue on this? Also with #7600 closed, it mentions a google docs are these issues being tracked and actively worked on? |
@subodh1810 are JDBC parameters even relevant for Debezium connection to MySQL? |
@grishick So based on @marcosmarxm comment there's a change in the debezium config that could work but that configuration is not exposed in the Airbyte UI for us to control. That's more likely the real fix, than it is JDBC parameters. The problem right now is I can't change it. Here's the current error we get from the airbyte logs. This is using MySQL with CDC and Incremental Append selected. Here's what the logs show for the debezium configuration.
per that stackoverflow link there's a debezium option for
So, it just seems like either there's a change that needs to optionally include this config value when it's needed. |
@grishick this is same as https://github.com/airbytehq/alpha-beta-issues/issues/183 |
Closing as duplicate of https://github.com/airbytehq/alpha-beta-issues/issues/183 |
@grishick So since this is closed, and there's a duplicate ticket on a board not accessible to the public.
|
@gsmith-schlesinger yes, the duplicate ticket is https://github.com/airbytehq/alpha-beta-issues/issues/183 (see @subodh1810's comment above) |
My apologies, I didn't realize alpha-beta was a closed repo. I'll reorganize these tickets to keep the public one open |
@digital-thinking @gsmith-schlesinger @agiletich we need some help in order to reproduce this. I and @akashkulk have been trying to setup a mysql database with another timezone and sync it with Airbyte in order to reproduce this but could not reproduce this issue. we followed the steps from here and here to setup the mysql database. Can you share your mysql setup so that we can replicate the same setup in our mysql instance. |
Hey @subodh1810! I'm not the DBA but with the access I have, I ran
So seeing as the error was inside Java specifically when dealing with I hope that helps if there's any other values you need from this to help configure your MySQL. I'll also make note that this MySQL server is setup via Azure. |
I was able to reproduce this error by setting the timezone to CEST (Stack Trace). The issue here is that Debezium is using the ZoneID library to figure out whether the given timezone is valid, and CEST is not in this list. You can see the full list here. You can verify this since spinning up a MySQL DB in a supported timezone (non-UTC) and verifying that data syncs properly. In this case, there is a drift between what MySQL considers a valid timezone and what the ZoneId considers to be valid. ZoneId supports timezones that follow the following standards :
They also provide a mapping of legacy timezones to the ones that adhere to the new standard, but unfortunately 'CEST' is not in them. Couple of options to address this:
|
@akashkulk did you try to run the sync by setting the debezium parameter |
Yes, this was by setting that parameter in MySQL properties |
update : So me and @akashkulk spend pairing on this issue and we have been able to reproduce this issue and identified a fix as well. @akashkulk is going to raise a PR for the fix. |
Hello all,
A full refresh works, as it is using the JDBC connection params defined in the UI, but the incremental update seems to ignore the configured jdbc parameters and fails.
Enviroment
Current Behavior
Fails to sync due MySQL errors, which are happening because the jdbc params are not applied in incemental mode (binlog)
Expected Behavior
Should use the JDBC params defined in the source settings and sync.
Logs
LOG
The text was updated successfully, but these errors were encountered: