when mysql sink's domain reflects to another IP, ticdc changefeed can't auto-resume

[lichunzhu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   Use TiCDC to replicate data to a MySQL sink. The downstream sink URI is a domain that reflects a MySQL database and its follower database. When the domain the reflection from the follower database to the leader database (the follower database meets some errors), TiCDC's changefeed can't auto-resume unless we resume this changefeed manually.

2. What did you expect to see?
   TiCDC won't pause or can auto-resume when a switch happens.

3. What did you see instead?
   We need to resume changefeed manually.
   TiCDC log reported

[2021/01/18 18:34:51.546 +08:00] [WARN] [mysql.go:722] ["execute DMLs with error, retry later"] [error="[CDC:ErrMySQLTxnError]Error 3100: Error on observer while running replication hook 'before_commit'."]
[2021/01/18 18:34:52.195 +08:00] [WARN] [mysql.go:722] ["execute DMLs with error, retry later"] [error="[CDC:ErrMySQLTxnError]Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement"]

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • TiCDC version (execute cdc version):

     (paste TiCDC version here)
     

     v4.0.10

[amyangfei]

Yep, TiCDC doesn't have an auto resume mechanism and it seems there is no explicit error in this situation, such as something that we can tell it is an error that VIP has changed to another real server.
What's is the behavior in DM in this scenario, seems DM can detect a task error and recovers the task automatically with new database connections.

[overvenus]

It's more like a feature request, we will add it to TiCDC roadmap, thanks!