disconnect call getting blocked with keepalive thread.

[upendra1024]

I'm using v_0_13_0 mysql-binlog-connector-java.
We have code to reconnect if any issue like - connection issue or new master take over.
And all things work fine.
But sometimes it getting disconnect without any onCommunicationFailure(). In this case, it never connects again.

for your info, I'm using blocking BinaryLogClient connect().

Below is the part of thread dump :

"blc-keepalive-172.25.231.81:3306" #33 prio=5 os_prio=0 tid=0x00007fbd5425e800 nid=0x9368 waiting for monitor entry [0x00007fbda4bc7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.github.shyiko.mysql.binlog.BinaryLogClient.registerLifecycleListener(BinaryLogClient.java:1078)

• waiting to lock <0x0000000641891688> (a java.util.LinkedList)
  at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:785)
  at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:745)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

....
......

"KafkaBinlogReader-172_25_231_81" #22 prio=5 os_prio=0 tid=0x00007fbee8e9e800 nid=0x730f waiting on condition [0x00007fbda5ad7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x0000000640d77040> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
  at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
  at com.github.shyiko.mysql.binlog.BinaryLogClient.awaitTerminationInterruptibly(BinaryLogClient.java:1130)
  at com.github.shyiko.mysql.binlog.BinaryLogClient.terminateKeepAliveThread(BinaryLogClient.java:1122)
  at com.github.shyiko.mysql.binlog.BinaryLogClient.disconnect(BinaryLogClient.java:1112)
  at com.zoho.mysqlbackup.producer.KafkaBinLogReader$BinLogLifeCycleListener.onDisconnect(KafkaBinLogReader.java:1018)
  at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:565)
• locked <0x0000000641891688> (a java.util.LinkedList)
  at com.zoho.mysqlbackup.producer.KafkaBinLogReader.connect(KafkaBinLogReader.java:1300)
  at com.zoho.mysqlbackup.producer.KafkaBinLogReader.readBinLog(KafkaBinLogReader.java:303)
  at com.zoho.mysqlbackup.producer.KafkaBinLogReader.run(KafkaBinLogReader.java:270)

[shyiko]

Confirmed.

com.zoho.mysqlbackup.producer.KafkaBinLogReader$BinLogLifeCycleListener.onDisconnect(KafkaBinLogReader.java:1018)
is called while holding BinaryLogClient::lifecycleListeners lock. Executing com.github.shyiko.mysql.binlog.BinaryLogClient.disconnect(BinaryLogClient.java:1112) inside it inherirts the lock, preventing com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:785) from moving forward (at com.github.shyiko.mysql.binlog.BinaryLogClient.registerLifecycleListener(BinaryLogClient.java:1078)). In other words, we've got a deadlock here (as com.github.shyiko.mysql.binlog.BinaryLogClient.disconnect(BinaryLogClient.java:1112) is waiting for keepAliveThread to terminate).

A fix is simple - change eventListeners & lifecycleListeners to CopyOnWriteArrayList (which together with removal of all that synchronized business will ensure there are no locks escaping BinaryLogClient boundary).

I'll double-check everything and make a release within the next couple of days.

Until then, you can avoid situation above by executing com.github.shyiko.mysql.binlog.BinaryLogClient.disconnect(BinaryLogClient.java:1112) in a different thread (i.e. change onDisconnect(() -> binaryLogClient.disconect()) to var executor = Executors.newSingleThreadExecutor(); onDisconnect(() -> executor.submit(() -> binaryLogClient.disconect())) (just make sure you are not waiting for executor.submit(...)'s Future inside onDisconnect callback)).

[upendra1024]

Thanks a lot for your quick response and details.
I'll wait for the release.

Such disconnect happening rarely(deadlock).
still could not found the reason behind triggering such disconnect.