Skip to content

Shut down ReplicationWorker and Auditor on non-recoverable ZK error#3374

Merged
eolivelli merged 1 commit into
apache:masterfrom
dlg99:ar-zk-shutdown
Jul 12, 2022
Merged

Shut down ReplicationWorker and Auditor on non-recoverable ZK error#3374
eolivelli merged 1 commit into
apache:masterfrom
dlg99:ar-zk-shutdown

Conversation

@dlg99

@dlg99 dlg99 commented Jun 28, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Descriptions of the changes in this PR:

Shut down Replication Worker and Auditor on non-recoverable ZK error

Motivation

Some errors require one to re-create Zk client.
Currently BK and underlying components cannot do that transparently.
When running AutoRecovery as a separate service it does not seem to crash in such cases and keeps on running while unable to do anything useful and requires operator restarting the service manually.

One can see such log messages like

ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - UnavailableException while replicating fragments
org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Error contacting zookeeper
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:610) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:264) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:230) [com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.76.Final.jar:4.1.76.Final]
   at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ledgers/underreplication/ledgers/0000/0001/0ebb
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
   at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2589) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
   at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$3701(ZooKeeperClient.java:70) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.zookeeper.ZooKeeperClient$27.call(ZooKeeperClient.java:1251) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.zookeeper.ZooKeeperClient$27.call(ZooKeeperClient.java:1245) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.zookeeper.ZooKeeperClient.getChildren(ZooKeeperClient.java:1245) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager$1.getChildren(ZkLedgerUnderreplicationManager.java:147) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.util.SubTreeCache.getChildren(SubTreeCache.java:118) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicateFromHierarchy(ZkLedgerUnderreplicationManager.java:550) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicateFromHierarchy(ZkLedgerUnderreplicationManager.java:562) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicateFromHierarchy(ZkLedgerUnderreplicationManager.java:562) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicateFromHierarchy(ZkLedgerUnderreplicationManager.java:562) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
   at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:603) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]

2022-06-24T17:15:55,405 [ZkLedgerManagerScheduler-11-1] ERROR org.apache.bookkeeper.replication.Auditor - Underreplication manager unavailable running periodic check
org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Error contacting zookeeper
	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:731) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.replication.Auditor.lambda$checkAllLedgers$7(Auditor.java:1254) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.lambda$operationComplete$0(AbstractZkLedgerManager.java:573) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.76.Final.jar:4.1.76.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ledgers/underreplication/disable
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2021) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2301(ZooKeeperClient.java:70) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:833) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:827) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:827) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2049) ~[com.datastax.oss-pulsar-zookeeper-2.7.2.1.1.33.jar:2.7.2.1.1.33]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2401(ZooKeeperClient.java:70) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:854) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:848) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:848) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]
	at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:726) ~[com.datastax.oss-bookkeeper-server-4.14.5.1.0.0.jar:4.14.5.1.0.0]

and other similar

Changes

Now Replication Worker and Auditor will shut down on such errors making their error state visible / letting k8s or service monitor restart them.

Added tests.

Removed KeeperException from some interfaces/implementations to prevent raw ZK exception sneaking through but there are other interfaces, see #3373

StevenLuMT
StevenLuMT approved these changes Jun 29, 2022
View reviewed changes

@StevenLuMT StevenLuMT left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

eolivelli
eolivelli approved these changes Jul 12, 2022
View reviewed changes

@eolivelli eolivelli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eolivelli
eolivelli merged commit c3706e9 into apache:master Jul 12, 2022
@hangc0276 hangc0276 added this to the 4.16.0 milestone Jul 25, 2022
zymap pushed a commit that referenced this pull request Aug 2, 2022
@dlg99 @zymap
Shut down ReplicationWorker and Auditor on non-recoverable ZK error (#…
0c3600d 
…3374)

(cherry picked from commit c3706e9)
hangc0276 pushed a commit to hangc0276/bookkeeper that referenced this pull request Nov 5, 2022
@dlg99 @hangc0276
Shut down ReplicationWorker and Auditor on non-recoverable ZK error (a…
4ccb72e 
…pache#3374)

(cherry picked from commit c3706e9)
hangc0276 pushed a commit to hangc0276/bookkeeper that referenced this pull request Nov 7, 2022
@dlg99 @hangc0276
Shut down ReplicationWorker and Auditor on non-recoverable ZK error (a…
8ce6538 
…pache#3374)

(cherry picked from commit c3706e9)
nicoloboschi pushed a commit to datastax/bookkeeper that referenced this pull request Jan 11, 2023
@dlg99 @nicoloboschi
Shut down ReplicationWorker and Auditor on non-recoverable ZK error (a…
0412272 
…pache#3374)

(cherry picked from commit c3706e9)
(cherry picked from commit 8ce6538)
@dlg99 dlg99 mentioned this pull request Feb 22, 2023
Closed
Ghatage pushed a commit to sijie/bookkeeper that referenced this pull request Jul 12, 2024
@dlg99
Shut down ReplicationWorker and Auditor on non-recoverable ZK error (a…
585ff1e 
…pache#3374)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eolivelli eolivelli eolivelli approved these changes
@StevenLuMT StevenLuMT StevenLuMT approved these changes

Assignees

@dlg99 dlg99

Labels

area/autorecovery cherry-picked/branch-4.14 cherry-picked/branch-4.15 release/4.14.6 release/4.15.1 type/bug

Projects

None yet

Milestone

4.16.0

Development

Successfully merging this pull request may close these issues.

5 participants

@dlg99 @eolivelli @StevenLuMT @hangc0276 @zymap