You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/main/asciidoc/_chapters/ops_mgt.adoc
+35-88Lines changed: 35 additions & 88 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2433,26 +2433,22 @@ Replication State Storage::
2433
2433
`ReplicationPeerStorage` and `ReplicationQueueStorage`. The former one is for storing the
2434
2434
replication peer related states, and the latter one is for storing the replication queue related
2435
2435
states.
2436
-
HBASE-15867 is only half done, as although we have abstract these two interfaces, we still only
2437
-
have zookeeper based implementations.
2436
+
And in HBASE-27109, we have implemented the `ReplicationQueueStorage` interface to store the replication queue in the hbase:replication table.
2438
2437
2439
2438
Replication State in ZooKeeper::
2440
2439
By default, the state is contained in the base node _/hbase/replication_.
2441
-
Usually this nodes contains two child nodes, the `peers` znode is for storing replication peer
2442
-
state, and the `rs` znodes is for storing replication queue state.
2440
+
After 3.0.0, it only contains one child node, but before 3.0.0, we still use zk to store queue data.
2443
2441
2444
2442
The `Peers` Znode::
2445
2443
The `peers` znode is stored in _/hbase/replication/peers_ by default.
2446
2444
It consists of a list of all peer replication clusters, along with the status of each of them.
2447
2445
The value of each peer is its cluster key, which is provided in the HBase Shell.
2448
2446
The cluster key contains a list of ZooKeeper nodes in the cluster's quorum, the client port for the ZooKeeper quorum, and the base znode for HBase in HDFS on that cluster.
2449
2447
2450
-
The `RS` Znode::
2451
-
The `rs` znode contains a list of WAL logs which need to be replicated.
2452
-
This list is divided into a set of queues organized by region server and the peer cluster the region server is shipping the logs to.
2453
-
The rs znode has one child znode for each region server in the cluster.
2454
-
The child znode name is the region server's hostname, client port, and start code.
2455
-
This list includes both live and dead region servers.
2448
+
The `Queue` State::
2449
+
After 3.0.0, the `Queue` has been stored in the hbase:replication table, where the row key is <PeerId>-<ServerName>[/<SourceServerName>], the WAL group will be the qualifier, and the serialized ReplicationGroupOffset will be the value.
2450
+
The ReplicationGroupOffset includes the wal file of the corresponding queue (<PeerId>-<ServerName>[/<SourceServerName>]) and its offset.
2451
+
Because we track replication offset per queue instead of per file, we only need to store one replication offset per queue.
2456
2452
2457
2453
Other implementations for `ReplicationPeerStorage`::
2458
2454
Starting from 2.6.0, we introduce a file system based `ReplicationPeerStorage`, which stores
@@ -2475,14 +2471,14 @@ When nodes are removed from the slave cluster, or if nodes go down or come back
2475
2471
2476
2472
==== Keeping Track of Logs
2477
2473
2478
-
Each master cluster region server has its own znode in the replication znodes hierarchy.
2479
-
It contains one znode per peer cluster (if 5 slave clusters, 5 znodes are created), and each of these contain a queue of WALs to process.
2474
+
Before 3.0.0, for zookeeper based implementation, it is like a tree, we have a znode for a peer cluster, but under the znode we have lots of files.
2475
+
But after 3.0.0, for table based implementation, we have server name in row key, which means we will have lots of rows for a given peer.
2480
2476
Each of these queues will track the WALs created by that region server, but they can differ in size.
2481
2477
For example, if one slave cluster becomes unavailable for some time, the WALs should not be deleted, so they need to stay in the queue while the others are processed.
2482
2478
See <<rs.failover.details,rs.failover.details>> for an example.
2483
2479
2484
2480
When a source is instantiated, it contains the current WAL that the region server is writing to.
2485
-
During log rolling, the new file is added to the queue of each slave cluster's znode just before it is made available.
2481
+
During log rolling, the new file is added to the queue of each slave cluster's record just before it is made available.
2486
2482
This ensures that all the sources are aware that a new log exists before the region server is able to append edits into it, but this operations is now more expensive.
2487
2483
The queue items are discarded when the replication thread cannot read more entries from a file (because it reached the end of the last block) and there are other files in the queue.
2488
2484
This means that if a source is up to date and replicates from the log that the region server writes to, reading up to the "end" of the current file will not delete the item in the queue.
@@ -2521,93 +2517,36 @@ NOTE: WALs are saved when replication is enabled or disabled as long as peers ex
2521
2517
[[rs.failover.details]]
2522
2518
==== Region Server Failover
2523
2519
2524
-
When no region servers are failing, keeping track of the logs in ZooKeeper adds no value.
2525
-
Unfortunately, region servers do fail, and since ZooKeeper is highly available, it is useful for managing the transfer of the queues in the event of a failure.
2526
-
2527
-
Each of the master cluster region servers keeps a watcher on every other region server, in order to be notified when one dies (just as the master does). When a failure happens, they all race to create a znode called `lock` inside the dead region server's znode that contains its queues.
2528
-
The region server that creates it successfully then transfers all the queues to its own znode, one at a time since ZooKeeper does not support renaming queues.
2529
-
After queues are all transferred, they are deleted from the old location.
2530
-
The znodes that were recovered are renamed with the ID of the slave cluster appended with the name of the dead server.
2520
+
When a region server fails, the HMaster of master cluster will trigger the SCP, and all replication queues on the failed region server will be claimed in the SCP.
2521
+
The claim queue operation is just to remove the row of a replication queue, and insert a new row.
2522
+
Here, we use multi row mutate endpoint atomically, so the data for a single peer must be in the same region.
2531
2523
2532
2524
Next, the master cluster region server creates one new source thread per copied queue, and each of the source threads follows the read/filter/ship pattern.
2533
2525
The main difference is that those queues will never receive new data, since they do not belong to their new region server.
2534
-
When the reader hits the end of the last log, the queue's znode is deleted and the master cluster region server closes that replication source.
2535
-
2536
-
Given a master cluster with 3 region servers replicating to a single slave with id `2`, the following hierarchy represents what the znodes layout could be at some point in time.
2537
-
The region servers' znodes all contain a `peers` znode which contains a single queue.
2538
-
The znode names in the queues represent the actual file names on HDFS in the form `address,port.timestamp`.
2539
-
2540
-
----
2541
-
2542
-
/hbase/replication/rs/
2543
-
1.1.1.1,60020,123456780/
2544
-
2/
2545
-
1.1.1.1,60020.1234 (Contains a position)
2546
-
1.1.1.1,60020.1265
2547
-
1.1.1.2,60020,123456790/
2548
-
2/
2549
-
1.1.1.2,60020.1214 (Contains a position)
2550
-
1.1.1.2,60020.1248
2551
-
1.1.1.2,60020.1312
2552
-
1.1.1.3,60020, 123456630/
2553
-
2/
2554
-
1.1.1.3,60020.1280 (Contains a position)
2555
-
----
2526
+
When the reader hits the end of the last log, the queue's record is deleted and the master cluster region server closes that replication source.
2556
2527
2557
-
Assume that 1.1.1.2 loses its ZooKeeper session.
2558
-
The survivors will race to create a lock, and, arbitrarily, 1.1.1.3 wins.
2559
-
It will then start transferring all the queues to its local peers znode by appending the name of the dead server.
2560
-
Right before 1.1.1.3 is able to clean up the old znodes, the layout will look like the following:
2528
+
Given a master cluster with 3 region servers replicating to a single slave with id `2`, the following info represents what the storage layout of queue in the hbase:replication at some point in time.
2529
+
Row key is <PeerId>-<ServerName>[/<SourceServerName>], and value is WAL && Offset.
2-1.1.1.1,60020,123456780 1.1.1.1,60020.1234 (Contains a position)
2535
+
2-1.1.1.2,60020,123456790 1.1.1.2,60020.1214 (Contains a position)
2536
+
2-1.1.1.3,60020,123456630 1.1.1.3,60020.1280 (Contains a position)
2583
2537
----
2584
2538
2585
-
Some time later, but before 1.1.1.3 is able to finish replicating the last WAL from 1.1.1.2, it dies too.
2586
-
Some new logs were also created in the normal queues.
2587
-
The last region server will then try to lock 1.1.1.3's znode and will begin transferring all the queues.
2588
-
The new layout will be:
2539
+
Assume that 1.1.1.2 failed.
2540
+
The survivors will claim queue of that, and, arbitrarily, 1.1.1.3 wins.
2541
+
It will claim all the queue of 1.1.1.2, including removing the row of a replication queue, and inserting a new row(where we change the server name to the region server which claims the queue).
0 commit comments