Disaster recovery integration test

[arhag]

Scenario 1

Create integration test with 4 nodes (A, B, C, and D) which each have their own producer and finalizer. The finalizer policy consists of the four finalizers with a threshold of 3. The proposer policy involves all four proposers.

The 4 nodes are cleanly shutdown in the following state:

• A has LIB N. A has a finalizer safety information file that locks on a block after N.
• B, C, and D have LIB less than N. They have finalizer safety information files that lock on N.

Nodes B, C, and D lose their reversible blocks. All nodes restart from an earlier snapshot.

A is restarted and replays up to its last reversible block (which is a block number greater than N) after restarting from snapshot. Blocks N and later is sent to the other nodes B, C, and D after they are also started up again.

Verify that LIB advances and that A, B, C, and D are eventually voting strong on new blocks.

Scenario 2

Create an integration test with 5 nodes (A, B, C, D, and P). Nodes A, B, C, and D each have one finalizer but no proposers. Node P has a proposer but no finalizers. The finalizer policy consists of the four finalizers with a threshold of 3. The proposer policy involves just the single proposer P.

A, B, C, and D can be connected to each other however we like as long as blocks sent to node A can traverse to the other nodes B, C, and D. However, node P should only be connected to node A.

At some point after IF transition has completed and LIB is advancing, block production on node P should be paused. Enough time should be given to allow and in-flight votes on the latest produced blocks to be delivered to node P. Then, the connection between node P and node A should be severed, and then block production on node P resumed. The LIB on node P should advance to but then stall at block N. Then shortly after that, node P should be cleanly shut down.

Verify that the LIB on A, B, C, and D has stalled and is less than block N. Then, nodes A, B, C, and D can all be cleanly shut down.

Then, reversible blocks from all nodes should be removed. All nodes are restarted from an earlier snapshot (prior to block N).

P is restarted and replays up to block N after restarting from snapshot. Blocks up to and including block N are sent to the other nodes A, B, C, and D after they are also started up again.

Verify that LIB advances and that A, B, C, and D are eventually voting strong on new blocks.

Scenario 3

Create integration test with 4 nodes (A, B, C, and D) which each have their own producer and finalizer. The finalizer policy consists of the four finalizers with a threshold of 3. The proposer policy involves all four proposers.

• At least two of the four nodes should have a LIB N and a finalizer safety information file that locks on a block after N. The other two nodes should have a LIB that is less than or equal to block N.

All nodes are shut down. The reversible blocks on all nodes is deleted. Then restart all nodes from an earlier snapshot.

All nodes eventually sync up to block N. Some nodes will consider block N to LIB but others may not.

Not enough finalizers should be voting because of the lock in their finalizer safety information file. Verify that LIB does not advance on any node.

Cleanly shut down all nodes and delete their finalizer safety information files. Then restart the nodes.

Verify that LIB advances on all nodes and they all agree on the LIB. In particular, verify that block N is the same ID on all nodes as the one before nodes were first shutdown.