IF: Add the beginning of a savanna disaster recovery test

[heifner]

Integration test with 4 finalizers (A, B, C, and D).

• The 4 nodes are cleanly shutdown in the following state:
  • A has LIB N. A has a finalizer safety information file that locks on a block after N.
  • B, C, and D have LIB less than N. They have finalizer safety information files that lock on N.

All nodes but A lose their reversible blocks and restart from an earlier snapshot.

A is restarted and replays up to block N after restarting from snapshot. Block N is sent to the other
nodes B, C, and D after they are also started up again.

Verify that LIB advances and that A, B, C, and D are eventually voting strong on new blocks.

[greg7mdp]

But yet we can't use waitForLibToAdvance() at line 87?

[heifner]

Would need to capture the LIB above and wait for LIB+1, but there is inherit race conditions on the get_info calls and where we are in the test. Waiting for head to advance should be sufficient.

[greg7mdp]

It would be nice if "Node::kill" would return the lib right before the node is killed, so we could verify the assertion that node0 has higher lib than 1,2,3 when it is killed.

[heifner]

Would need to report in a log statement at exit or use leap util to look at the block log.

[greg7mdp]

Can't we do in Node.py?
currentLib = self.getIrreversibleBlockNum()

Looks like it does a get_info

[heifner]

Yes, but LIB can change immediately after that call or right before that call.

[greg7mdp]

Yes, but still it is the best indication of what lib was right before the node is killed.

[ericpassmore]

Note:start
group: IF
category: TEST
summary: Disaster recovery test with four finalizers. Ensure block N on one node may be recovered after losing reservable blocks and starting from snapshot.
Note:end