Handle etcd compaction while offline

[solidsnakedev]

Context & versions

We have identified an issue related to the etcd file located at /opt/hydra-node/persistence/last-known-revision. When restarting the Hydra node repeatedly, we observed that the file last-known-revision is unexpectedly reset to “0”.

Hydra version
0.22.2-b984abccf65921decc0e36452fb59cd3f478ba7d

Steps to reproduce

Steps to Reproduce the Issue:

Start the Hydra node.

Stop the Hydra node and observe the value in the last-known-revision file (e.g., 6745).

Manually change the last-known-revision value to a lower number (e.g., 3000).

Restart the Hydra node and observe that it is unable to operate. The last-known-revision is reset to 0.

To recover from this state:

Set the last-known-revision back to the original value (e.g., 6745).

Restart the Hydra node.

Run the etcd command:

etcdctl endpoint status --write-out json

This will return the current revision known (in our case, it was 33047).

Update the last-known-revision file with this value.

Restart the Hydra node and verify that it comes back online successfully.

Actual behavior

When restarting the Hydra node repeatedly, we observed that the file last-known-revision is unexpectedly reset to “0”.

Expected behavior

Under normal circumstances, in the event of a node failure, etcd is expected to retrieve the last known revision from the other nodes in the cluster. However, this does not seem to be happening as expected.

[ch1bo]

etcd is expected to retrieve the last known revision from the other nodes in the cluster

No. This is not expected.

Stop the Hydra node and observe the value in the last-known-revision file (e.g., 6745).
Manually change the last-known-revision value to a lower number (e.g., 3000).

You are messing with internal state here. The hydra-node is not guaranteed to be safe against this kind of tampering with its internal state.

I suspect that our default etcd revision compaction setting of 1000 is used. So if the latest known revision is 6745, 3000 will not exist (anymore) and the node can't fetch it. This is probably making it stuck.

@solidsnakedev What is the scenario you are trying to explore here? I would not consider it a bug of the hydra-node if someone changes internal journals inconsistently.

[jmagan]

@ch1bo please note that the revision number was changed manually to help you to reproduce the issue. Normally, it's common that the revision number falls behind the last compaction when the node is down. This is even more critical when you have mirror nodes which continue with the consensus.

The Watch API of ETCD provides history starting from the last compacted revision.

Start_Revision - An optional revision for where to inclusively begin watching. If not given, it will stream events following the revision of the watch creation response header revision. The entire available event history can be watched starting from the last compaction revision.

I think the problem is when you put some start revision before the last compaction. I would say that the unexpected behavior is that if you get this last-know-revision to 0, it stops to receive events and the node halts forever. I would go for either one of these two options: one to continue from the last header revision (not start revision provided) or to look for the last compacted revision to replay the biggest possible amount of the state.

[ch1bo]

@jmagan I am confused to why this would be needed. Of course, we can look into the behavior of the node in such situations, but reading and accepting the latest revision from the cluster will most likely mean that the node's persistence is not matching the messages we get to see from that starting revision.

We should even consider asserting that our last-known-revision was not compacted and stop the process altogether in such situations because the hydra-node can't guarantee it is in a consistent state.

[jmagan]

I think now I understand your reservations. I'll try to put some context to the problem. We have two main possible problems in the Hydra Node universe. One is to stop producing snapshots and the other one is stop receiving network communication.

Particularly, when we loose events due to compaction/downtime/etc... The Hydra node won't be able to continue signing snapshots anymore. That's why you meant that at that case the node should crash to say that we have a inconsistent state.

In our case, an inconsistent state in one of the node is acceptable as we have mirror nodes and we aren't particularly worried to have one inconsistent node. Here it's where side loading snapshots come to play. If the node is continue receiving network events, even these events won't produce new snapshots, we can catch up one of the last snapshots with side loading and that node will begin signing snapshot again.

The problem described here, it's a problem in the initialization of the ETCD events stream. If the start revision is compacted, the node will do nothing, completely dead, anything. So, yes, we won't continue signing snapshots fixing this problem and yes we don't mind to be in a inconsistent state. We only want to have the possibility of recoordinate the stuck node or all the nodes in a latest snapshot but first we need to have network.

[ch1bo]

This makes a lot more sens in the context of running multiple hydra-node instances with the same set of keys (a.k.a. mirror nodes). I updated the title of this bug description to reflect the root cause of the issue we want to handle and implemented a fix here: #2137

[noonio]

Resolved by #2137