Fix Tendermint bug that requires state exports to have two nodes

[ValarDragon]

Whenever we make a state export based testnet, we need at least two nodes to be peered in order for a node to start. This is a massive overhead in running state export based testnets. We have isolated where the bug is, have not figured out yet what is the abstraction preserving fix.

We wait for this to get called in testnet: https://github.com/tendermint/tendermint/blob/v0.34.14/blockchain/v0/reactor.go#L321-L322 (This is the message we see once we get our peer and move to actually starting the nodes)
If we look into pool.IsCaughtUp, it just waits for one peer. https://github.com/tendermint/tendermint/blob/master/internal/blocksync/pool.go#L199-L212

[okwme]

based on the fact that an exported state denotes some previous network, i could imagine that tendermint wants to ensure that there have been no new blocks produced from other block producers and waits for at least one node to confirm it? thus the IsCaughtUp name for the method. I wouldn't be sure of course as I am very far from being aware of the expectations.

@creachadair pointed out that there may be more context inside of docs/architecture/adr-040-blockchain-reactor-refactor.md but also said that he didn't see anything obvious about this case.

would it make sense to move this issue to the tendermint repo to have a single place to track it? or at least reference between issues on both repos for discoverability.

[okwme]

made an issue on tendermint: tendermint/tendermint#7557

[cmwaters]

Just dug around a little and it seems like this bug is solved in v0.35 and master by a global timeout (hard coded to 60 seconds):

https://github.com/tendermint/tendermint/blob/7f8b75e1ee6bbd838d64923dab1a7b271cc727aa/internal/blocksync/reactor.go#L472

This means a single node from an exported state should be able to progress after that timeout because it will fallback to the consensus reactor

[cmwaters]

For v0.34 versions, I would suggest disabling blocksync in the config before starting so the node immediately goes to cosensus.

[ValarDragon]

Oh awesome! Thank you so much for investigating @cmwaters. @czarcas7ic can we add this to our testnet scripts, its turning fast_sync off in the config.toml

[czarcas7ic]

@ValarDragon when setting up testnet V2, setting fast_sync to false did not solve the tendermint bug issue

[ValarDragon]

ah yikes :/

[ValarDragon]

Done, fast_sync did fix it upon further learnings