Summary of Bug

One issue that we are noticing when upgrading to v0.47.0 is that accessing IAVL tree store at height 0 returns an error. The bad side effects for this is that RPC calls at height 0 will start resulting in errors.

Similarly in PrepareProposalHandler, when accessing the state via the context throws a runtime error and halts the chain.

IIUC, the state is not ready at height 0. Or perhaps, height 0 is reserved for the app to get initialized.

Previously, this did not cause an error because the default behavior for accessing the state at height == 0 was to return an empty state. But this changed as of this PR

It seems like there are paths that trigger the app to access the state at height 0 and this access should not happen. If so, should there be a better handling of such paths? For example, adding "if height == 0, this access should be rejected, rather than throwing an error".

Shoutout to Lucas for debugging this. He discovered:

Error is being returned when creating the context for the query. It seems that the IAVL tree does not exist at height 0, so the height hasn't been created yet (here is the VersionExists method on the IAVL struct).
Our previous Cosmos SDK version previously returned an empty IAVL tree in this scenario. PR that introduced the change is here.

Version

v0.47.0-alpha2

Steps to Reproduce

For the RPC path, steps to reproduce:

1. have a thread that ping some RPC endpoint that accesses IAVL store repeatedly
2. start the node
3. observe error logs

For the PrepareProposal path,

1. in PrepareProposalHandler call a method that accesses state via ctx
   Note that this might not crash the node after adding panic recovery