KVStore queries return empty when starting from `genesis.json` with non-zero initial_height

[michaelfig]

Summary of Bug

When the genesis.json has a non-zero initial_height, the module KVStores appear to be empty

Version

v0.40.0-rc3

Steps to Reproduce

1. initialize a single-node chain
2. start the chain node
3. appd query auth params - note that output looks normal
4. appd export >& exported.json
5. appd unsafe-reset-all
6. cp exported.json ~/.appd/config/genesis.json
7. start the chain node, note that it has a nonzero initial height
8. appd query auth params fails with:

Error: RPC error -32603 - Internal error: UnmarshalJSON cannot decode empty bytes

Verifying with other modules (such as my custom x/swingset module), they all have empty KVStores when starting from nonzero height, despite their InitGenesis functions running correctly.

Note that if we replace copy of exported.json (step 6) to instead be jq '. + { initial_height: "0" }' exported.json > ~/.appd/config/genesis.json then it all works (but of course, starts the chain with zero height which I don't want).

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[zmanian]

Blocker for completing the upgrade simulation for the Hub.

[zmanian]

I bet this about store indexing with non-zero-height starts.

[alexanderbez]

Is there a more verbose stracktrace you can provide @michaelfig? I presume this affects all stores/modules? I want to know where/when it reads the store and panics. Also the chain starts successfully but you can't query any state?

[zmanian]

What we experienced on cosmoshub-test-stargate was the chain started correctly but all query returned nothing.

We aren't seeing panics just empty responses.

I really suspect this is an indexing problem. Startup takes tens of minutes writing data to the stores from the genesis file.

It just seems like during queries it is trying to load stores that don't exist.

[alexanderbez]

Ahh I see, then that sounds a little different than what @michaelfig is describing. It sounds like data is being properly initialized and persisted during InitChain (albeit slowly unfortunately), but the stores are somehow incorrectly configured s.t. they're returning empty responses most likely due to height misconfiguration?

[michaelfig]

Ahh I see, then that sounds a little different than what @michaelfig is describing. It sounds like data is being properly initialized and persisted during InitChain (albeit slowly unfortunately), but the stores are somehow incorrectly configured s.t. they're returning empty responses most likely due to height misconfiguration?

Same issue... The specific query I made panicked only because it was trying to unmarshal the nil kvstore content.

[zmanian]

Ahh I see, then that sounds a little different than what @michaelfig is describing. It sounds like data is being properly initialized and persisted during InitChain (albeit slowly unfortunately), but the stores are somehow incorrectly configured s.t. they're returning empty responses most likely due to height misconfiguration?

Queries returning empty responses most likely due to height misconfiguration is what I think is going on.

[alexanderbez]

Ok, I will do some debugging then.