uneven store heights prevent pruning and state sync

[faddat]

Hi, this issue is to track a set of issues, and may become an epic.

Pruning and/ or state sync

• dig
• konstellation
  • code too old for valid comparisons
• akash
• stride

---

guess

This is because of issues in the CMD folder, and possible validator misconfiguration.

I think that this can be fixed by assisting crypto.org with migrating, then compacting their archival node state, so that they can use fast node and pebble, which makes archives much more performant.

Some of these issues were introduced here:

https://github.com/cosmos/iavl/commits/master#:~:text=feat%3A%20Add%20skipFastStorageUpgrade%20to%20MutableTree%20to%20control%20fast%20s%E2%80%A6

[faddat]

Related

notional-labs/dig#423

[faddat]

Related

notional-labs/dig#426

[chillyvee]

Comdex v5.0.0 also has broken pruning
https://github.com/comdex-official/comdex/releases/tag/v5.0.0

[chillyvee]

Konstellation snashot/pruning broke because feegrant and authz (in rootmulti) are the wrong height (tree.ndb is nil)
Team notified, no response

[chillyvee]

Comdex snashot/pruning broke because lendV1 store (in rootmulti) is the wrong height (tree.ndb is nil)
Team notified, no repsonse

[faddat]

.... stride has this issue, as well, in v3.0.1

[chillyvee]

Stride just hit the same problem with their authz store being uninitialized and caused issues for validators.
vishal is aware, says it makes sense, and is trying to fix it

[faddat]

probably because of the uneven iavl state height issue that @chillyvee is mentioning here, state sync broke on stride:

git clone https://github.com/stride-labs/stride
git checkout v3.0.0
bash scripts/statesync.bash

I don't remember having issues with uneven state heights in the past when adding modules.

[Reecepbcups]

@faddat Juno was having block header apphash issue as well back on Nov 7th.

Golden , Windpower, TedCrypto, Coldchain - fixed when bumped GO 1.19.2 from 1.18.3

[faddat]

@Reecepbcups really good.

Let's break that out separately, though. I'll maybe do issue/pr for it, doing some 'splain... I think I know why that occurs....

The mixed go runtime issue is now here:

#13976

@chillyvee 's PR related to this issue is here:

#13935

[chillyvee]

another validator also reports statesync for konstellation still doesnt work (maybe a store with information with the wrong height?)

[faddat]

@chillyvee -- my feeling is that you've correctly identified the problem.

Not certain just yet.

Here's my stride working branch:

https://github.com/Stride-Labs/stride/tree/no-state-breaks

That said, I figure that the fix is somewhere between here, iavl, and ibc-go, or such.