v0.44.4 `x/capability` map lost between restarts

[michaelfig]

Summary of Bug

When I start a node from genesis, the x/capability map correctly matches IBC portIDs to the right modules. If I restart a node, it appears to lose the mapping somehow and diverges from the nodes that aren't restarted.

Version

v0.44.4

Steps to Reproduce

1. Start a single-validator network, whose genesis.json has app_state.transfer.port_id set to "cosmos-transfer". Open the GRPC/RPC ports.
2. Start one peer follower, also with GRPC/RPC ports open.
3. Create an IBC channel to the IBC "cosmos-transfer" port via the peer's RPC/GRPC ports. Note that it opens successfully and the peer continues to process blocks. I used: hermes create channel cosmoshub-testnet ibcdbg-1 --port-a transfer --port-b cosmos-transfer -o unordered -v ics20-1
4. Restart the peer.
5. Repeat step 3 (create an IBC channel), note that the channel cannot create successfully, failing with: failed to execute message; message index: 1: could not retrieve module from port-id: ports/cosmos-transfer: capability not found
6. Repeat step 3 but this time use the validator's RPC/GRPC ports. The channel opens successfully.
   However, note that the peer has died with: Dec 09 02:26:19 69fe6a35438c ag-chain-cosmos[2029]: 2:26AM ERR CONSENSUS FAILURE!!! err="+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected FD7D...

I've posted my exported genesis.json which was identical at the crashed block height for both the working validator and the divergent peer. It is interesting to note that the app_state.capability section in it appears to have all the correct entries.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[michaelfig]

I've tracked down the AppHash crashes on the Agoric devnet (Agoric/agoric-sdk#4159) to this issue.

FYI: @adizere, @dtribble, @JimLarson, @dckc

[michaelfig]

I knew it was going to be something silly. I was missing capabilitytypes.ModuleName from SetOrderBeginBlockers. Yikes!

diff --git a/golang/cosmos/app/app.go b/golang/cosmos/app/app.go
index f6115772f..1a27813f6 100644
--- a/golang/cosmos/app/app.go
+++ b/golang/cosmos/app/app.go
@@ -469,10 +469,23 @@ func NewAgoricApp(
 	// CanWithdrawInvariant invariant.
 	// NOTE: staking module is required if HistoricalEntries param > 0
 	app.mm.SetOrderBeginBlockers(
-		upgradetypes.ModuleName, minttypes.ModuleName, distrtypes.ModuleName, slashingtypes.ModuleName,
-		evidencetypes.ModuleName, stakingtypes.ModuleName, ibchost.ModuleName, swingset.ModuleName,
+		upgradetypes.ModuleName,
+		capabilitytypes.ModuleName,
+		minttypes.ModuleName,
+		distrtypes.ModuleName,
+		slashingtypes.ModuleName,
+		evidencetypes.ModuleName,
+		stakingtypes.ModuleName,
+		ibchost.ModuleName,
+		swingset.ModuleName,
+	)
+	app.mm.SetOrderEndBlockers(
+		vbank.ModuleName,
+		swingset.ModuleName,
+		crisistypes.ModuleName,
+		govtypes.ModuleName,
+		stakingtypes.ModuleName,
 	)
-	app.mm.SetOrderEndBlockers(vbank.ModuleName, swingset.ModuleName, crisistypes.ModuleName, govtypes.ModuleName, stakingtypes.ModuleName)
 
 	// NOTE: The genutils module must occur after staking so that pools are
 	// properly initialized with tokens from genesis accounts.

[amaury1093]

Maybe a good thing to do is to panic a startup if the SetOrder* functions forgot any module.

[robert-zaremba]

Good idea @AmauryM

[amaury1093]

PR in #10711

[michaelfig]

@colin-axner here is the bug that was discussed in the IBC meeting.