Improve debuggability of chain halts

[facundomedica]

Summary

When a chain halts we need to get as much information as possible and as easy as possible.

Problem Definition

Currently, the usual way to debug a chain halt is not that easy. We should provide tools and guides on how to debug and pinpoint the root cause of the most common failures.

Proposal

Provide tools and guides for solving:

• wrong Block.Header.LastResultsHash.
• wrong Block.Header.ConsensusHash.
• wrong Block.Header.AppHash.

I think those are the most common errors for chain halts (besides panics).

Some stuff that comes to mind:

• Command to obtain a specific block (inputs and results) along with hashes, txs, etc. (even when the chain is halted)
• Dump to a file info that might be helpful
• Improved iavl-viewer + detailed how-to (maybe iterate over this https://github.com/cosmos/iavl/tree/master/cmd/iaviewer)
• Remote debugging tools (like running iavl-viewer over http instead of having to pull the entire state?) so validators can give read-only access to chain devs if they desire to do so.

[alexanderbez]

Idea I have for debugging LastResultsHash:

Take two data directories and inspect the last block. For both blocks (from both data sets), compare all fields, especially gas used.

For the other two types, it's a bit more difficult. Perhaps we create a tool that encapsulates all of these tools into a single tool.

[julienrbrt]

For the IAVL Viewer, we've already created a tracking issue here: cosmos/iavl#567 as it was as well requested in our DEV UX calls.

[yihuang]

About app hash mismatch, one idea is save the the change stream of last few blocks(based on adr-038), compare with good node, can pinpoint which tx caused the mismatch.

[alexanderbez]

What I was thinking is that we have a tool/binary that is provided two data directories and returns to you debugging output for each of the three types of mismatches.

• For LastResultsHash it inspects the gas used between the two states and reports a diff if any
• For ConsensusHash it inspects the entire structures between the two states and reports a diff if any
• For AppHash it reports the module hashes of each module between the two states and execution order of txs and begin/end block

This is to give the developer or operator a high-level overview for where to start looking. In the end, further in depth analysis will still be required, especially in the case of AppHash mismatch.

[eliasnaur]

@alexanderbez I'm looking into this. Do you have a way to reproduce or fetch two (or more) representative data directories, to demonstrate the usefulness of the tool?

[elias-orijtech]

The initial release of the chdbg tool can now help diagnose ConsensusHash differences:

go run github.com/orijtech/chdbg bns-a.db bns-b.db
chdbg: hash mismatch: 96AAD58DBDF2BA87D90BE1F620E80AC3D1662B5113A7667B51303596163A5969 != 56E581EBD9C0A3D726A91579839F7FF8A9251BEB063FDF0FA0415A0B3429DF6E
chdbg: key _i.bchnft_owner:4C97A7423B1782D7C8CAB362247B848DEC96B1EC: key proofs differ
chdbg: key _i.bchnft_owner:E28AE9A6EB94FC88B73EB7CBD6B87BF93EB9BEF0: key proofs differ
chdbg: key _i.tkrnft_owner:E28AE9A6EB94FC88B73EB7CBD6B87BF93EB9BEF0: key proofs differ
chdbg: key _i.usrnft_chainaddr:1152542575310734325L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:12256717727036376470L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:14285752342776807606L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:177168082075485743L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:2980033962229439650L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:3070406526139113375L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdbg: key _i.usrnft_chainaddr:8565302995323734695L;da3ed6a45429278bac2666961289ca17ad86595d33b31037615d4b8e8f158bba: key proofs differ
chdgb: ... (additional diffs omitted)
chdbg: database mismatch at version 190258 with 88 differences
exit status 2

I'm looking forward to seeing examples of LastResultsHash or AppHash mismatches to develop the tool further.

[tac0turtle]

@elias-orijtech are you able to open a pr adding the tool to the tools directory.

[elias-orijtech]

I can, but I suggest keeping it out of tools until we're happy with its UX and feature set. Personally, I'd like to see more chain halts that I can analyze and use to refine the tool.

[tac0turtle]

Would love others to chime in, but personally I think adding it to sdk or a tools repo so users can see it and analyse it will get you feedback. Right now it's hard to get feedback when users don't know about it.

The first feedback is this is only for iavl but the issue talks about three issues, lastresulthash, consensus hash and app hash. The tool seems to only work with app hash issues but it's unclear how a user will identify which module the issue comes from. It nots hard to recreate chain halts and test that way too.

[elias-orijtech]

Then perhaps a good time to put this into wider use is when the tool can debug all three types of chain halts? That is, when this issue can be closed in favor of specific issues in the tool.

If you have the time, please sketch a way to achieve realistic chain halts of the 3 interesting variants mentioned.

[ValarDragon]

• LastResultsHash: Just increment gas in some message, and attempt to replay on same network
• AppHash: use random to change the value you write to a store

I don;t think I've ever seen consensus hash issues