In the spirit of pragmatic solutions to provide higher assurance of correctness relatively quickly, I suggest we consider adding a --assert-invariants mode to gaiad (perhaps even enabled by default). When run in this mode, gaiad asserts all the specified invariants at the end of each block and panics if any fail.

This is easy to implement since we can just reuse the existing invariants from the simulation (and any future invariants which might be added to the simulation) and adds a reasonable degree of assurance from certain classes of bugs - if at least 1/3 of stake runs with this invariant assertion mode enabled, and an invariant breaks, the chain will safely halt and we can figure out what's going on & fix it.

This is not a long-term solution as it isn't necessarily incentive-compatible (extra compute by validators which they aren't getting paid for) - but in the short term I think enough might do it. We'd need to ensure the current invariants aren't too slow (< ~0.1s maybe) to cause problems.

cc @gamarin2 @sunnya97 @zmanian thoughts?