Merge 0.39.0 RC0 to 0.39.0

CHANGELOG.md

@@ -35,6 +35,60 @@ Ref: https://keepachangelog.com/en/1.0.0/
 
 # Changelog
 
+## [v0.39.0] - TBD
+
+### Improvements
+
+The pruning changes introduced in 0.38.0 has bugs that may cause data loss, even after upgrading
+to 0.38.5. When upgrading from 0.38.x it is important to follow the instructions below, to prevent
+data loss and database corruption.
+
+Do not modify pruning settings with <=0.38.4 as that may cause data corruption - the following
+assumes pruning settings have not been modified since the node started using 0.38.x. The default
+pruning setting `syncable` used `KeepEvery:100`.
+
+* If using `KeepEvery:1` (pruning settings `nothing` or `everything`), upgrading to 0.38.5 is safe.
+* Otherwise, halt block processing with `--halt-height` after committing a height divisible by
+  `KeepEvery` - e.g. at block 147600 with `KeepEvery:100`. The node must _never_ have processed a
+  height beyond that at any time in its past. Upgrading to 0.38.5 is then safe.
+* Otherwise, set the 0.38.5 `KeepEvery` setting to the same as the previous `KeepEvery` setting
+  (<=0.38.4 and 0.38.5 both default to `KeepEvery:100`). Upgrading to 0.38.5 is then safe as long
+  as you wait one `KeepEvery` interval plus one `KeepRecent` interval plus one pruning `Interval`
+  before changing pruning settings or deleting the last <=0.38.4 height (so wait 210 heights with
+  the default configuration).
+* Otherwise, make sure the last version persisted with <=0.38.4 is never deleted after upgrading to
+  0.38.5, as doing so may cause data loss and data corruption.
+* Otherwise, consider syncing the node from scratch with 0.38.5.
+
+* (deps) Bump Tendermint version to [v0.33.6](https://github.com/tendermint/tendermint/releases/tag/v0.33.6)
+* (deps) Bump IAVL version to [v0.14.0](https://github.com/cosmos/iavl/releases/tag/v0.14.0)
+* (client) [\#5585](https://github.com/cosmos/cosmos-sdk/pull/5585) `CLIContext` additions:
+  * Introduce `QueryABCI` that returns the full `abci.ResponseQuery` with inclusion Merkle proofs.
+  * Added `prove` flag for Merkle proof verification.
+
+### API Breaking Changes
+
+* (baseapp) [\#5837](https://github.com/cosmos/cosmos-sdk/issues/5837) Transaction simulation now returns a `SimulationResponse` which contains the `GasInfo` and `Result` from the execution.
+
+### Bug Fixes
+
+* (store) Revert IAVL pruning functionality introduced in [v0.13.0](https://github.com/cosmos/iavl/releases/tag/v0.13.0),
+where the IAVL no longer keeps states in-memory in which it flushes periodically. IAVL now commits and
+flushes every state to disk as it did pre-v0.13.0. The SDK's multi-store will track and ensure the proper
+heights are pruned. The operator can set the pruning options via a `pruning` config via the CLI or
+through `app.toml`. The `pruning` flag exposes `default|everything|nothing|custom` as options --
+see docs for further details. If the operator chooses `custom`, they may provide granular pruning
+options `pruning-keep-recent`, `pruning-keep-every`, and `pruning-interval`. The former two options
+dictate how many recent versions are kept on disk and the offset of what versions are kept after that
+respectively, and the latter defines the height interval in which versions are deleted in a batch.
+**Note, there are some client-facing API breaking changes with regard to IAVL, stores, and pruning settings.**
+* (x/distribution) [\#6210](https://github.com/cosmos/cosmos-sdk/pull/6210) Register `MsgFundCommunityPool` in distribution amino codec.
+* (types) [\#5741](https://github.com/cosmos/cosmos-sdk/issues/5741) Prevent `ChainAnteDecorators()` from panicking when empty `AnteDecorator` slice is supplied.
+* (baseapp) [\#6306](https://github.com/cosmos/cosmos-sdk/issues/6306) Prevent events emitted by the antehandler from being persisted between transactions.
+* (client/keys) [\#5091](https://github.com/cosmos/cosmos-sdk/issues/5091) `keys parse` does not honor client app's configuration.
+* (x/bank) [\#6674](https://github.com/cosmos/cosmos-sdk/pull/6674) Create account if recipient does not exist on handing `MsgMultiSend`.
+* (x/auth) [\#6287](https://github.com/cosmos/cosmos-sdk/pull/6287) Fix nonce stuck when sending multiple transactions from an account in a same block.
+
 ## [v0.38.5] - 2020-07-02
 
 ### Improvements
@@ -311,7 +365,6 @@ to detail this new feature and how state transitions occur.
   now exists a single `Params` type with a getter and setter along with a getter for each individual parameter.
 
 ### Bug Fixes
-
 * (rest) [\#5508](https://github.com/cosmos/cosmos-sdk/pull/5508) Fix `x/distribution` endpoints to properly return height in the response.
 * (x/genutil) [\#5499](https://github.com/cosmos/cosmos-sdk/pull/) Ensure `DefaultGenesis` returns valid and non-nil default genesis state.
 * (client) [\#5303](https://github.com/cosmos/cosmos-sdk/issues/5303) Fix ignored error in tx generate only mode.

Makefile

@@ -35,6 +35,7 @@ update-swagger-docs: statik
 
 mocks: $(MOCKS_DIR)
 	mockgen -source=x/auth/types/account_retriever.go -package mocks -destination tests/mocks/account_retriever.go
+	mockgen -source=types/handler.go -package mocks -destination tests/mocks/types_handler.go
 .PHONY: mocks
 
 $(MOCKS_DIR):

NEWS.md

@@ -0,0 +1,111 @@
+# Cosmos-SDK v0.39.0 Release Notes
+
+This is the inaugural release of the **Cosmos SDK 0.39 «Launchpad»** release series.
+
+See the [Cosmos SDK 0.39.0 milestone](https://github.com/cosmos/cosmos-sdk/milestone/27?closed=1) on our issue tracker for details.
+
+## Changes to IAVL and store pruning
+
+The pruning features introduced in the `0.38` release series are buggy and might lead to data loss,
+even after upgrading to `v0.39.0`. When upgrading from `0.38` it is important to follow the instructions
+below, to prevent data loss and database corruption.
+
+**Note: there are are several breaking changes with regard to IAVL, stores, and pruning settings that affect command line clients, server configuration, and Golang API.**
+
+### Migrate an application from 0.38.5 to 0.39.0
+
+The IAVL's `v0.13.0` release introduced a pruning functionality that turned out to be buggy and flawed.
+IAVL's new `v0.14.0` release now commits and flushes every state to disk as it did in pre-`v0.13.0` release.
+The SDK's multi-store will track and ensure the proper heights are pruned. The operator can now set the pruning
+options by passing a `pruning` configuration via command line option or `app.toml`. The `pruning` flag supports the following
+options: `default`, `everything`, `nothing`, `custom` - see docs for further details. If the operator chooses `custom`, they
+may want to provide either of the granular pruning values:
+- `pruning-keep-recent`
+- `pruning-keep-every`
+- `pruning-interval`
+
+The former two options dictate how many recent versions are kept on disk and the offset of what versions are kept after that
+respectively, and the latter defines the height interval in which versions are deleted in a batch. **Note: there are are some
+client application breaking changes with regard to IAVL, stores, and pruning settings.** An example patch follows:
+
+```patch
+From 5884171ba73c3054e98564c39adc9cbbab8d4646 Mon Sep 17 00:00:00 2001
+From: Alessio Treglia <alessio@tendermint.com>
+Date: Tue, 14 Jul 2020 14:54:19 +0100
+Subject: [PATCH 2/4] use new pruning options
+
+---
+ cmd/cnd/main.go | 8 ++++++--
+ 1 file changed, 6 insertions(+), 2 deletions(-)
+
+diff --git a/cmd/cnd/main.go b/cmd/cnd/main.go
+index b0c86f4a..4a3a8518 100644
+--- a/cmd/cnd/main.go
++++ b/cmd/cnd/main.go
+@@ -23,7 +23,6 @@ import (
+ 	comgenutilcli "github.com/commercionetwork/commercionetwork/x/genutil/client/cli"
+ 	"github.com/cosmos/cosmos-sdk/baseapp"
+ 	"github.com/cosmos/cosmos-sdk/server"
+-	"github.com/cosmos/cosmos-sdk/store"
+ 	genutilcli "github.com/cosmos/cosmos-sdk/x/genutil/client/cli"
+ 	"github.com/cosmos/cosmos-sdk/x/staking"
+ )
+@@ -87,9 +86,14 @@ func main() {
+ }
+
+ func newApp(logger log.Logger, db dbm.DB, traceStore io.Writer) abci.Application {
++	pruningOpts, err := server.GetPruningOptionsFromFlags()
++	if err != nil {
++		panic(err)
++	}
++
+ 	return app.NewCommercioNetworkApp(
+ 		logger, db, traceStore, true, invCheckPeriod,
+-		baseapp.SetPruning(store.NewPruningOptionsFromString(viper.GetString("pruning"))),
++		baseapp.SetPruning(pruningOpts),
+ 		baseapp.SetMinGasPrices(viper.GetString(server.FlagMinGasPrices)),
+ 		baseapp.SetHaltHeight(uint64(viper.GetInt(server.FlagHaltHeight))),
+ 	)
+```
+
+### Migrate a node from 0.38.5 to 0.39.0
+
+Note: **do not modify pruning settings with any release prior to `v0.39.0` as that may cause data corruption**.
+The following instructions assume that **pruning settings have not been modified since the node started using 0.38.x. Note: the default pruning setting `syncable` used `KeepEvery:100`.
+
+* The simple upgrade strategy: **perform a full sync of the node from scratch**. Else, follow one of the other strategies.
+
+* If your node had started with using `KeepEvery:1` (e.g. pruning settings `nothing` or `everything`), upgrading to `v0.39.0` is safe.
+
+* Otherwise, do halt block processing with `--halt-height` after committing a height divisible by `KeepEvery` - e.g. at block 147600 with `KeepEvery:100`. The **node must never have processed a height beyond that at any time in its past**. Upgrading to `v0.39.0` is then safe.
+
+* Otherwise, set the `KeepEvery` setting to the same as the previous `KeepEvery` setting (both `<=v0.38.5` and `v0.39.0` default to `KeepEvery:100`). Upgrade to `v0.39.0` is then safe as long as you wait one `KeepEvery` interval plus one `KeepRecent` interval **plus** one pruning `Interval` before changing pruning settings or deleting the last `<=v0.38.5` height (so wait *210* heights with the default configuration).
+
+* Otherwise, make sure the last version persisted with `<=v0.38.5` is never deleted after upgrading to `v0.39.0`, as doing so may cause data loss and data corruption.
+
+## Regression in the signature verification when multiple transactions in the same block are sent from the same account
+
+When multiple transactions in the same block are sent (and correctly signed) by the same account, chances are that some of them could be rejected and the error `unauthorized: signature verification failed` would be returned due to the account's sequence (*nonce*) getting stuck and not being incremented by the ante handler. This behaviour was [a regression](https://github.com/cosmos/cosmos-sdk/issues/6287) introduced in the `v0.38` release series, it did not occur in the `v0.37` release series and is now fixed in this release.
+
+## Changes to ABCI Query's "app/simulate" path
+
+The `app/simulate` query path is used to simulate the execution transactions in order to obtain an estimate
+of the gas consumption that would be required to actually execute them. The response used to return only
+the amount of gas, it now returns the result of the transaction as well.
+
+## bank.send event comes with sender information
+
+The `bank.send` event used to carry only the recipient and amount. It was assumed that the sender of the funds was `message.sender`.
+This is often not true when a module call the bank keeper directly. This may be due to staking distribution, or via a cosmwasm contract that released funds (where I discovered the issue).
+
+`bank.send` now contains the entire triple `(sender, recipient, amount)`.
+
+## trace option is no longer ignored
+
+The `--trace` option is reintroduced. It comes in very handy for debugging as it causes the full stack trace to be included in the ABCI error logs.
+
+## appcli keys parse command didn't honor client application's bech32 prefixes
+
+The `key parse` command ignored the application-specific address bech32
+prefixes and used to return `cosmos*1`-prefixed addresses regardless
+of the client application's configuration.

baseapp/abci.go

@@ -163,7 +163,7 @@ func (app *BaseApp) EndBlock(req abci.RequestEndBlock) (res abci.ResponseEndBloc
 func (app *BaseApp) CheckTx(req abci.RequestCheckTx) abci.ResponseCheckTx {
 	tx, err := app.txDecoder(req.Tx)
 	if err != nil {
-		return sdkerrors.ResponseCheckTx(err, 0, 0)
+		return sdkerrors.ResponseCheckTx(err, 0, 0, app.trace)
 	}
 
 	var mode runTxMode
@@ -181,7 +181,7 @@ func (app *BaseApp) CheckTx(req abci.RequestCheckTx) abci.ResponseCheckTx {
 
 	gInfo, result, err := app.runTx(mode, req.Tx, tx)
 	if err != nil {
-		return sdkerrors.ResponseCheckTx(err, gInfo.GasWanted, gInfo.GasUsed)
+		return sdkerrors.ResponseCheckTx(err, gInfo.GasWanted, gInfo.GasUsed, app.trace)
 	}
 
 	return abci.ResponseCheckTx{
@@ -201,12 +201,12 @@ func (app *BaseApp) CheckTx(req abci.RequestCheckTx) abci.ResponseCheckTx {
 func (app *BaseApp) DeliverTx(req abci.RequestDeliverTx) abci.ResponseDeliverTx {
 	tx, err := app.txDecoder(req.Tx)
 	if err != nil {
-		return sdkerrors.ResponseDeliverTx(err, 0, 0)
+		return sdkerrors.ResponseDeliverTx(err, 0, 0, app.trace)
 	}
 
 	gInfo, result, err := app.runTx(runTxModeDeliver, req.Tx, tx)
 	if err != nil {
-		return sdkerrors.ResponseDeliverTx(err, gInfo.GasWanted, gInfo.GasUsed)
+		return sdkerrors.ResponseDeliverTx(err, gInfo.GasWanted, gInfo.GasUsed, app.trace)
 	}
 
 	return abci.ResponseDeliverTx{
@@ -326,12 +326,20 @@ func handleQueryApp(app *BaseApp, path []string, req abci.RequestQuery) abci.Res
 				return sdkerrors.QueryResult(sdkerrors.Wrap(err, "failed to decode tx"))
 			}
 
-			gInfo, _, _ := app.Simulate(txBytes, tx)
+			gInfo, res, err := app.Simulate(txBytes, tx)
+			if err != nil {
+				return sdkerrors.QueryResult(sdkerrors.Wrap(err, "failed to simulate tx"))
+			}
+
+			simRes := sdk.SimulationResponse{
+				GasInfo: gInfo,
+				Result:  res,
+			}
 
 			return abci.ResponseQuery{
 				Codespace: sdkerrors.RootCodespace,
 				Height:    req.Height,
-				Value:     codec.Cdc.MustMarshalBinaryLengthPrefixed(gInfo.GasUsed),
+				Value:     codec.Cdc.MustMarshalBinaryBare(simRes),
 			}
 
 		case "version":

baseapp/baseapp.go

@@ -106,6 +106,9 @@ type BaseApp struct { // nolint: maligned
 
 	// application's version string
 	appVersion string
+
+	// trace set will return full stack traces for errors in ABCI Log field
+	trace bool
 }
 
 // NewBaseApp returns a reference to an initialized BaseApp. It accepts a
@@ -127,6 +130,7 @@ func NewBaseApp(
 		queryRouter:    NewQueryRouter(),
 		txDecoder:      txDecoder,
 		fauxMerkleMode: false,
+		trace:          false,
 	}
 	for _, option := range options {
 		option(app)
@@ -354,6 +358,10 @@ func (app *BaseApp) setInterBlockCache(cache sdk.MultiStorePersistentCache) {
 	app.interBlockCache = cache
 }
 
+func (app *BaseApp) setTrace(trace bool) {
+	app.trace = trace
+}
+
 // Router returns the router of the BaseApp.
 func (app *BaseApp) Router() sdk.Router {
 	if app.sealed {
@@ -596,8 +604,9 @@ func (app *BaseApp) runTx(mode runTxMode, txBytes []byte, tx sdk.Tx) (gInfo sdk.
 		// writes do not happen if aborted/failed.  This may have some
 		// performance benefits, but it'll be more difficult to get right.
 		anteCtx, msCache = app.cacheTxContext(ctx, txBytes)
-
+		anteCtx = anteCtx.WithEventManager(sdk.NewEventManager())
 		newCtx, err := app.anteHandler(anteCtx, tx, mode == runTxModeSimulate)
+
 		if !newCtx.IsZero() {
 			// At this point, newCtx.MultiStore() is cache-wrapped, or something else
 			// replaced by the AnteHandler. We want the original multistore, not one