CLI (query staking validator) failed with empty expected root hash

[whunmr]

Summary of Bug

CLI gaiacli query staking validator failed:

$ gaiacli query staking validator  --chain-id=testing cosmosvaloper12e8kaetaudaqeql6wukpu7tcsh37la2cp4knfp
ERROR: failed to prove merkle proof: Calculated root hash is invalid: expected [] but got [229 102 78 0 167 147 2 81 154 129 139 136 88 125 188 107 41 46 168 17 25 43 65 130 233 181 57 175 198 253 232 28]

Version

gaia v1.0.0-rc3

Steps to Reproduce

• setup single node validator

$ gaiad init --chain-id=testing testing
$ gaiacli keys add validator
$ gaiad add-genesis-account $(gaiacli keys show validator -a) 200000000stake,900000000validatortoken
$ gaiad gentx --name validator
$ gaiad collect-gentxs
$ gaiad start

• execute query cmd gaiacli query staking validator

$ gaiacli keys show validator --bech=val
- name: validator
  type: local
  address: cosmosvaloper12e8kaetaudaqeql6wukpu7tcsh37la2cp4knfp

$ gaiacli query staking validator  --chain-id=testing cosmosvaloper12e8kaetaudaqeql6wukpu7tcsh37la2cp4knfp
ERROR: failed to prove merkle proof: Calculated root hash is invalid: expected [] but got [229 102 78 0 167 147 2 81 154 129 139 136 88 125 188 107 41 46 168 17 25 43 65 130 233 181 57 175 198 253 232 28]

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[whunmr]

(dlv) c
> github.com/tendermint/tendermint/crypto/merkle.ProofOperators.Verify() ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/crypto/merkle/proof.go:61 (hits goroutine(1):1 total:1) (PC: 0xbfc34a)
Current event: 4784
    56:			if err != nil {
    57:				return
    58:			}
    59:		}
    60:		if !bytes.Equal(root, args[0]) {
=>  61:			return cmn.NewError("Calculated root hash is invalid: expected %+v but got %+v", root, args[0])
    62:		}
    63:		if len(keys) != 0 {
    64:			return cmn.NewError("Keypath not consumed all")
    65:		}
    66:		return nil
(dlv) bt
 0  0x0000000000bfc34a in github.com/tendermint/tendermint/crypto/merkle.ProofOperators.Verify
    at ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/crypto/merkle/proof.go:61
 1  0x0000000000bfd6bb in github.com/tendermint/tendermint/crypto/merkle.(*ProofRuntime).Verify
    at ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/crypto/merkle/proof.go:127
 2  0x0000000000bfd300 in github.com/tendermint/tendermint/crypto/merkle.(*ProofRuntime).VerifyValue
    at ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/crypto/merkle/proof.go:113
 3  0x0000000001032d4a in github.com/cosmos/cosmos-sdk/client/context.CLIContext.verifyProof
    at ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/client/context/query.go:157
 4  0x0000000001032538 in github.com/cosmos/cosmos-sdk/client/context.CLIContext.query
    at ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/client/context/query.go:104
 5  0x0000000001033201 in github.com/cosmos/cosmos-sdk/client/context.CLIContext.queryStore
    at ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/client/context/query.go:170
 6  0x0000000001031cf0 in github.com/cosmos/cosmos-sdk/client/context.CLIContext.QueryStore
    at ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/client/context/query.go:49
 7  0x00000000010d5c9e in github.com/cosmos/cosmos-sdk/x/staking/client/cli.GetCmdQueryValidator.func1
    at ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/x/staking/client/cli/query.go:68
 8  0x00000000008af8f1 in github.com/spf13/cobra.(*Command).execute
    at ./lab/go_linux/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
 9  0x00000000008b0547 in github.com/spf13/cobra.(*Command).ExecuteC
    at ./lab/go_linux/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
10  0x00000000008afe6b in github.com/spf13/cobra.(*Command).Execute
    at ./lab/go_linux/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
11  0x0000000000a13bd0 in github.com/tendermint/tendermint/libs/cli.Executor.Execute
    at ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/libs/cli/setup.go:89
12  0x00000000011b8664 in main.main
    at /media/psf/sf/lab/dex/cmd/cetcli/main.go:46
13  0x000000000043a12f in runtime.main
    at /usr/local/go/src/runtime/proc.go:200
14  0x000000000046a901 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1337
(dlv) frame 3
> github.com/tendermint/tendermint/crypto/merkle.ProofOperators.Verify() ./lab/go_linux/pkg/mod/github.com/tendermint/tendermint@v0.32.2/crypto/merkle/proof.go:61 (hits goroutine(1):1 total:1) (PC: 0xbfc34a)
Current event: 4784
Frame 3: ./lab/go_linux/pkg/mod/github.com/cosmos/cosmos-sdk@v0.36.0-rc4/client/context/query.go:157 (PC: 1032d4a)
   152:			if err != nil {
   153:				return errors.Wrap(err, "failed to prove merkle proof")
   154:			}
   155:			return nil
   156:		}
=> 157:		err = prt.VerifyValue(resp.Proof, commit.Header.AppHash, kp.String(), resp.Value)
   158:		if err != nil {
   159:			return errors.Wrap(err, "failed to prove merkle proof")
   160:		}
   161:
   162:		return nil
(dlv) p commit.Header.AppHash
github.com/tendermint/tendermint/libs/common.HexBytes len: 0, cap: 0, []

[alexanderbez]

You have to run with the --trust-node option. Unfortunately, I don't know why the proofs are failing.

Any further insight here @zmanian @jackzampolin ?

[alexanderbez]

I will dig into this and make sure nothing is broken that isn't expected to be (as I know lite client work is still somewhat a WIP).

[alexanderbez]

In the Tendermint/Lite Client rabbit hole. Some observations thus far:

1. q staking validators is a subspace query; subspace queries do not require validation as the proof is nil in any case.
2. q staking validator <address> is a key query. The proof is not nil as expected, however when we get the commit via ctx.Verify(resp.Height + 1), the commit.Header.AppHash is empty!

---

EDIT 2: Root of the problem seems to stem from GetCertifiedCommit. The ResultCommit's AppHash is empty, but the signed block header does exist and is cached:

SignedHeader{
  Header{
    Version:        {10 0}
    ChainID:        chain-WtIMEu
    Height:         1
    Time:           2019-08-12 17:16:49.5350195 +0000 UTC
    NumTxs:         0
    TotalTxs:       0
    LastBlockID:    :0:000000000000
    LastCommit:
    Data:
    Validators:     07FE11CA64BC47ECB82686C216A4588EFBD264EBC141E18872E508FD871AAC35
    NextValidators: 07FE11CA64BC47ECB82686C216A4588EFBD264EBC141E18872E508FD871AAC35
    App:
    Consensus:       048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F
    Results:
    Evidence:
    Proposer:       447673D738A82F202744B6D9778B7D60551F75F3
  }#479A47F454D3B034322D458BC3458DB1B9CA1684110B4C136E3242B6D7F07981
  Commit{
    BlockID:    479A47F454D3B034322D458BC3458DB1B9CA1684110B4C136E3242B6D7F07981:1:2656B6BF2B5D
    Precommits:
      Vote{0:447673D738A8 1/00/2(Precommit) 479A47F454D3 D4E7530F4408 @ 2019-08-12T17:17:02.166148Z}
      Vote{1:521FAA8D985E 1/00/2(Precommit) 479A47F454D3 C0823557DAEE @ 2019-08-12T17:17:02.1738255Z}
      Vote{2:54A9E565109C 1/00/2(Precommit) 479A47F454D3 7C82911161C4 @ 2019-08-12T17:17:02.1209463Z}
      Vote{3:86BC65EFA081 1/00/2(Precommit) 479A47F454D3 A4BCE9622344 @ 2019-08-12T17:17:02.1174948Z}
  }#
}

I think this makes sense because it's getting the signed header at height: 0 + 1 = 1 at which point there is no app hash yet.

---

EDIT 3: Using ctx.Verify(resp.Height + 2) instead, we get an app hash.

---

TL;DR:

• I do not believe we can provide and verify proofs for signed headers at height <= 1.
  • Verified this is working when providing a --height=N where N > 1 AND N is not pruned.
• We should provide a friendly error message when querying with a proof for which the height has been pruned from the node.

[zmanian]

This makes sense!