Supply by denom Migrations #8780

amaury1093 · 2021-03-04T11:11:56Z

Description

follow-up from #8517
closes: #7092

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

amaury1093 · 2021-03-04T11:12:37Z

proto/cosmos/bank/v1beta1/bank.proto

+// amounts in the network.
+// This message is deprecated now that supply is indexed by denom.
+message Supply {
+  option deprecated = true;


ref: #8517 (comment)

Is it documented somewhere how we want to deal with changes in proto. As I understand it, we can make changes as it's still labellled beta. wdyt?

Yes, https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-023-protobuf-naming.md. From what I understand, if buf is shouting, then something's wrong and the pkg version needs to be bumped.

We need to require proto-breaking CI checks...

x/bank/spec/01_state.md

…dk into am/7092-migrations

…092-migrations

codecov · 2021-03-04T13:29:03Z

Codecov Report

Merging #8780 (1613e52) into master (e17953a) will increase coverage by 0.00%.
The diff coverage is 68.75%.

@@           Coverage Diff           @@
##           master    #8780   +/-   ##
=======================================
  Coverage   59.36%   59.37%           
=======================================
  Files         563      563           
  Lines       31257    31270   +13     
=======================================
+ Hits        18557    18566    +9     
- Misses      10544    10546    +2     
- Partials     2156     2158    +2

Impacted Files	Coverage Δ
x/bank/keeper/migrations.go	`50.00% <0.00%> (ø)`
x/bank/keeper/querier.go	`74.13% <ø> (ø)`
x/bank/legacy/v042/store.go	`82.60% <73.33%> (-17.40%)`	⬇️

aaronc

LGTM

x/bank/spec/01_state.md

jgimeno · 2021-03-05T13:18:38Z

LGTM, only a little conflict to fix.

…092-migrations

* Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com>

* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com>

* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com> (cherry picked from commit 55fc465)

* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com> (cherry picked from commit 55fc465) Co-authored-by: Charly <charly@interchain.berlin>

amaury1093 added 2 commits March 4, 2021 11:38

Add back supply proto

d1171aa

Add migration for supply

a60f493

amaury1093 added the C:x/bank label Mar 4, 2021

amaury1093 requested review from aaronc, alessio and alexanderbez as code owners March 4, 2021 11:11

amaury1093 requested a review from sahith-narahari March 4, 2021 11:12

amaury1093 commented Mar 4, 2021

View reviewed changes

amaury1093 requested review from robert-zaremba and anilcse March 4, 2021 11:15

Fix lint

100a4ae

amaury1093 commented Mar 4, 2021

View reviewed changes

x/bank/spec/01_state.md Outdated Show resolved Hide resolved

amaury1093 and others added 5 commits March 4, 2021 12:29

Update x/bank/spec/01_state.md

10257db

Merge branch 'master' into am/7092-migrations

275f17a

Fix test

4dd1d8f

Merge branch 'am/7092-migrations' of ssh://github.com/cosmos/cosmos-s…

7d5ffee

…dk into am/7092-migrations

Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am/7…

3f7717d

…092-migrations

Proto gen

54402fd

aaronc approved these changes Mar 4, 2021

View reviewed changes

amaury1093 commented Mar 4, 2021

View reviewed changes

x/bank/spec/01_state.md Outdated Show resolved Hide resolved

Update x/bank/spec/01_state.md

495892c

sahith-narahari mentioned this pull request Mar 4, 2021

Revert unintended skip test in feegrant #8789

Merged

9 tasks

jgimeno approved these changes Mar 5, 2021

View reviewed changes

sahith-narahari approved these changes Mar 5, 2021

View reviewed changes

amaury1093 added 2 commits March 5, 2021 14:53

Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am/7…

2173d37

…092-migrations

Make proto gen

1613e52

amaury1093 added the A:automerge Automatically merge PR once all prerequisites pass. label Mar 5, 2021

mergify bot merged commit 6520997 into master Mar 5, 2021

mergify bot deleted the am/7092-migrations branch March 5, 2021 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supply by denom Migrations #8780

Supply by denom Migrations #8780

amaury1093 commented Mar 4, 2021

amaury1093 Mar 4, 2021

sahith-narahari Mar 4, 2021

amaury1093 Mar 4, 2021

aaronc Mar 4, 2021

codecov bot commented Mar 4, 2021 •

edited

Loading

aaronc left a comment

jgimeno commented Mar 5, 2021

Supply by denom Migrations #8780

Supply by denom Migrations #8780

Conversation

amaury1093 commented Mar 4, 2021

Description

amaury1093 Mar 4, 2021

Choose a reason for hiding this comment

sahith-narahari Mar 4, 2021

Choose a reason for hiding this comment

amaury1093 Mar 4, 2021

Choose a reason for hiding this comment

aaronc Mar 4, 2021

Choose a reason for hiding this comment

codecov bot commented Mar 4, 2021 • edited Loading

Codecov Report

aaronc left a comment

Choose a reason for hiding this comment

jgimeno commented Mar 5, 2021

codecov bot commented Mar 4, 2021 •

edited

Loading