-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reduce gas costs by 10x for transient store operations #8790
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8790 +/- ##
==========================================
- Coverage 59.37% 59.35% -0.02%
==========================================
Files 563 563
Lines 31270 31277 +7
==========================================
- Hits 18566 18565 -1
- Misses 10546 10554 +8
Partials 2158 2158
|
Sounds reasonable to me. Benchmarks would be nice to determine whether 10x is the right factor... |
I agree. @alexanderbez wdyt? |
@albertchon can you add a CHANGELOG entry, please? |
@alessio done! |
Awesome work!! This makes me think we should try to allow app developers to set these with a sane default? |
We could use variables instead of literals. Wdyt @albertchon? |
@alessio I have no strong opinion on this, I was just following the convention set by the default KVStore gas config. https://github.com/cosmos/cosmos-sdk/pull/8790/files#diff-c87a5008bbfa0e28dce0be2d9bad4dda7261e9ee762178fd8829993f55c1b6abL165-L174 The same could be said to use variables instead of literals there as well. |
* reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com>
* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com>
* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com> (cherry picked from commit 55fc465)
* fixed broken links, typos * Update docs/ibc/overview.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * Update docs/intro/sdk-app-architecture.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * Update docs/building-modules/simulator.md Co-authored-by: Marko <markobaricevic3778@gmail.com> * build(deps): bump JamesIves/github-pages-deploy-action from 4.0.0 to 4.1.0 (#8792) Bumps [JamesIves/github-pages-deploy-action](https://github.com/JamesIves/github-pages-deploy-action) from 4.0.0 to 4.1.0. - [Release notes](https://github.com/JamesIves/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@4.0.0...3dbacc7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix multisig account pubkeys migration (#8794) closes: #8776 * Update mergify (#8784) * Update mergify Prep for the v0.42 release series. * retire v0.41, the hub can upgrade to v0.42 smoothly * perf change (#8796) Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Supply by denom Migrations (#8780) * Add back supply proto * Add migration for supply * Fix lint * Update x/bank/spec/01_state.md * Fix test * Proto gen * Update x/bank/spec/01_state.md * Make proto gen Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> * fix make protoc error (#8799) * reduce gas costs by 10x for transient store operations (#8790) * reduce gas costs by 10x for transient store operations * fix TestTransientGasConfig for ReadCostFlat * added changelog entry * fix changelog * fix changelog Co-authored-by: Alessio Treglia <alessio@tendermint.com> * x/gov: fix NormalizeProposalType() return values (#8808) Closes: #8806 * store/cachekv: use typed types/kv.List instead of container/list.List (#8811) Reduces CPU burn by using a typed List to avoid the expensive type assertions from using an interface. This is the only option for now until Go makes generics generally available. The change brings time spent on the time assertion cummulatively to: 580ms down from 6.88s Explanation: The type assertions were showing up heavily in profiles: * Before this commit ```shell Total: 42.18s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 14.01s 18.87s (flat, cum) 44.74% of Total . . 17: items []*kv.Pair . . 18: ascending bool . . 19:} . . 20: . . 21:func newMemIterator(start, end []byte, items *list.List, ascending bool) *memIterator { . 620ms 22: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 23: . . 24: var entered bool . . 25: 510ms 870ms 26: for e := items.Front(); e != nil; e = e.Next() { 6.85s 6.88s 27: item := e.Value.(*kv.Pair) 5.71s 8.19s 28: if !dbm.IsKeyInDomain(item.Key, start, end) { 120ms 120ms 29: if entered { . . 30: break . . 31: } . . 32: . . 33: continue . . 34: } . . 35: 820ms 980ms 36: itemsInDomain = append(itemsInDomain, item) . . 37: entered = true . . 38: } . . 39: . 1.21s 40: return &memIterator{ . . 41: start: start, . . 42: end: end, . . 43: items: itemsInDomain, . . 44: ascending: ascending, . . 45: } ``` and given that the list only uses that type, it is only right to lift the code from container/list.List, and only modify Element.Value's type. For emphasis, the code is basically just a retrofit of container/list/list.go but with a typing, and we'll keep it as is until perhaps Go1.17 or Go1.18 or when everyone uses Go1.17+ after generics have landed. * After this commit ```shell Total: 45.25s ROUTINE ======================== github.com/cosmos/cosmos-sdk/store/cachekv.newMemIterator in /Users/emmanuelodeke/go/src/github.com/cosmos/cosmos-sdk/store/cachekv/memiterator.go 4.84s 6.77s (flat, cum) 14.96% of Total . . 16: items []*kv.Pair . . 17: ascending bool . . 18:} . . 19: . . 20:func newMemIterator(start, end []byte, items *kv.List, ascending bool) *memIterator { . 330ms 21: itemsInDomain := make([]*kv.Pair, 0, items.Len()) . . 22: . . 23: var entered bool . . 24: 60ms 160ms 25: for e := items.Front(); e != nil; e = e.Next() { 580ms 580ms 26: item := e.Value 3.68s 4.78s 27: if !dbm.IsKeyInDomain(item.Key, start, end) { 80ms 80ms 28: if entered { . . 29: break . . 30: } . . 31: . . 32: continue . . 33: } . . 34: 440ms 580ms 35: itemsInDomain = append(itemsInDomain, item) . . 36: entered = true . . 37: } . . 38: . 260ms 39: return &memIterator{ . . 40: start: start, . . 41: end: end, . . 42: items: itemsInDomain, . . 43: ascending: ascending, . . 44: } ``` Fixes #8810 * Move all migration scripts to v043 (#8814) * Move all migration scripts to v043 * Fix permaling * Fix test * Fix test again Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * permalinks Co-authored-by: chrly <chrly@chrlys-MacBook-Pro.local> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Jonathan Gimeno <jgimeno@gmail.com> Co-authored-by: Marko <markobaricevic3778@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akhil Kumar P <36399231+akhilkumarpilli@users.noreply.github.com> Co-authored-by: Alessio Treglia <alessio@tendermint.com> Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Albert Chon <albert@injectiveprotocol.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com> (cherry picked from commit 55fc465) Co-authored-by: Charly <charly@interchain.berlin>
Glad this got done! |
Description: Reduce Transient Store Gas costs by 10x
Recall that a transient store is a base-layer KVStore which is automatically discarded at the end of the block and is implemented using an in-memory DB.
Currently, the transient store uses the same gas configuration as normal KVStores. This wrongfully discourages usage of the transient store, which can be a useful for intra-block data persistence (e.g. store data in the handler, then access it in the Endblocker).
This PR proposes reducing the gas costs for Transient Store operations by 10x.
Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.
docs/
) or specification (x/<module>/spec/
)godoc
comments.Unreleased
section inCHANGELOG.md
Files changed
in the Github PR explorerCodecov Report
in the comment section below once CI passes