fix: snapshot recover from exporter error #13935

chillyvee · 2022-11-20T17:06:01Z

Description

During snapshots, it is difficult to tell which store was not able to export properly.

In the case of Dig chain, one store was not fully configured (no nodedb) so the returned exporter should be nil.

Cosmos-sdk can then decide whether to panic or skip the possibly unused store.

Added print when each store begins and completes an export.

This related update in cosmos/iavl properly returns a nil exporter on error:
https://github.com/cosmos/iavl/pull/622/files#diff-9d9ca52939a19f26358e2b5b35ca938679b381be35f4c54379df94c4fa4d1233R44

Logging for rootmulti was previous hardcoded to NewNopLogger which did NOT log.

Now developers/users can set an environment variable to turn on logging for this package:
LOG_ROOTMULTI=1 daemond start

Author Checklist

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

chillyvee · 2022-11-20T17:45:36Z

Example output which is very helpful for debugging snapshots:

12:00PM INF creating state snapshot height=4412550
12:00PM INF indexed block exents height=4412550 module=txindex
Snapshot.Export Store Key acc
Snapshot.Export Store Key Done acc
Snapshot.Export Store Key authz
Snapshot.Export Store Key Done authz
Snapshot.Export Store Key bank
Snapshot.Export Store Key Done bank
Snapshot.Export Store Key capability
Snapshot.Export Store Key Done capability
Snapshot.Export Store Key distribution
Snapshot.Export Store Key Done distribution
Snapshot.Export Store Key evidence
Snapshot.Export Store Key Done evidence
Snapshot.Export Store Key feegrant
Snapshot.Export Store Key Done feegrant
Snapshot.Export Store Key gov
Snapshot.Export Store Key Done gov
Snapshot.Export Store Key ibc
Snapshot.Export Store Key Done ibc
Snapshot.Export Store Key icahost
iavl/export newExporter failed to create because tree.ndb is nil  <-- this is helpful
Snapshot.Export Store Key icahost exporter nil
Snapshot.Export Store Key mint
Snapshot.Export Store Key Done mint
Snapshot.Export Store Key params
Snapshot.Export Store Key Done params
Snapshot.Export Store Key slashing
Snapshot.Export Store Key Done slashing
Snapshot.Export Store Key staking
Snapshot.Export Store Key Done staking
Snapshot.Export Store Key transfer
Snapshot.Export Store Key Done transfer
Snapshot.Export Store Key upgrade
Snapshot.Export Store Key Done upgrade
Snapshot.Export Store Key wasm
Snapshot.Export Store Key Done wasm
12:05PM INF completed state snapshot format=2 height=4412550

It could be argued that a program should crash when a store doesn't have a nodedb since that could indicate a deeper problem. If that is desired let me know.

The idea here is to make it clear exactly which misconfigured store is crashing the daemon rather than a panic in a location which is called many times with different conditions.

In the case of Dig Chain, snapshots without the icahost store restored fine and validator nodes are signing with the proper apphash values. So it is likely that is an unused store that is mistakenly added, or mistakenly unconfigured.

tac0turtle · 2022-11-20T21:34:31Z

store/rootmulti/store.go

@@ -757,8 +757,17 @@ func (rs *Store) Snapshot(height uint64, protoWriter protoio.Writer) error {
 	// and the following messages contain a SnapshotNode (i.e. an ExportNode). Store changes
 	// are demarcated by new SnapshotStore items.
 	for _, store := range stores {
+		fmt.Printf("Snapshot.Export Store Key %s\n", store.name)


we should use the logger and place this in debug mode.

will fix tomorrow. didn't see the logger earlier, but I see it now

store/iavl/store.go

chillyvee · 2022-11-21T13:26:32Z

Changed all fmt.Printf to rs.logger

Found out that the original logger was a NopLogger which DID NOT respect the cli --log_level

The code path does not make it obvious how to attach to the global app logger, nor does it make sense to turn on consensus debugging when trying to debug rootmulti.

Thus we allow a runtime setting of environment variable LOG_ROOTMULTI to turn on logging for this package.

Example usage:

LOG_ROOTMULTI=1 daemond start

tac0turtle · 2022-11-21T16:40:31Z

store/store.go

@@ -10,7 +12,15 @@ import (
 )

 func NewCommitMultiStore(db dbm.DB) types.CommitMultiStore {
-	return rootmulti.NewStore(db, log.NewNopLogger())
+	// Developers may enable rootmulti logging by setting env LOG_ROOTMULTI=1
+	if len(os.Getenv("LOG_ROOTMULTI")) > 0 {


I think this may be cleaner in the app.toml. But not sure, lets wait to see what others think

Which of these makes the most sense?

passing the app the NewCommitMultiStore

passing the logging level desired to NewCommitMultiStore

configuring the logger in the baseapp and supplying it to NewCommitMultiStore

Also not sure where such a setting would go in app.toml (other than root?) since store spans quite a number of functions/reactors.

Removed this change to eliminate complexity within this PR

faddat · 2022-11-22T21:02:31Z

#12228

chillyvee · 2022-11-24T22:10:52Z

Do not merge, found a bug (noted above)

store/iavl/store.go

odeke-em · 2022-11-25T23:19:43Z

store/rootmulti/store.go

 		if err != nil {
+			rs.logger.Error("Snapshot Failed", "store", store.name, "err", err)
 			return err
 		}
 		defer exporter.Close()


This code has a bug, defers don't run until the function is returning, this assumes that they'll be closed per iteration of the loop. I've mailed PR #14027

An interesting note is that exporter.Close() is called at the end of this loop.

How about we remove the defer and call expoter.Close() manually only on error conditions (of which there are only a few)

I see your rewrite in #14027, that will work also of course.

So in my PR I just put that code into a closure then kept the defer.

chillyvee · 2022-12-02T23:25:28Z

Here's an update on this PR.

Previously the effort was focused on preventing panic/nil/crashes during a snapshot.

After review of chains with the current crashing problem, we have determined that chains introducing a new store without a keeper and a call to SetStoreLoader(upgradetypes.UpgradeStoreLoader(upgradeInfo.Height will introduce the new store at version = 0.

This new store will continue to increment versions by 1 as enforced by cosmos/iavl on each block.

The crashing problem appear then when a store within rootmulti must be selected for the current block height for all stores.

However the new improperly introduced store, does not have a "version" matching the current height, and thus any effort to snapshot will fail.

For example, if your rootmulti has stores at height 1234

bank - version 1234
slashing - version 1234

Then a chain adds a module such as authz

This results in commit with stores in rootmulti like:

bank - version 1234
slashing - version 1234
authz - version 0

A proper snapshot now needs to consider the actual referenced version of each store during the commit. This PR now loads the commit, determines the actual version of each store, and places that information into the snapshot.

The snapshot protobuf format had to be modified because the restore function could no longer assume that all height match. Since IAVL checksums include the store version, nodes will AppHash error on the first modification to that store since the new checksum will include the mismatched version.

For example, all other nodes on the network 100 blocks after the upgrade will have:

bank - version 1334
slashing - version 1334
authz - version 100

But the restored node which assumes all restored stores have the same height will have

bank - version 1334
slashing - version 1334
authz - version 1334

To avoid this, we utilize the newly added Version to ensure that authz is restored to version 100 to match all other validator nodes.

This avoid the situation where a team deploying an upgrade in an imperfect function immediately penalizes all validators by breaking snapshots creation and statesync restore. The fact that a team is not perfect should not manifest as a panic especially since the the chain works otherwise.

Please note, we encountered issues using the protobuf generation functions and would appreciate if another member could run the protobuf generation and see if the results match or if additional corrections are needed.

chillyvee · 2022-12-02T23:48:24Z

The below documents the reason help is necessary to properly regenerate the snapshot protobuf pb.go files.

This error is current as of cosmos/cosmos-sdk main branch at commit 25449b5

 % make proto-format 
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./tx/textual/internal/testpb/1.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/base/store/snapshots/v1beta1/snapshot.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/base/store/v1beta1/listening.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/evidence/v1beta1/tx.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/evidence/v1beta1/evidence.proto


 % make proto-gen
Generating Protobuf files
Generating gogo proto code
cp: can't stat 'github.com/cosmos/cosmos-sdk/*': No such file or directory
make: *** [Makefile:400: proto-gen] Error 1

With a manual run inside the docker image:

/workspace/proto $ buf generate --template buf.gen.gogo.yaml ./cosmos/base/store/snapshots/v1beta1/snapshot.proto

Does not produce an output file. No errors are reported.

/workspace/proto $ buf --version
1.9.0

CHANGELOG.md

alexanderbez · 2022-12-07T16:25:03Z

store/rootmulti/store.go


 	for key, store := range rs.stores {
+		rs.logger.Debug("prune store", "key", key)


Let's add the store as a field to the logger too

Saw the suggestion to log "name', store.Name, but store.name is a private lower case variable.

store/rootmulti/store.go

alexanderbez · 2022-12-07T16:31:07Z

store/rootmulti/store.go

+			if item.Store.Version == 0 {
+				fmt.Printf("!!!!!!!!!!!!!!!!!  Restore Error, Snapshot Missing Version, Use Another Peer !!!!!!!!!!!!!!!!")
+				rs.logger.Error("Restore Error, Snapshot Missing Version, Use Another Peer", "store", item.Store.Name, "version", item.Store.Version)
+				return snapshottypes.SnapshotItem{}, sdkerrors.Wrap(sdkerrors.ErrLogic, "Snapshot Without Store Version Not Supported, Use Another Peer")


nit: Please do not capitalize or punctuate errors.

Suggestion noted. This code is no longer relevant to the scope of this PR

store/rootmulti/store.go

chillyvee · 2022-12-07T23:41:26Z

Thank you for the review @alexanderbez

Noted above is that protobuf to go code generation appears broken. Is the code generation working for your team? Would be best to have matching .proto and .go code in the final commit.

tac0turtle · 2022-12-08T11:37:27Z

The below documents the reason help is necessary to properly regenerate the snapshot protobuf pb.go files.

This error is current as of cosmos/cosmos-sdk main branch at commit 25449b5

 % make proto-format 
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./tx/textual/internal/testpb/1.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/base/store/snapshots/v1beta1/snapshot.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/base/store/v1beta1/listening.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/evidence/v1beta1/tx.proto
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clang-format -i ./proto/cosmos/evidence/v1beta1/evidence.proto


 % make proto-gen
Generating Protobuf files
Generating gogo proto code
cp: can't stat 'github.com/cosmos/cosmos-sdk/*': No such file or directory
make: *** [Makefile:400: proto-gen] Error 1

With a manual run inside the docker image:

/workspace/proto $ buf generate --template buf.gen.gogo.yaml ./cosmos/base/store/snapshots/v1beta1/snapshot.proto

Does not produce an output file. No errors are reported.

/workspace/proto $ buf --version
1.9.0

@julienrbrt any idea why this is happening, I cant reproduce it

store/rootmulti/store.go

julienrbrt · 2022-12-08T11:44:13Z

@julienrbrt any idea why this is happening, I cant reproduce it

I cannot reproduce it either. Works for me in GitHub Codespaces as well. @chillyvee can you retry in codespaces too and what is your OS?
@likhita-809 I believe this you had something similar, have you discovered why?

store/rootmulti/store.go

chillyvee · 2022-12-08T11:52:54Z

Works in github codespaces but not Ubuntu 20.04 x64

This is the OS I'm using for the x64 dev environment

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"

REPOSITORY                     TAG           IMAGE ID       CREATED        SIZE
ghcr.io/cosmos/proto-builder   0.11.2        32668103e373   4 weeks ago    1.24GB

I will delete the docker image I have and pull it again to see what happens - does not work

# docker rmi ghcr.io/cosmos/proto-builder:0.11.2
# docker pull ghcr.io/cosmos/proto-builder:0.11.2

0.11.2: Pulling from cosmos/proto-builder
213ec9aee27d: Already exists 
4583459ba037: Pull complete 
93c1e223e6f2: Pull complete 
cbef1b99e503: Pull complete 
f99825e86aac: Pull complete 
be55fa1995db: Pull complete 
b05bebbcb2df: Pull complete 
54962337969b: Pull complete 
5aff3b65794d: Pull complete 
530c7ee0eccf: Pull complete 
d68ed598193d: Pull complete 
4e4099510e6a: Pull complete 
94369a9c35c8: Pull complete 
Digest: sha256:538f334c1069f2dd544e9f9997ada75c7fb03da21ae2816d8029de172d9c2365
Status: Downloaded newer image for ghcr.io/cosmos/proto-builder:0.11.2
ghcr.io/cosmos/proto-builder:0.11.2

 % make proto-gen
Generating Protobuf files
Generating gogo proto code
cp: can't stat 'github.com/cosmos/cosmos-sdk/*': No such file or directory
make: *** [Makefile:403: proto-gen] Error 1

REPOSITORY                     TAG           IMAGE ID       CREATED        SIZE
ghcr.io/cosmos/proto-builder   0.11.2        32668103e373   4 weeks ago    1.24GB

In codespaces

 $ docker images
REPOSITORY                     TAG       IMAGE ID       CREATED       SIZE
ghcr.io/cosmos/proto-builder   0.11.2    32668103e373   4 weeks ago   1.24GB

And the codespaces output is much more verbose

make proto-gen
Generating Protobuf files
Generating gogo proto code
go: downloading github.com/cosmos/gogoproto v1.4.3
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
...

Looks like something is preventing the go build/mod process from completing properly.

Same docker image. Must just be something very strange with that particular release of Ubuntu+Docker?

Will set up alternate dev environment/use codespaces if that's all it takes to fix. Might not be worth the time to debug.

likhita-809 · 2022-12-09T08:37:18Z

@julienrbrt any idea why this is happening, I cant reproduce it

I cannot reproduce it either. Works for me in GitHub Codespaces as well. @chillyvee can you retry in codespaces too and what is your OS? @likhita-809 I believe this you had something similar, have you discovered why?

I believe it has something to do with Ubuntu+Docker release as mentioned by @chillyvee because I had tried debugging in several different ways but nothing seems to work locally, but it works fine in codespaces with the same docker image.

chillyvee · 2023-03-03T21:03:38Z

FYI got my node synced up with the latest juno testnet so working on testing now.

chillyvee · 2023-03-03T22:38:15Z

Previous branch renamed to v0466_snapshot_recover_from_exporter_error_master_preclean

Removed enhanced snapshot format related code

Kept debugging code which can assist validators with real-time monitoring.

…aster

odeke-em

Thanks for updating these changes @chillyvee. Just one more change then good to go. Please also update the base branch to the latest.

store/rootmulti/store.go

chillyvee · 2023-03-05T18:46:35Z

Thank you for the feedback. encoding/hex dependency removed.

One more item:

@tac0turtle @odeke-em - Earlier someone asked to log store.Name, but store.name is a private variable. Okay to delete the comment from this PR? (code reference here). If there was a clean way to access the private .name, please recommend it.

https://github.com/cosmos/cosmos-sdk/pull/13935/files#diff-2bc7a86169d4c2f891d7ae74b21ca4bcc5787d10114e2eee215fe1ddba5c999eR614

…aster

odeke-em

LGTM now, thank you @chillyvee!

store/rootmulti/store.go

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

…aster

julienrbrt · 2023-03-06T16:51:15Z

Given it ended up only adding logs and improving UX, we should get this in v0.47 final imho.

julienrbrt · 2023-03-06T16:51:30Z

@Mergifyio backport release/v0.47.x

mergify · 2023-03-06T16:51:34Z

backport release/v0.47.x

✅ Backports have been created

#15273 fix: snapshot recover from exporter error (backport #13935) has been created for branch release/v0.47.x

Co-authored-by: Marko <marbar3778@yahoo.com> Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com> Co-authored-by: Marko <marko@baricevic.me> Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> (cherry picked from commit 183dde8) # Conflicts: # store/rootmulti/store.go

julienrbrt · 2023-03-06T16:54:49Z

store/rootmulti/store.go

 		exporter, err := store.Export(int64(height))
+		if exporter == nil {
+			rs.logger.Error("snapshot failed; exporter is nil", "store", store.name)


Looking at store.Export this will always trigger when there is an error, and the error check below is now not so useful. Was it intended?

cosmos-sdk/store/iavl/store.go

Lines 275 to 288 in 5d559dd

func (st *Store) Export(version int64) (*iavl.Exporter, error) {

istore, err := st.GetImmutable(version)

if err != nil {

return nil, errorsmod.Wrapf(err, "iavl export failed for version %v", version)

}

tree, ok := istore.tree.(*immutableTree)

if !ok || tree == nil {

return nil, fmt.Errorf("iavl export failed: unable to fetch tree for version %v", version)

}

export, err := tree.Export()

if err != nil {

return nil, err

}

return export, nil

we modified the exporter function after the fact and probably never cleaned this up

Co-authored-by: Chill Validation <92176880+chillyvee@users.noreply.github.com> Co-authored-by: marbar3778 <marbar3778@yahoo.com> Co-authored-by: Julien Robert <julien@rbrt.fr>

chillyvee requested a review from a team as a code owner November 20, 2022 17:06

github-actions bot added the C:Store label Nov 20, 2022

chillyvee changed the title ~~fix: snapshot recover from exporter error master~~ fix: snapshot recover from exporter error Nov 20, 2022

chillyvee mentioned this pull request Nov 20, 2022

CV add snapshot manager to dig app notional-labs/dig#425

Merged

tac0turtle reviewed Nov 20, 2022

View reviewed changes

store/iavl/store.go Outdated Show resolved Hide resolved

tac0turtle assigned tac0turtle and atheeshp Nov 21, 2022

tac0turtle reviewed Nov 21, 2022

View reviewed changes

This was referenced Nov 22, 2022

uneven store heights prevent pruning and state sync #13933

Closed

initial set of bumps akash-network/node#1719

Closed

odeke-em reviewed Nov 26, 2022

View reviewed changes

chillyvee force-pushed the v0466_snapshot_recover_from_exporter_error_master branch from ce26122 to fd0e4a7 Compare December 2, 2022 23:14

chillyvee force-pushed the v0466_snapshot_recover_from_exporter_error_master branch from fd0e4a7 to 626a01d Compare December 2, 2022 23:36

alexanderbez suggested changes Dec 7, 2022

View reviewed changes

github-advanced-security bot found potential problems Dec 7, 2022

View reviewed changes

store/rootmulti/store.go Fixed Show fixed Hide fixed

store/rootmulti/store.go Fixed Show fixed Hide fixed

store/rootmulti/store.go Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Dec 7, 2022

View reviewed changes

store/rootmulti/store.go Fixed Show fixed Hide fixed

store/rootmulti/store.go Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Dec 8, 2022

View reviewed changes

store/rootmulti/store.go Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Dec 8, 2022

View reviewed changes

store/rootmulti/store.go Fixed Show fixed Hide fixed

chillyvee force-pushed the v0466_snapshot_recover_from_exporter_error_master branch from 52fe0c9 to 8027524 Compare March 3, 2023 22:36

chillyvee force-pushed the v0466_snapshot_recover_from_exporter_error_master branch 2 times, most recently from 92e3955 to aae329a Compare March 3, 2023 22:48

Snapshot create and restore debug logs for real-time monitoring

267c64e

chillyvee force-pushed the v0466_snapshot_recover_from_exporter_error_master branch from aae329a to 267c64e Compare March 3, 2023 22:50

tac0turtle approved these changes Mar 4, 2023

View reviewed changes

Merge branch 'main' into v0466_snapshot_recover_from_exporter_error_m…

14d4f35

…aster

odeke-em requested changes Mar 5, 2023

View reviewed changes

store/rootmulti/store.go Outdated Show resolved Hide resolved

store/rootmulti/store.go Outdated Show resolved Hide resolved

chillyvee and others added 2 commits March 6, 2023 13:49

Remove encoding/hex dependency

dbf802d

Merge branch 'main' into v0466_snapshot_recover_from_exporter_error_m…

217f71c

…aster

odeke-em approved these changes Mar 6, 2023

View reviewed changes

alexanderbez approved these changes Mar 6, 2023

View reviewed changes

store/rootmulti/store.go Outdated Show resolved Hide resolved

tac0turtle and others added 2 commits March 6, 2023 17:22

Update store/rootmulti/store.go

40cdf66

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

Merge branch 'main' into v0466_snapshot_recover_from_exporter_error_m…

b2d811e

…aster

tac0turtle enabled auto-merge (squash) March 6, 2023 16:23

tac0turtle added the A:automerge Automatically merge PR once all prerequisites pass. label Mar 6, 2023

tac0turtle merged commit 183dde8 into cosmos:main Mar 6, 2023

mergify bot mentioned this pull request Mar 6, 2023

fix: snapshot recover from exporter error (backport #13935) #15273

Merged

julienrbrt reviewed Mar 6, 2023

View reviewed changes

julienrbrt mentioned this pull request Mar 6, 2023

chore: remove unnecessary condition #15277

Merged

19 tasks

faddat mentioned this pull request Mar 28, 2023

update linter to 1521 ChihuahuaChain/cosmos-sdk#6

Closed

19 tasks

0xstepit mentioned this pull request Jul 1, 2024

chore(cachemulti): add Copy method evmos/cosmos-sdk#52

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: snapshot recover from exporter error #13935

fix: snapshot recover from exporter error #13935

chillyvee commented Nov 20, 2022 •

edited

Loading

chillyvee commented Nov 20, 2022 •

edited

Loading

tac0turtle Nov 20, 2022

chillyvee Nov 20, 2022

chillyvee commented Nov 21, 2022 •

edited

Loading

tac0turtle Nov 21, 2022

chillyvee Nov 22, 2022

chillyvee Dec 2, 2022

faddat commented Nov 22, 2022

chillyvee commented Nov 24, 2022

odeke-em Nov 25, 2022

chillyvee Nov 27, 2022

chillyvee Nov 27, 2022

odeke-em Nov 27, 2022

chillyvee commented Dec 2, 2022 •

edited

Loading

chillyvee commented Dec 2, 2022 •

edited

Loading

alexanderbez Dec 7, 2022

chillyvee Mar 3, 2023

alexanderbez Dec 7, 2022

chillyvee Mar 3, 2023

chillyvee commented Dec 7, 2022

tac0turtle commented Dec 8, 2022

julienrbrt commented Dec 8, 2022 •

edited

Loading

chillyvee commented Dec 8, 2022 •

edited

Loading

likhita-809 commented Dec 9, 2022

chillyvee commented Mar 3, 2023

chillyvee commented Mar 3, 2023

odeke-em left a comment

chillyvee commented Mar 5, 2023 •

edited

Loading

odeke-em left a comment

julienrbrt commented Mar 6, 2023

julienrbrt commented Mar 6, 2023

mergify bot commented Mar 6, 2023 •

edited

Loading

julienrbrt Mar 6, 2023 •

edited

Loading

tac0turtle Mar 6, 2023


		for key, store := range rs.stores {
		rs.logger.Debug("prune store", "key", key)

	func (st Store) Export(version int64) (iavl.Exporter, error) {
	istore, err := st.GetImmutable(version)
	if err != nil {
	return nil, errorsmod.Wrapf(err, "iavl export failed for version %v", version)
	}
	tree, ok := istore.tree.(*immutableTree)
	if !ok \|\| tree == nil {
	return nil, fmt.Errorf("iavl export failed: unable to fetch tree for version %v", version)
	}
	export, err := tree.Export()
	if err != nil {
	return nil, err
	}
	return export, nil

fix: snapshot recover from exporter error #13935

fix: snapshot recover from exporter error #13935

Conversation

chillyvee commented Nov 20, 2022 • edited Loading

Description

Author Checklist

Reviewers Checklist

chillyvee commented Nov 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chillyvee commented Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

faddat commented Nov 22, 2022

chillyvee commented Nov 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chillyvee commented Dec 2, 2022 • edited Loading

chillyvee commented Dec 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chillyvee commented Dec 7, 2022

tac0turtle commented Dec 8, 2022

julienrbrt commented Dec 8, 2022 • edited Loading

chillyvee commented Dec 8, 2022 • edited Loading

likhita-809 commented Dec 9, 2022

chillyvee commented Mar 3, 2023

chillyvee commented Mar 3, 2023

odeke-em left a comment

Choose a reason for hiding this comment

chillyvee commented Mar 5, 2023 • edited Loading

odeke-em left a comment

Choose a reason for hiding this comment

julienrbrt commented Mar 6, 2023

julienrbrt commented Mar 6, 2023

mergify bot commented Mar 6, 2023 • edited Loading

✅ Backports have been created

julienrbrt Mar 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chillyvee commented Nov 20, 2022 •

edited

Loading

chillyvee commented Nov 20, 2022 •

edited

Loading

chillyvee commented Nov 21, 2022 •

edited

Loading

chillyvee commented Dec 2, 2022 •

edited

Loading

chillyvee commented Dec 2, 2022 •

edited

Loading

julienrbrt commented Dec 8, 2022 •

edited

Loading

chillyvee commented Dec 8, 2022 •

edited

Loading

chillyvee commented Mar 5, 2023 •

edited

Loading

mergify bot commented Mar 6, 2023 •

edited

Loading

julienrbrt Mar 6, 2023 •

edited

Loading