WIP store utxo set in the database and not in ram 2 #2159

tmpolaczyk · 2022-03-03T12:49:57Z

This PR stores consolidated entries of the UTXO set in the storage, and the non-consolidated entries in RAM. Consolidated means included in a consolidated superblock.

Close #1275

Warning: this PR makes breaking changes to the storage, so backup before testing.

tmpolaczyk · 2022-03-03T14:24:55Z

Unresolved problem: migrations. Currently the node stores the chain state using the magic number from consensus constants as the key. In this PR, UTXOs are stored without any magic number. This means that the storage cannot be reused by the same node on different testnets, because the old UTXOs will still be marked as valid. So we need to either add the magic number to the UTXO key, as format!("UTXO-{}-{}", magic, output_pointer), or find some way to detect if the environment has changed and delete all the existing UTXOs from the storage (this should now be possible thanks to the iteration support in storage).

This is still unresolved but I think I will just add a check to ensure that there is only one chain state, and panic otherwise.

data_structures/src/utxo_pool.rs

node/src/actors/chain_manager/actor.rs

lrubiorod · 2022-03-10T11:10:37Z

I think we should have a discussion before merge this PR due to be a change without backward compatible in the database

tmpolaczyk · 2022-03-29T16:24:55Z

There are some problems when the synchronization process is interrupted, it is possible for the node to become forked and the only way to recover is to use the rewind command. I think it is related to #2102, will keep investigating.

node/src/actors/chain_manager/actor.rs

feat(node): allow access to db outside StorageManager feat(storage): allow mutable operations using shared references

Add db_iter to UnspentOutputsPool

To ensure that the utxo set is always updated atomically: it should be impossible to leave the database in a state where the utxo set was only halfway updated

This fixes a bug where recently written UTXOs were not detected by the validation code, leading to errors and possible forks.

tmpolaczyk · 2022-04-01T14:35:14Z

There are some problems when the synchronization process is interrupted, it is possible for the node to become forked and the only way to recover is to use the rewind command. I think it is related to #2102, will keep investigating.

This is fixed now

tmpolaczyk · 2022-04-04T14:16:27Z

This is still unresolved but I think I will just add a check to ensure that there is only one chain state, and panic otherwise.

This is also implemented now

lrubiorod · 2022-04-05T10:22:26Z

node/src/storage_mngr.rs


 pub use node_migrations::*;
+use witnet_storage::storage::WriteBatch;


lrubiorod · 2022-04-05T10:23:00Z

storage/src/backends/hashmap.rs

@@ -3,38 +3,97 @@
 //! Storage backend that keeps data in a heap-allocated HashMap.
 use std::collections::HashMap;

-use crate::storage::{Result, Storage};
+use crate::storage::{Result, Storage, StorageIterator, WriteBatch, WriteBatchItem};
+use std::sync::{RwLock, RwLockReadGuard};


tmpolaczyk mentioned this pull request Mar 3, 2022

WIP store utxo set in the database and not in ram #1583

Closed

tmpolaczyk force-pushed the utxo-set-db-2 branch 2 times, most recently from 99321c3 to 9aaed58 Compare March 4, 2022 10:51

lrubiorod reviewed Mar 4, 2022

View reviewed changes

data_structures/src/utxo_pool.rs Outdated Show resolved Hide resolved

lrubiorod reviewed Mar 4, 2022

View reviewed changes

data_structures/src/utxo_pool.rs Outdated Show resolved Hide resolved

tmpolaczyk force-pushed the utxo-set-db-2 branch from 9aaed58 to 4ba3001 Compare March 8, 2022 14:48

lrubiorod reviewed Mar 9, 2022

View reviewed changes

data_structures/src/utxo_pool.rs Outdated Show resolved Hide resolved

tmpolaczyk force-pushed the utxo-set-db-2 branch 2 times, most recently from 9e4e0b3 to 4f90bca Compare March 9, 2022 15:09

tmpolaczyk commented Mar 9, 2022

View reviewed changes

data_structures/src/utxo_pool.rs Outdated Show resolved Hide resolved

tmpolaczyk commented Mar 9, 2022

View reviewed changes

node/src/actors/chain_manager/actor.rs Outdated Show resolved Hide resolved

tmpolaczyk force-pushed the utxo-set-db-2 branch from 1f4c255 to 75c00e7 Compare March 10, 2022 10:44

tmpolaczyk force-pushed the utxo-set-db-2 branch 2 times, most recently from 2926c6d to ede5917 Compare March 10, 2022 16:20

tmpolaczyk added the breaking change ⚠️ Introduces breaking changes label Mar 15, 2022

tmpolaczyk force-pushed the utxo-set-db-2 branch from ede5917 to d5b5311 Compare March 21, 2022 10:48

tmpolaczyk force-pushed the utxo-set-db-2 branch from d5b5311 to ca16cdc Compare March 28, 2022 14:31

lrubiorod marked this pull request as ready for review March 28, 2022 14:32

tmpolaczyk force-pushed the utxo-set-db-2 branch from 6134ca1 to 297a4d3 Compare March 30, 2022 14:17

tmpolaczyk commented Mar 31, 2022

View reviewed changes

node/src/actors/chain_manager/actor.rs Outdated Show resolved Hide resolved

tmpolaczyk force-pushed the utxo-set-db-2 branch from 78662e0 to fff534e Compare April 1, 2022 14:29

tmpolaczyk added 6 commits April 1, 2022 16:31

feat: first approach to UTXO set in db

079e972

feat(node): allow access to db outside StorageManager feat(storage): allow mutable operations using shared references

feat(storage): implement iteration support

311d2df

feat: use Diff in UTXO set

e36380c

Add db_iter to UnspentOutputsPool

refactor(utxo_pool): move block_number to UtxoDiff

b9d3ccb

feat(node): persist UTXO set on superblock consolidation

c26f291

fix(storage): use rocksdb iterator that works with rocksdb 5.8

2a882cb

tmpolaczyk added 13 commits April 1, 2022 16:32

fix: delete all UTXOs from DB when doing a rewind

1c1ecd3

feat(node): check that there is only one ChainState in the DB

f19fdc3

feat: optimize getBalance when using own pkh and simple flag

17abed5

refactor(node): improve storage migrations code

b50c656

chore: fix clippy warnings after rebase

9e7aea7

fix: handle case of interrupted utxo set migration

be767fc

feat(storage): implement atomic WriteBatch

45c1579

feat: use WriteBatch in UtxoSet

7ebd656

To ensure that the utxo set is always updated atomically: it should be impossible to leave the database in a state where the utxo set was only halfway updated

feat: add progress indicator to utxo set migration

4057bca

fix: fix utxo set insert and remove in the same superblock

15cb65c

feat: remove utxo get put logs

4de1364

feat: allow persist chain state atomically in a WriteBatch

853d713

fix(node): wait for chain state consolidation

6d42a2b

This fixes a bug where recently written UTXOs were not detected by the validation code, leading to errors and possible forks.

tmpolaczyk force-pushed the utxo-set-db-2 branch from fff534e to 6d42a2b Compare April 1, 2022 14:32

fix(node): check for multiple chain states in storage

28f8084

tmpolaczyk force-pushed the utxo-set-db-2 branch from cca29fa to 28f8084 Compare April 4, 2022 15:10

lrubiorod reviewed Apr 5, 2022

View reviewed changes

node/src/storage_mngr.rs Outdated

pub use node_migrations::*;

use witnet_storage::storage::WriteBatch;

Copy link

Contributor

lrubiorod Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fmt

lrubiorod reviewed Apr 5, 2022

View reviewed changes

lrubiorod approved these changes Apr 5, 2022

View reviewed changes

chore: fmt

f69598e

tmpolaczyk merged commit f69598e into witnet:master Apr 5, 2022

This was referenced Apr 6, 2022

feat(storage): implement iteration support #1714

Closed

feat: remove crypto storage backend #2172

Merged

Fix performance of getBalance and getUtxoInfo methods #2173

Closed

Read block_chain from database instead of from memory #2177

Open

tmpolaczyk mentioned this pull request Jun 9, 2022

feat(nodestats): add counters to monitor failed data requests #2218

Draft

Tommytrg mentioned this pull request Jun 27, 2022

Missing blocks in storage #2102

Open

tmpolaczyk deleted the utxo-set-db-2 branch July 11, 2022 15:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP store utxo set in the database and not in ram 2 #2159

WIP store utxo set in the database and not in ram 2 #2159

Uh oh!

tmpolaczyk commented Mar 3, 2022 •

edited

Loading

Uh oh!

tmpolaczyk commented Mar 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrubiorod commented Mar 10, 2022 •

edited

Loading

Uh oh!

tmpolaczyk commented Mar 29, 2022

Uh oh!

Uh oh!

tmpolaczyk commented Apr 1, 2022

Uh oh!

tmpolaczyk commented Apr 4, 2022

Uh oh!

lrubiorod Apr 5, 2022

Uh oh!

lrubiorod Apr 5, 2022

Uh oh!

Uh oh!


		pub use node_migrations::*;
		use witnet_storage::storage::WriteBatch;

WIP store utxo set in the database and not in ram 2 #2159

WIP store utxo set in the database and not in ram 2 #2159

Uh oh!

Conversation

tmpolaczyk commented Mar 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmpolaczyk commented Mar 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lrubiorod commented Mar 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmpolaczyk commented Mar 29, 2022

Uh oh!

Uh oh!

tmpolaczyk commented Apr 1, 2022

Uh oh!

tmpolaczyk commented Apr 4, 2022

Uh oh!

lrubiorod Apr 5, 2022

Choose a reason for hiding this comment

Uh oh!

lrubiorod Apr 5, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tmpolaczyk commented Mar 3, 2022 •

edited

Loading

lrubiorod commented Mar 10, 2022 •

edited

Loading