core, trie: intermediate mempool between trie and database #15857

karalabe · 2018-01-11T17:07:50Z

This PR is backwards incompatible! A chain starting using this PR will not be able to go back.

Currently the primary bottleneck on go-ethereum is disk IO. The database and especially the state trie within became gigantic (70M nodes on mainnet), which means that block processing takes a very long time to read data, and a very long time to write data. To make things worse, data constantly accumulates on disk, making both reads and writes constantly slower.

This PR tires to address to lowest hanging fruit of this problem, namely to try and reduce the amount of useless (or only temporarily useful) data written to disk in the first place. The important observation behind this PR is that a small part of every state update (i.e. new block) is useful data (e.g. new contract, new storage entry), but the most part of a state update is internal trie nodes which have a limited lifetime. This means, that most of the data hitting disk will become stale within a few blocks, so if we can avoid writing them in the first place, data usage and disk IO get significantly reduced.

The PR achieves this goal by inserting an in-memory trie.Database between our low level disk database, and our higher level trie abstraction. Whenever we commit a trie to "disk" it actually ends up cached in memory. From all intents and purposes, tries and go-ethereum think they've committed the data to disk, but they in reality get short circuited halfway down.

Keeping the recent nodes (trie diffs) in memory permits us to do state pruning based on reference counting garbage collection. Every trie node added to the memory database maintains a list of children references and parent counts. Each block processed adds a new set of nodes and references references; whereas decently old blocks dereference old state roots. This results in the trie.Database being a rolling garbage-collected trie-forest containing the most recent 128 tries only.

Of course, this in-memory database needs to be flushed eventually as it just grows with useful data ad infinitum. The PR contains two threshold that triggers the flushing of the tries:

If the memory consumption exceeds the user's allowance (25% of --cache), the tries are flushed to disk. This ensures that we keep and discard as much junk in memory-only as we can, but we have a hard limit that prevents the node from crashing off.
If the total block processing time exceeds 5 minutes, the tries are flushed to disk. This ensures that if the node terminates suddenly (crash, power failure), we lose at most 5 minutes worth of processing time.

The rest of this PR is the massive changes necessary to integrate this concept into go-ethereum. An important-to-mention extra modification is in the downloader:

Fast sync no longer randomizes pivot points from the last 256 blocks (since we only maintain the most recent 128 blocks), rather picks head - 64.
Fast sync no longer locks in the pivot point, rather rolls it forward if it gets older than head - 128 blocks. This is needed since we discard old state (unlike currently).
The PR also fixes various downloader bugs, which made state sync hang until now.

Some random benchmarks from this PR (they've been run during implementation, final results are a tad better):

Disk usage is significantly reduced (more so on chain genesis, less on chain head), resulting in significantly less disk IO, disk usage and processing time. Homestead (blocks 1..1919999) full import on a Core2 CPU, 4GB RAM laptop, 1GB cache:

Geth	Disk	Elapsed	Database	Disk IO¹
v1.8.0	HDD 5400RPM	7:55:07	4.232 GB	43.712 GB
v1.7.3	HDD 5400RPM	33:35:11	26.535 GB	716.300 GB

Disk growth is significantly reduced on mainnet after a successful fast sync, and after about a week of running this PR versus master, the block processing time is 20% faster (originally they are the same, but the disk growth of the non-pruned node eventually slows everything down).

Beside modifying the downloader to cater for state unavailability due to trie pruning, the PR also fixes the fast sync memory consumption issue, resulting in a much more pleasant fast sync experience:

Note, if you want to run an archive node without garbage collection, please run with --gcmode=archive.

Checklist to test before merging:

Arachnid · 2018-01-12T11:30:12Z

trie/mempool.go

@@ -0,0 +1,181 @@
+// Copyright 2017 The go-ethereum Authors


Arachnid · 2018-01-12T11:39:05Z

trie/mempool.go

+	parents  map[common.Hash]int                      // Number of live nodes referencing a given one
+	children map[common.Hash]map[common.Hash]struct{} // Set of children referenced by given nodes
+
+	gctime  time.Duration      // Time spent on garbage collection since last commit


This is reference counting, not garbage collection.

Actually, it is garbage collection, it measures the data/time saved/spent on dereferencing stuff in memory and cleaning up things we don't need to save to disk in the end. So imho it is exactly gc :)

Arachnid · 2018-01-12T12:44:19Z

trie/hasher.go

 			if herr != nil {
 				h.mu.Lock() // rarely if ever locked, no congenstion
 				err = herr
 				h.mu.Unlock()
 			}
 		}
 		// If we're not running in threaded mode yet, span a goroutine for each child
-		if !h.threaded {
+		/*if !h.threaded {


Can this be deleted outright?

Yeah, left it in to investigate

Arachnid · 2018-01-12T12:46:26Z

trie/hasher.go

+	}
+	// Account nodes have very specific sizes, discard anything else
+	// TODO(karalabe): Seriously? Dafuq man?!
+	if size := len(val); size < 70 || size > 102 {


Why not pass in a function to decode values and return references to the trie constructor instead of this hack?

Arachnid · 2018-01-12T12:48:23Z

trie/trie.go

@@ -447,12 +448,19 @@ func (t *Trie) resolve(n node, prefix []byte) (node, error) {
 func (t *Trie) resolveHash(n hashNode, prefix []byte) (node, error) {
 	cacheMissCounter.Inc(1)

+	// Try to load the node from the recent mempool


Why not make the mempool do the disk fallback?

Arachnid · 2018-01-12T12:50:18Z

🔫

ekryski · 2018-01-19T20:32:05Z

@karalabe let me know if you need help testing this. We're close to launching bitblocks.io and are actively testing a bunch of the stuff you have in your list on both SSDs and HDs.

Prior to this 1.8 was getting hung up on the last few hundred blocks during a full node sync and wasn't pulling in new chain segments, just new state entries and with 1.7.3 a 7200rpm disk can't catch up.

Nice work! 👏

holiman · 2018-01-25T13:53:48Z

@ekryski If you can test this PR out, that'd be appreciated. It needs all the testing it can get, and I think it's getting close to finished.

karalabe · 2018-01-25T14:11:04Z

But please note, once you start importing blocks with this PR, there's no going back to 1.7.3. The old release will consider the database corrupted due to missing data.

zsfelfoldi · 2018-01-26T20:47:01Z

core/blockchain.go

-
+		for !bc.triegc.Empty() {
+			root, number := bc.triegc.Pop()
+			if uint64(-number) > chosen {


Since we are going to lose precision here after 16M blocks (which seems a bit ugly but acceptable for the time being), I'd at least make this comparison err on the safe side in order to never delete anything newer than chosen:

if -number >= float32(chosen+1) {

holiman · 2018-01-30T16:42:18Z

eth/downloader/queue.go

+		for _, result := range results {
+			var size common.StorageSize
+
+			blob, _ := rlp.EncodeToBytes(result.Receipts) // TODO(karalabe): Whoah, nice hack, super optimal


:)
Don't forget that TODO.
Q: Wouldn't it be pretty simple to implement rlp.CountBytes(...), which recursively just counts but does not actually allocate memory?

Actually, you can do this instead, in rlp.go:

// EncodedBytesize returns the size that RLP encoding of val would have, // if EncodeToBytes was called func EncodedBytesize(val interface{}) (int, error) { eb := encbufPool.Get().(*encbuf) defer encbufPool.Put(eb) eb.reset() if err := eb.encode(val); err != nil { return 0, err } return eb.size(), nil }

And then

bloblen,_ := rlp.EncodeByteSize(result.Receipts) ....

That'll roughly halve the memory allocations there.

holiman · 2018-01-31T10:04:13Z

At commit c137a71, tested with a copydb:

GoodMirek · 2018-01-31T16:02:54Z

While running any of four latest commits (c83ad93 - d0991c5) on mainnet, the fast sync eventually blocks before it downloads all the block headers and receipts. The DB does not get corrupt, after geth process restart it proceeds from where it left off and after some time it blocks again. Slower the node is the more often the issue happens. I am testing on one relatively slow node and one faster, in terms of block storage speed.
Are you aware of this issue? Is there anything I can collect to help?

karalabe · 2018-02-01T11:50:07Z

@GoodMirek Thanks for the report! Yes, I've added some modifications to fast sync to cap the amount of memory it uses (currently it OOMs on mainnet) and it touches some sensitive scheduling logic, so there's a high probability I screwed something up. I will try to repro it locally, in the mean time any logs you can give us might help (potentially run with --vmodule=downloader=5).

karalabe · 2018-02-01T12:18:45Z

@GoodMirek Can you also provide us with hw specs so we can better understand/try to repro? Also, when it "hangs" does it hang forever, or does it continue after a few (5-10 mins)? Pondering whether it could be database compaction in the background on an HDD.

karalabe · 2018-02-01T14:23:31Z

@GoodMirek Tried to sync the PR on both a new laptop + SSD and an old laptop + HDD. Both seem to work fine, but the database compaction does catch both, causing a 8 / 18 minute pause on SSD / HDD. We probably need to add some user warning to display what's happening, but the code does seem to work correctly nonetheless.

GoodMirek · 2018-02-02T10:26:38Z

@karalabe You are right, I was not patient enough. After 10-20 minutes the sync proceeded. Now I face another issue - once all block headers and block receipts are downloaded, I am receiving thousands of warnings "Stalling state sync, dropping peer" and only rarely new state entries are imported. After geth restart, it imports new state entries quickly for about 5-30 minutes and then issue starts happening.
On stable version, both nodes were able to import the state entries at constant slow pace and then switched from fast to full sync.
I am going to retry with your yesterday's commits.

Regarding HW, these are cloud VPS, from different providers, without reasonable HW spec beside number of cores, size of RAM, diskspace size. They differ in speed of disk subsystem in order of magnitude.

karalabe · 2018-02-02T10:46:14Z

@GoodMirek Yup, the Stalling state sync issue was a long standing one we had in Geth. It was really hard to reproduce so we couldn't properly debug it. My current work made it pop up very often, so I found the issue and also fixed it :) Latest code from yesterday should sync without issues (although it's disk IO intensive, especially after pulling in 70M states).

GoodMirek · 2018-02-02T11:53:27Z

UPDATE: Turned out it just took much longer to allocate the memory, not sure whether it is a cache or a leak.

@karalabe I confirm that with commit 951d398 state entries are being imported smoothly. Also, your branch no longer dies on OOM, what used to be very frequent issue during initial sync in my setup. Surprisingly, it even does not allocate all the memory specified as --cache, being it 2048 or 4096. It allocates just 75%, looking like the 25% dedicated to intermediate mempool is not used, at least during state entries import; however this is pure guess, I did not use delve or anything else to analyze the actual behavior.

My plan so far have been to let both systems fully sync with commit 951d398 and observe the behavior. In past:
Slow disk system, 8GB RAM : Geth no luck, Parity with 20 peers frequently lagging, needed outgoing bandwidth limitation on per-connection basis.
Fast disk system, 4GB RAM: Geth 1.8.0 with 500 peers kept in sync, most of the peers were LES, outgoing bandwidth limitation on per-connection basis applied only occassionaly.

Then, I plan to drop DB on both and retry with latest commit.

fjl

I don't have time to review in detail now, the changes look OK but there are still some rough edges. It's up to you when this gets merged.

fjl · 2018-02-02T15:08:01Z

cmd/utils/flags.go

+	CacheGCTimeoutFlag = cli.DurationFlag{
+		Name:  "cache.gc.timeout",
+		Usage: "Maximum time after which to flush memory-only tries to disk",
+		Value: 5 * time.Minute,


I don't think we need CLI flags for such settings. You can always set them in the config file if you really want to tune details like this.

Fixed, PTAL.

fjl · 2018-02-02T15:11:55Z

eth/downloader/queue.go

+		for _, result := range results {
+			var size common.StorageSize
+
+			blob, _ := rlp.EncodeToBytes(result.Receipts) // TODO(karalabe): Whoah, nice hack, super optimal


You don't really need the encoded size here, any approximation of the size is enough. Maybe add a Size method to Receipt, Block, Transaction that returns the size of the struct + size of the content of slice fields.

The transactions already had a Size method which worked based on RLP serialization. I've added a size method to the rest (which only approximates). PTAL

mariushudea · 2018-02-06T00:19:25Z

As long as this is gonna involve something backwards incompatible, would you maybe consider changing the defaults of the database (larger files, less often compacting), so that each individual file in the db is larger than ~ 2MB and we don't have tens of thousands of files (see the thread i posted in issues and the feedback for that) ?

rjl493456442 · 2018-02-06T07:16:21Z

Not look at the code. But if so, it means we can lost most of versioned data(e.g. We can not access historical data via old state root).

…15857) This commit reduces database I/O by not writing every state trie to disk.

vsdigitall · 2018-04-02T07:14:50Z

I have the following hardware:

- Intel Xeon E3-1270v6 3.80GHz
- 16 GB ECC DDR4 RAM
- 250 GB PCI-E SSD
- 100 Mbit internet

And it took about 30 hours already to sync with following command:
/root/p/geth/geth --fast --cache=1024 --datadir /root/p/geth/data --ipcpath /root/p/geth/geth.ipc

Geth version: Geth/v1.8.3-stable-329ac18e/linux-amd64/go1.10.
pulledStates: 62924164, geth is still syncing...

@karalabe I can give you an access to the server so you can check why it syncing so slow.

duanbing · 2018-05-21T10:05:17Z

In the interface state.Trie, the
CommitTo(trie.DatabaseWriter) (common.Hash, error) was changed to Commit(onleaf trie.LeafCallback) (common.Hash, error), so how can I commit the changes to Leveldb by manual?
@karalabe

karalabe · 2018-05-21T10:17:54Z

https://godoc.org/github.com/ethereum/go-ethereum/trie#Database.Commit

duanbing · 2018-05-21T10:43:24Z

Thanks, solved it like this:

mdb, err := ethdb.NewLDBDatabase("/tmp/a.txt", 100, 100)
must(err)
db := state.NewDatabase(mdb)
root := common.Hash{}
statedb, err := state.New(root, db)

//update statedb

root, err = statedb.Commit(true)
must(err)
fmt.Println("Root Hash", root.Hex())

err = db.TrieDB().Commit(root, true)
must(err)

BTW, what a nice design of "intermediate mempool between trie and database"!

karalabe · 2018-05-21T10:46:39Z

BTW, what a nice design of "intermediate mempool between trie and database"!

You're welcome to suggest cleaner APIs that retain the garbage collection capability.

duanbing · 2018-05-21T11:37:07Z

Would like to and am doing it. @karalabe

…15857) This commit reduces database I/O by not writing every state trie to disk.

karalabe requested review from fjl and Arachnid January 11, 2018 17:07

karalabe requested a review from holiman as a code owner January 11, 2018 17:07

karalabe mentioned this pull request Jan 11, 2018

why my chaindata size up to 400GB? #15797

Closed

Arachnid approved these changes Jan 12, 2018

View reviewed changes

karalabe force-pushed the trie-node-mempool branch 2 times, most recently from 69a919f to 8998ed2 Compare January 17, 2018 08:31

zsfelfoldi mentioned this pull request Jan 18, 2018

State trie garbage collection #15903

Closed

ekryski mentioned this pull request Jan 19, 2018

What is the upper bound of "imported new state entries"? When will it end? #15616

Closed

karalabe force-pushed the trie-node-mempool branch 2 times, most recently from 06b2545 to ff31e8e Compare January 24, 2018 14:32

zsfelfoldi reviewed Jan 26, 2018

View reviewed changes

karalabe force-pushed the trie-node-mempool branch 3 times, most recently from 69e00e1 to c137a71 Compare January 30, 2018 14:49

holiman reviewed Jan 30, 2018

View reviewed changes

karalabe force-pushed the trie-node-mempool branch from c137a71 to c83ad93 Compare January 31, 2018 10:49

fjl reviewed Feb 2, 2018

View reviewed changes

karalabe added 11 commits February 5, 2018 16:47

light: make the trie mempool play nice with CHTs

a14bfec

core: warn the user that we're saving state on shutdown

a4cff48

core: report pretty storage sizes for receipt inserts

4aeed07

eth/downloader: termiante state sync cleanly

92716dd

core: make receipt and headers have the same log format

87a37c3

core, eth, les, light: hide triedb behind state database

104ccf5

core: remote a debugging warning log

3c30b28

eth/downloader: fix statesync fast restart race, hang test

5f253f6

cmd: drop trie cache timeout flag (available via config)

30ddc90

core, eth: avoid RLP serialization in downloader queue

49bdc8b

cmd, core, eth, light, trie: gcmode flag + supress archive logs

54b8ded

karalabe force-pushed the trie-node-mempool branch from 88f5fe4 to 54b8ded Compare February 5, 2018 14:48

holiman added this to the 1.8.0 milestone Feb 5, 2018

fjl merged commit 55599ee into ethereum:master Feb 5, 2018

holiman mentioned this pull request Feb 8, 2018

Very slow syncing on Hard Drive #14895

Closed

karalabe mentioned this pull request Feb 11, 2018

eth: only disable fast sync after success #16062

Merged

pedropombeiro mentioned this pull request Feb 24, 2018

Update to go-ethereum 1.8.1 status-im/status-go#702

Merged

5 tasks

prestonvanloon pushed a commit to prestonvanloon/go-ethereum that referenced this pull request Apr 2, 2018

core, trie: intermediate mempool between trie and database (ethereum#…

fcff711

…15857) This commit reduces database I/O by not writing every state trie to disk.

whilei mentioned this pull request May 28, 2018

[WIP]: Fix/sync stuck near head ethereumproject/go-ethereum#606

Closed

karalabe mentioned this pull request May 29, 2018

core, eth, trie: streaming GC for the trie cache #16810

Merged

mariameda pushed a commit to NiluPlatform/go-nilu that referenced this pull request Aug 23, 2018

core, trie: intermediate mempool between trie and database (ethereum#…

601fa11

…15857) This commit reduces database I/O by not writing every state trie to disk.

summerpro mentioned this pull request Aug 14, 2021

Pruning optimization suggestions cosmos/iavl#415

Closed

core, trie: intermediate mempool between trie and database #15857

core, trie: intermediate mempool between trie and database #15857

Conversation

karalabe commented Jan 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arachnid commented Jan 12, 2018

ekryski commented Jan 19, 2018 • edited Loading

holiman commented Jan 25, 2018

karalabe commented Jan 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holiman Jan 31, 2018 • edited Loading

Choose a reason for hiding this comment

holiman commented Jan 31, 2018

GoodMirek commented Jan 31, 2018 • edited Loading

karalabe commented Feb 1, 2018

karalabe commented Feb 1, 2018

karalabe commented Feb 1, 2018

GoodMirek commented Feb 2, 2018 • edited Loading

karalabe commented Feb 2, 2018

GoodMirek commented Feb 2, 2018 • edited Loading

fjl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariushudea commented Feb 6, 2018

rjl493456442 commented Feb 6, 2018

vsdigitall commented Apr 2, 2018 • edited Loading

duanbing commented May 21, 2018 • edited Loading

karalabe commented May 21, 2018

duanbing commented May 21, 2018 • edited Loading

karalabe commented May 21, 2018

duanbing commented May 21, 2018

karalabe commented Jan 11, 2018 •

edited

Loading

ekryski commented Jan 19, 2018 •

edited

Loading

holiman Jan 31, 2018 •

edited

Loading

GoodMirek commented Jan 31, 2018 •

edited

Loading

GoodMirek commented Feb 2, 2018 •

edited

Loading

GoodMirek commented Feb 2, 2018 •

edited

Loading

vsdigitall commented Apr 2, 2018 •

edited

Loading

duanbing commented May 21, 2018 •

edited

Loading

duanbing commented May 21, 2018 •

edited

Loading