Possible memory leak with commits after v0.2.2

[txhsl]

System information

Geth version: 0.3.0-unstable
OS & Version: Linux
Commit hash : a0e8c9c

Expected behaviour

The node memory usage is quite stable and low with v0.2.2.

Actual behaviour

But when we are using the latest commit code to upgrade an existing network, @songb2 confirmed a significant memory usage pump.

The above graph tracks the memory usage of one of our servers, it runs 3 CNs as a part of our privnet.

We stopped the network and started to try to upgrade it from Jan 8th 15:00, and finally started it up in Jan 10th 18:00.

So regardless of the middle, we can see an obvious increase, and finally the server crashed.

Steps to reproduce the behaviour

Set up a privnet with make privnet_start_seven, and keep running it, maybe more than 1 day.

Backtrace

The first graph is from a node running on a0e8c9c for half a week.

The following two are from another node running on 93810b5 for about half a month.

When only select /consensus as scope, it shows that 75% memory usage came from here.

[txhsl]

🤔Quite low RAM on these machines, @songb2 is now upgrading the hardware to see what will happen.

[txhsl]

To have a comparison, here are some memory dumps from T4 v0.2.2. The memory usage is 64.01 MB and 71.23 MB in total.

Seems like the consensus part increased a lot.

[txhsl]

Things work well with #232, so it is not from the original Geth code.

[songb2]

On devnet, I use one machine to run a single node of seven nodes, it also shows mem leak symptom
Below is the commit which run:
Version: 0.3.0-unstable
Git Commit: 1e9eaf6
Git Commit Date: 20250110
Architecture: amd64
Go Version: go1.22.2
Operating System: linux
Below is the mem consumption graph and the mem dump file

memnode6.txt

[AnnaShaleva]

State DB copy takes a lot of resources, and we need to optimize this part. The tricky thing is that we can't just skip some of this calls. I'll take a closer look onto it.

[txhsl]

State DB copy takes a lot of resources.

Yes, maybe.

[txhsl]

The following is the memory history of #326.

The following is the memory history of #360.

The following is the memory history of #332.