[Bonsai] OutOfMemory on ETC full sync

[diega]

Description

As an ETC client, I want to full sync my node so that can interact with the network.

Acceptance Criteria

• passes ETC block 13771821

Steps to Reproduce (Bug)

1. $ ~/besu-21.10.0-RC3/bin/besu --network=classic --data-path=/home/admin/classic-besu-data-bonsai --p2p-host=<YOUR_IP> --Xdata-storage-format=BONSAI --sync-mode=FULL
2. wait to sync (~2 weeks)

Expected behavior: synchronization completes

Actual behavior: loops indefinitely with

2021-10-26 16:23:00.389+00:00 | EthScheduler-Services-5 (importBlock) | DEBUG | BonsaiWorldStateArchive | Paired Rollback 0xb68fe78bef86e6ef979742c440c05af83f0510181576e4cec31a9b3c8723d4fc
2021-10-26 16:23:00.389+00:00 | EthScheduler-Services-5 (importBlock) | DEBUG | BonsaiWorldStateArchive | Paired Rollforward 0x48b932cbb4593afc7b2174f619ad0fc25900411f275072be47df7bd62a180dda

and eventually starts throwing OutOfMemoryException

Frequency: everytime

Versions

• Software version: 21.10.0-RC3
• Java version: openjdk version "11.0.6" 2020-01-14
• OS Name & Version:

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel Version: Linux ip-10-0-0-209 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1+deb9u1 (2020-06-07) x86_64 GNU/Linux

• Cloud VM, type, size: Amazon Web Services m5-large

Additional Information

A few weeks ago I started a full sync of ETC using Bonsai just to measure storage size. Everything was moving smooth but at block 13771821 it started to throw OutOfMemoryExceptions. Even on restart I got the same behavior. I tried doubling the heap size, but it ate it all. I ran Java Flight Recorder and I saw that memory was consumed mostly by an ArrayList<TrieLogLayer> (I have the .jfr file if anyone is interested, I don't want to upload it here b/c it may contain some sensitive information). Then I restarted with DEBUG logging and I saw the repeating lines pasted above.

Looking at the code, in BonsaiWorldStateArchive:215, it seems to me that the while condition isn't updated during the execution, which may explain the repetitive log and the increasing List<TrieLogLayer>.

I can also share the Bonsai DB just to test so you don't need to fully sync it (50GB)

[leontastic]

I'm seeing this issue on ETH too in 21.7.4

[shemnon]

The restart is one needed change. There is a second change in that the ArrayList<TrieLogLayer> should probably be replaced with a cache that is either FIFO or prioritized based on log height, and configurable size via the CLI.

[shemnon]

The sync code still needs to be updated to not out of memory.

[matkt]

The sync code still needs to be updated to not out of memory.

We fixed that on this one #2934

[shemnon]

Got it, it was a local variable based list not a object field list growing out of control