Holesky Full Forest Node OutOfMemoryError: Java heap space

[siladu]

During burn-in of 24.5.0-RC1, dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest received OutOfMemoryError: Java heap space on multiple occassions.

I have collected some analysis here https://hackmd.io/9QSLUQ-bSP2k3R4d_LETKg?view

Relevant DEBUG logging was added but the issue hasn't reoccurred since the latest Besu restart.

We are waiting for the issue to present itself again, or for the node to get in sync.

Attaching the INFO logs and the Eclipse MemoryAnalyzer reports generated from the heapdump.

dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest_info_logs.tar.gz
dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest_Leak_Suspects.zip
dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest_Top_Components.zip

Heapdump and DEBUG logs were too large to add here but let me know if you want them.
Note, DEBUG logs may not help as they are were taken during a period of what appears to be normal behaviour.