fatal error: runtime: cannot allocate memory

[AntonieDavid]

System information

Erigon version: v3.0.0

OS & Version: Linux, Ubuntu 22.04

Erigon Command (with flags/config):

ExecStart=/usr/local/bin/erigon \
  --ws \
  --ws.port=8545 \
  --http \
  --http.port=8545 \
  --http.addr=0.0.0.0 \
  --http.corsdomain="*" \
  --rpc.gascap=1100000000 \
  --authrpc.addr=127.0.0.1 \
  --authrpc.jwtsecret=/data/jwt.hex \
  --authrpc.port=8551 \
  --datadir=/data/ \
  --http.api=debug,eth,net,trace,web3,rpc,txpool,erigon,engine \
  --metrics \
  --metrics.port=6060 \
  --prune.mode=archive \
  --caplin.blocks-archive \
  --caplin.blobs-archive \
  --caplin.states-archive \
  --caplin.blobs-no-pruning=true \
  --beacon.api=beacon,builder,config,debug,node,validator,lighthouse,events \
  --private.api.addr=127.0.0.1:9190 \
  --beacon.api.port=3500 \
  --beacon.api.addr=0.0.0.0

Consensus Layer: Caplin

Consensus Layer Command (with flags/config):

Chain/Network: Issue is happening on both eth mainnet and eth sepolia

Expected behaviour

Nodes are stable all the time

Actual behaviour

The node seems to occasionaly restart itself and follows an OOM behaviour, but server metrics don't show memory spikes above 50% or even lower.

Steps to reproduce the behaviour

server specs are mixed between 64gb with 8 cores and 128gb with 12 cores and the issue happens on both types of servers

Logs

root@eth-beacon-archive-mainnet-euc1-2:~# journalctl -u erigon.service --since "21:31:00" --until "21:32:19"
Apr 08 21:31:00 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:00.757] head updated                             hash=0x3c0a3c06fa00874c62164d5766ede5b40fb0e0645d9ec3f14adaaf2f26c1cd78 numbe>
Apr 08 21:31:03 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:03.225] [Caplin-Archive] Historical States reconstruction slot=11442639 blk/sec=15.18
Apr 08 21:31:13 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:13.442] head updated                             hash=0xf265ff03c70c87717db4ad38707b0e2b51909f29dea4cf7c2b623b6a5f71032f numbe>
Apr 08 21:31:22 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:22.410] P2P                                      app=caplin peers=63
Apr 08 21:31:25 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:25.256] head updated                             hash=0x2c5655b9c895cfbb3204fb9b3e2f24e6cdff61cdb5bda7bdd8850f6bff508074 numbe>
Apr 08 21:31:38 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:38.307] head updated                             hash=0xd2a03ea4a5772482c5e781cec99c0a266924e60232fbe6c0b5c60a35db88a1a3 numbe>
Apr 08 21:31:46 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:46.761] Stopped Caplin to load states
Apr 08 21:31:47 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:47.012] [Caplin-Archive] Historical states antiquated slot=11443518 root=0xeec1d9d83a7c63807de27249fb19839ed671d532d019c063f25>
Apr 08 21:31:47 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [WARN] [04-08|21:31:47.012] Could not to increment beacon state, trying again later app=caplin err="cannot allocate memory, fileName: /data/snapsh>
Apr 08 21:31:50 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:50.480] head updated                             hash=0xee75ece2ab08f093f8b3402ff4cdb42c34a1ad29e3499d658ceeacb7832499c3 numbe>
Apr 08 21:31:54 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:31:54.363] Starting state processing                from=11428129 to=11443615
Apr 08 21:32:01 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:32:01.150] head updated                             hash=0xdcad14e78a160c74cc4cca7be43437bf1e3c9dc6935c7b2dcdc41bf1b5cb9b36 numbe>
Apr 08 21:32:13 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:32:13.206] head updated                             hash=0x6d532c74cb0c88bacaecb5d05fb0d2344a86601806bbbb1ab4fee2e02ab6cb1a numbe>
Apr 08 21:32:16 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:32:16.007] [p2p] GoodPeers                          eth68=10 eth67=10
Apr 08 21:32:16 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:32:16.094] [txpool] stat                            pending=71 baseFee=3500 queued=962 alloc=19.4GB sys=33.9GB
Apr 08 21:32:17 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: [INFO] [04-08|21:32:17.613] [mem] memory stats                       Rss=104.0GB Size=0B Pss=97.3GB SharedClean=6.7GB SharedDirty=0B PrivateClean=>
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: fatal error: runtime: cannot allocate memory
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime stack:
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.throw({0x2c89ce3?, 0x0?})
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/panic.go:1023 +0x5c fp=0x79fa0fffe990 sp=0x79fa0fffe960 pc=0x4613dc
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.persistentalloc1(0x118, 0x20?, 0x5f0ca30)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/malloc.go:1576 +0x24d fp=0x79fa0fffe9e0 sp=0x79fa0fffe990 pc=0x434e4d
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.persistentalloc.func1()
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/malloc.go:1529 +0x28 fp=0x79fa0fffea10 sp=0x79fa0fffe9e0 pc=0x434be8
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.systemstack(0x800000)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/asm_amd64.s:509 +0x4a fp=0x79fa0fffea20 sp=0x79fa0fffea10 pc=0x49b3ea
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: goroutine 124008 gp=0xc18612f880 m=22 mp=0xc19e2c5008 [running]:
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.systemstack_switch()
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/asm_amd64.s:474 +0x8 fp=0xc2e4c74820 sp=0xc2e4c74810 pc=0x49b388
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.persistentalloc(0x49b3d2?, 0xc2e4c748d8?, 0x48fc52?)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/malloc.go:1528 +0x5c fp=0xc2e4c74868 sp=0xc2e4c74820 pc=0x434b7c
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.newBucket(0x1, 0xd)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/mprof.go:228 +0x4e fp=0xc2e4c74890 sp=0xc2e4c74868 pc=0x45750e
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.stkbucket(0x1, 0xb0, {0xc2e4c74920, 0xd, 0xc2e4c74950?}, 0x1)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/mprof.go:316 +0x252 fp=0xc2e4c748e8 sp=0xc2e4c74890 pc=0x4578d2
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.mProf_Malloc(0xc214adea50, 0xb0)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/mprof.go:431 +0xbd fp=0xc2e4c74a68 sp=0xc2e4c748e8 pc=0x457d9d
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.profilealloc(0x0?, 0xc214adea50?, 0xb0?)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/malloc.go:1426 +0x79 fp=0xc2e4c74a90 sp=0xc2e4c74a68 pc=0x4349b9
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.mallocgc(0xa4, 0x2682740, 0x1)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/malloc.go:1267 +0x70a fp=0xc2e4c74b18 sp=0xc2e4c74a90 pc=0x43454a
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.makeslice(0xc2a3cf7200?, 0xc2e4c74b70?, 0x434885?)
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/slice.go:107 +0x49 fp=0xc2e4c74b40 sp=0xc2e4c74b18 pc=0x47bf09
Apr 08 21:32:18 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: github.com/erigontech/erigon-lib/rlp.(*Stream).Bytes(0xc2a3cf7200)


Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.goexit({})
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/asm_amd64.s:1695 +0x1 fp=0xc1697e6fe8 sp=0xc1697e6fe0 pc=0x49d3a1
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: created by github.com/anacrolix/torrent.(*trackerScraper).Run in goroutine 7437
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/tracker_scraper.go:246 +0x108
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: goroutine 7482 gp=0xc1697fca80 m=nil [select, 22 minutes]:
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.gopark(0xc252a97f68?, 0x3?, 0xd8?, 0x7d?, 0xc252a97e9a?)
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/proc.go:402 +0xce fp=0xc3ec527d40 sp=0xc3ec527d20 pc=0x464fce
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.selectgo(0xc3ec527f68, 0xc252a97e94, 0x5e8be20?, 0x0, 0x0?, 0x1)
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/select.go:327 +0x725 fp=0xc3ec527e60 sp=0xc3ec527d40 pc=0x476ea5
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: github.com/anacrolix/torrent.(*trackerScraper).Run(0xc1697e5e00)
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/tracker_scraper.go:290 +0x4d1 fp=0xc3ec527fc8 sp=0xc3ec527e60 pc=0x1074b91
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: github.com/anacrolix/torrent.(*Torrent).startScrapingTracker.func1.gowrap1()
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/torrent.go:2182 +0x25 fp=0xc3ec527fe0 sp=0xc3ec527fc8 pc=0x1065405
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.goexit({})
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/asm_amd64.s:1695 +0x1 fp=0xc3ec527fe8 sp=0xc3ec527fe0 pc=0x49d3a1
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: created by github.com/anacrolix/torrent.(*Torrent).startScrapingTracker.func1 in goroutine 1
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/torrent.go:2182 +0x25c
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: goroutine 7388 gp=0xc16965c1c0 m=nil [select, 1293 minutes]:
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.gopark(0xc1697f8fa0?, 0x2?, 0x80?, 0x11?, 0xc1697f8f8c?)
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/proc.go:402 +0xce fp=0xc1697f8e30 sp=0xc1697f8e10 pc=0x464fce
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.selectgo(0xc1697f8fa0, 0xc1697f8f88, 0x0?, 0x0, 0x0?, 0x1)
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/select.go:327 +0x725 fp=0xc1697f8f50 sp=0xc1697f8e30 pc=0x476ea5
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: github.com/anacrolix/torrent.(*trackerScraper).Run.func1()
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/tracker_scraper.go:248 +0xb4 fp=0xc1697f8fe0 sp=0xc1697f8f50 pc=0x1074cb4
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: runtime.goexit({})
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         runtime/asm_amd64.s:1695 +0x1 fp=0xc1697f8fe8 sp=0xc1697f8fe0 pc=0x49d3a1
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: created by github.com/anacrolix/torrent.(*trackerScraper).Run in goroutine 7430
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]:         github.com/anacrolix/torrent@v1.52.6-0.20231201115409-7ea994b6bbd8/tracker_scraper.go:246 +0x108
Apr 08 21:32:19 eth-beacon-archive-mainnet-euc1-2 erigon[1152689]: goroutine 7462 gp=0xc1695741c0 m=nil [select, 1293 minutes]:
Apr 08 21:32:24 eth-beacon-archive-mainnet-euc1-2 systemd[1]: erigon.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 08 21:32:24 eth-beacon-archive-mainnet-euc1-2 systemd[1]: erigon.service: Failed with result 'exit-code'.
Apr 08 21:32:24 eth-beacon-archive-mainnet-euc1-2 systemd[1]: erigon.service: Consumed 22h 48min 38.931s CPU time.
Apr 08 21:32:30 eth-beacon-archive-mainnet-euc1-2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 8.
Apr 08 21:32:30 eth-beacon-archive-mainnet-euc1-2 systemd[1]: Stopped Erigon Service.
Apr 08 21:32:30 eth-beacon-archive-mainnet-euc1-2 systemd[1]: erigon.service: Consumed 22h 48min 38.931s CPU time.
Apr 08 21:32:30 eth-beacon-archive-mainnet-euc1-2 systemd[1]: Started Erigon Service.

[yperbasis]

Might be related to https://github.com/erigontech/metarepo-internal-issues/issues/1

[VBulikov]

VBulikov added Imp1 as Importance

[VBulikov]

Update documentation with workaround from https://github.com/erigontech/metarepo-internal-issues/issues/1

[Giulio2002]

I hereby summarise this problem with the latest findings. Please read everything because that is very interesting, I think.

1. gnosis-n1 and sepolia-n1 snapshotter used to crash because of fatal error: runtime: cannot allocate memory (mentioned by @yperbasis in the problem description)

2. @Giulio2002 and @AskAlexSharov requested new machines for sepolia and gnosis (i.e. sepolia-n2 and gnosis-n2) because it was thought that such a problem was caused by some machines hardware trouble. So new machines (sepolia-n2 and gnosis-n2) got created

3. this fatal error: runtime: cannot allocate memory problem occurred in the new sepolia-n2 and gnosis-n2 machines too

4. I initially tweaked the OS kernel as follows:
   vm.overcommit_memory = 1 (it is 0 by default)
   vm.max_map_count = 2097152 (it is 1048576 by default)
   This means that:

• the OS started to allow memory allocations (in this case from erigon) exceeding the maximum amount of virtual memory assigned to applications WITHOUT actually allocating it at once but rather performing such allocation any time it is actually needed (in other terms: it started performing LAZY memory ALLOCATIONS)
• the max amount of virtual memory map areas assigned to applications was doubled from 1048576 to 2097152

5. After doing point 4, I observed that the time interval between subsequent crashes got extended for both sepolia-n2 and gnosis-n2 new machines. instead of 1.5/2 days such interval got extended to 4/5 days.

6. sepolia-n2 and gnosis-n2 still crashed (with a prolonged time interval between subsequent crashes)

7. I eventually tweaked the OS kernel as follows:
   vm.overcommit_memory = 1 (it is 0 by default)
   vm.max_map_count = 8388608 (it is 1048576 by default)
   So:

• the OS is performing LAZY memory ALLOCATIONS for applications whose memory requests exceed the allowed threshold
• the max amount of virtual memory map areas assigned to applications is now eight times as big as the default one: from 1048576 to 8388608 map areas

8. sepolia-n2 and gnosis-n2 haven't crashed yet after performing point 7 (it's now about 3 weeks that point 7 was done).

Originally posted by @michelemodolo in #1

Try this! if it works we will add to README.