Bug: "oversized vector allocation" due to large transaction (lots of witness inputs)

[C-Otto]

A recent transaction broke lnd (Lightning Network) and mempool.space (block explorer). Based on reports on Twitter electrs is also affected, but I didn't check myself.

"oversized vector allocation"

btcsuite/btcd#1906

[oliverlj]

I confirm it is broken :

Starting electrs 0.9.4 on aarch64 linux with Config { network: Bitcoin, db_path: "/data/db/bitcoin", daemon_dir: "/data/.bitcoin", daemon_auth: CookieFile("/data/.bitcoin/.cookie"), daemon_rpc_addr: 10.21.21.8:8332, daemon_p2p_addr: 10.21.21.8:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: "Umbrel Electrs (0.9.4-build-2)", args: [] }
[2022-11-01T11:45:01.791Z INFO  electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
[2022-11-01T11:45:01.791Z INFO  electrs::server] serving Electrum RPC on 0.0.0.0:50001
[2022-11-01T11:45:01.957Z INFO  electrs::db] "/data/db/bitcoin": 151 SST files, 37.456491275 GB, 4.696187474 Grows
[2022-11-01T11:45:07.042Z INFO  electrs::chain] loading 761248 headers, tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f
[2022-11-01T11:45:11.850Z INFO  electrs::chain] chain updated: tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f, height=761248
[2022-11-01T11:45:12.293Z INFO  electrs::index] indexing 10 blocks: [761249..761258]
thread 'p2p_loop' panicked at 'invalid message: OversizedVectorAllocation { requested: 12000072, max: 4000000 }', src/p2p.rs:250:70
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2022-11-01T11:45:12.353Z INFO  electrs::db] closing DB at /data/db/bitcoin
Error: electrs failed

Caused by:
    0: sync failed
    1: failed to get block 000000000000000000070742427fa10ec3c66d006160155a704d9f56d090a3ea
    2: receiving on an empty and disconnected channel

[0xB10C]

My electrs 0.9.9 on x86_64 linux seems to be unaffected.

[modl21]

My electrs v0.9.8 seems unaffected as well.

[elkimek]

My electrs v0.9.8 seems unaffected as well.

Same here on Nodl but bricked on another implementation.

[Kixunil]

This could be inside rust-bitcoin but we added a protection recently so maybe not. Will investigate.

[steepdawn974]

Issue confirmed electrs 0.9.7 on x86_64 linux

Nov 01 12:55:38 shallan electrs[3991523]: Starting electrs 0.9.7 on x86_64 linux with Config { network: Bitcoin, db_path: "/mnt/hdd/app-storage/electrs/db/bitcoin", daemon_dir: "/home/electrs/.bitcoin", daemon_auth: UserPass("raspibolt", "<sensitive>"), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 127.0.0.1:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs v0.9.7 - the Electrum Rust Server on your RaspiBlitz", args: [] }
Nov 01 12:55:38 shallan electrs[3991523]: [2022-11-01T12:55:38.681Z INFO  electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
Nov 01 12:55:38 shallan electrs[3991523]: [2022-11-01T12:55:38.681Z INFO  electrs::server] serving Electrum RPC on 0.0.0.0:50001
Nov 01 12:55:38 shallan electrs[3991523]: [2022-11-01T12:55:38.869Z INFO  electrs::db] "/mnt/hdd/app-storage/electrs/db/bitcoin": 150 SST files, 37.595860264 GB, 4.696187475 Grows
Nov 01 12:55:41 shallan electrs[3991523]: [2022-11-01T12:55:41.288Z INFO  electrs::chain] loading 761248 headers, tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f
Nov 01 12:55:43 shallan electrs[3991523]: [2022-11-01T12:55:43.678Z INFO  electrs::chain] chain updated: tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f, height=761248
Nov 01 12:55:43 shallan electrs[3991523]: [2022-11-01T12:55:43.695Z INFO  electrs::index] indexing 17 blocks: [761249..761265]
Nov 01 12:55:43 shallan electrs[3991523]: thread 'p2p_loop' panicked at 'invalid message: OversizedVectorAllocation { requested: 12000072, max: 4000000 }', src/p2p.rs:250:70
Nov 01 12:55:43 shallan electrs[3991523]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Nov 01 12:55:43 shallan electrs[3991523]: [2022-11-01T12:55:43.706Z INFO  electrs::db] closing DB at /mnt/hdd/app-storage/electrs/db/bitcoin
Nov 01 12:55:43 shallan electrs[3991523]: Error: electrs failed
Nov 01 12:55:43 shallan electrs[3991523]: Caused by:
Nov 01 12:55:43 shallan electrs[3991523]:     0: sync failed
Nov 01 12:55:43 shallan electrs[3991523]:     1: failed to get block 000000000000000000070742427fa10ec3c66d006160155a704d9f56d090a3ea
Nov 01 12:55:43 shallan electrs[3991523]:     2: receiving on an empty and disconnected channel
Nov 01 12:55:43 shallan systemd[1]: electrs.service: Main process exited, code=exited, status=1/FAILURE
Nov 01 12:55:43 shallan systemd[1]: electrs.service: Failed with result 'exit-code'.
Nov 01 12:55:43 shallan systemd[1]: electrs.service: Consumed 4.888s CPU time.

[yanascz]

Upgrading electrs to 0.9.9 fixed this for me as well on aarch64.

Nov 01 13:54:26 xyz electrs[1866423]: [2022-11-01T12:54:26.626Z INFO  electrs::server] serving Electrum RPC on 127.0.0.1:50001
Nov 01 13:54:26 xyz electrs[1866423]: [2022-11-01T12:54:26.690Z INFO  electrs::db] "/var/lib/electrs/db/bitcoin": 158 SST files, 37.636174394 GB, 4.696187479 Grows
Nov 01 13:54:30 xyz electrs[1866423]: [2022-11-01T12:54:30.565Z INFO  electrs::chain] loading 761248 headers, tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f
Nov 01 13:54:34 xyz electrs[1866423]: [2022-11-01T12:54:34.225Z INFO  electrs::chain] chain updated: tip=0000000000000000000640e114b9e6272b9688178273e193ef957e02fad1069f, height=761248
Nov 01 13:54:34 xyz electrs[1866423]: [2022-11-01T12:54:34.262Z INFO  electrs::index] indexing 17 blocks: [761249..761265]
Nov 01 13:54:35 xyz electrs[1866423]: [2022-11-01T12:54:35.049Z INFO  electrs::chain] chain updated: tip=000000000000000000051ca14f3ea01d5672b44d9ef3c95587afc0e3bd39aad6, height=761265
Nov 01 13:58:09 xyz electrs[1866423]: [2022-11-01T12:58:09.432Z INFO  electrs::index] indexing 1 blocks: [761266..761266]
Nov 01 13:58:09 xyz electrs[1866423]: [2022-11-01T12:58:09.504Z INFO  electrs::chain] chain updated: tip=00000000000000000003ac39882e5127ce163ee6bdca75d4b9bcf806c83afe56, height=761266

[Kixunil]

Oh, it looks like the DoS protection in rust-bitcoin is causing this. Cc @RCasatta

[NicolasDorier]

I bumped to 0.15.4, it fixed the issue. (our own build) wrong issue, I panic-commented.

[Kixunil]

@NicolasDorier you're replying to electrs issue, I guess you wanted some LND-related issue. :)