Regression: indexing bitcoin mainnet from scratch makes electrs panic #214

Activity

1. EddieHouston commented on May 7, 2026

   

   EddieHouston

   on May 7, 2026

   Last edited by EddieHouston

   Collaborator

   More actions

   Thanks for reporting this and sorry it is causing you troubles. It's possible that
   something is happening with the blk*.dat fetcher in that it may not send blocks
   in chain order as the new single pass indexing seems to require.
   ...more investigation needed to fully root cause and resolve.

   Workaround: add --jsonrpc-import to your invocation:

     cargo run --release --bin electrs -- -vvvv \                                                                                                                                                                                                  
       --daemon-dir=/data/blockchains/BTC/ \
       --db-dir=/data/blockchains/BTC/electrs/ \                                                                                                                                                                                                   
       --network mainnet --address-search --index-unspendables \
       --jsonrpc-import                                                                                                                                                                                                                            
   

   That routes block fetching through bitcoind's JSON-RPC, which delivers blocks in
   chain order and avoids the ordering assumption that f66e7fb relies on. It's
   slightly slower than reading blk files directly, but should let you sync from scratch on
   the latest commit without reverting.

   👍React with 👍1Reacted by Clemens Otto
2. EddieHouston commented on May 7, 2026

   

   EddieHouston

   on May 7, 2026

   Collaborator

   More actions

   btw... the blk file fetcher is likely going away in the future and --jsonrpc-import will become the default.

   👀React with 👀1Reacted by Clemens Otto
3. nikicat commented on Aug 4, 2026

   

   nikicat

   on Aug 4, 2026

   Last edited by nikicat

   More actions

   We hit this on a fresh mainnet sync and root-caused it. Confirming your suspicion: the blk*.dat fetcher is the trigger, and f66e7fb is the regression point.

   Environment: new-index @ ef15bf9 (+1 private electrum-RPC commit, no index changes), bitcoind v31.1 (txindex=1, no prune, node originally synced headers-first), aarch64, fresh empty DB, default fetch (no --jsonrpc-import).

   Symptom: deterministic panic at ~12.5% of initial sync, always in the same region:

   thread 'main' panicked at src/new_index/schema.rs:535:76:
   called `Result::unwrap()` on an `Err` value: Error(Msg("missing txo 99850eb2a92e50b5ede644dca3689a27312625c886190f436c9750b86f971a00:0"))
   

   Last progress line: processing blocks 120014/960999 (12.5%).

   Root cause: before f66e7fb, ordering never mattered: pass 1 (add) wrote the T/O rows for every block before pass 2 (index) performed any lookup_txos(). The single-pass merge made per-batch indexing depend on an implicit invariant — blocks arrive in dependency (height) order — which the blk-file fetcher does not provide. blk*.dat files store blocks in arrival order: headers-first IBD downloads from many peers across a ~1024-block moving window (BLOCK_DOWNLOAD_WINDOW), so blocks land in the files out of height order routinely. Whenever a spending tx is processed in a batch before the batch containing its funding tx, lookup_txos() finds nothing and the unwrap at schema.rs:535 aborts. Any node whose block files were written by a headers-first sync (i.e., every node synced with Core ≥0.10) can reproduce this; exact panic height varies with the node's individual download history.

   Worth noting: #206 (closed as suspected local corruption) failed at height ~119937 with the same signature — that is the same bug, not a corrupt datadir; the reporter's files were merely out of order in the same early region where fast respends were common.

   Fix directions we see, in increasing order of effort:

   1. Make --jsonrpc-import the default (as you suggested) — correct by construction, but slower and it abandons the blk fast path entirely.
   2. Tolerate-and-defer (our preference): let lookup_txos() report missing outpoints instead of panicking; blocks with unresolved spends are skipped — not marked in indexed_blockhashes — while add() still runs for every block. After the main sweep, re-process the deferred blocks; a single retry round is guaranteed to succeed because by then every funding tx has been added. Bounded extra work (only blocks containing out-of-window spends), keeps the single-pass memory/pipelining win, and the existing done-marker crash-recovery machinery already models "added but not indexed".
   3. A height-reordering buffer inside the blk fetcher — fixes ordering for all consumers, but needs unbounded-buffer handling for pathological file layouts.
4. EddieHouston commented on Aug 28, 2026

   

   EddieHouston

   on Aug 28, 2026

   Collaborator

   More actions

   Please see MR #217 which we are seeing significantly speed up initial sync... from taking over a week to now taking about 47 hours.
   Hoping that this lands in new-index in the next week or so.