Preserve orphaned blocks

[michaelsproul]

Description

We currently have no way of preserving orphaned blocks, even though they are useful for block explorers and analytics.

Steps to resolve

• Add a flag to lighthouse bn called something like --keep-orphaned-blocks
• Add a new column to the database for mapping slots to vecs of orphaned block roots (DBColumn::OrphanedBlocks). The blocks themselves can be stored in the hot DB's main block storage, where all other blocks are currently stored.
• In the pruning code that runs on finalization add logic to keep the blocks when --keep-orphaned-blocks is set, storing their block roots under the slot => Vec<Hash256> mapping.
• In the API:
  • Modify /eth/v1/beacon/headers?slot so that it reads block roots from the orphaned block roots table, and then looks them up in the main block table.
  • Work out what to do with /eth/v1/beacon/headers?parent_root (I don't know what we do currently)
  • The orphaned blocks themselves can then be accessed by their root using /eth/v1/beacon/blocks/{root} but not by their slot directly.

Version

Lighthouse v1.4.0

[eklm]

I would like to take this.

[michaelsproul]

Thanks @eklm that would be awesome!

I'm working on some other database changes at the moment which are making me reconsider this part of the design:

The blocks themselves can be stored in the hot DB's main block storage, where all other blocks are currently stored.

I think it might be better to move the orphaned blocks to a separate part of the database. So maybe the column DBColumn::OrphanedBlocks could store values of Vec<SignedBeaconBlock<E>>, or we keep the layer of redirection and add two columns: OrphanedBlockRoots: Slot => Vec<Hash256> and OrphanedBlocks: Hash256 => SignedBeaconBlock<E>.

[eklm]

Hi @michaelsproul! Didn't have a chance to look at this yet, but I now get back to it and investigated the current implementation. So you are suggesting to move the orphaned blocks to different column in the hot db or also to the cold db?

So maybe the column DBColumn::OrphanedBlocks could store values of Vec<SignedBeaconBlock>, or we keep the layer of redirection and add two columns: OrphanedBlockRoots: Slot => Vec and OrphanedBlocks: Hash256 => SignedBeaconBlock.

The second variant looks much better for me (OrphanedBlockRoots: Slot => Vec<Hash256> and OrphanedBlocks: Hash256 => SignedBeaconBlock<E>) as DBColumn::OrphanedBlocks in the first case can grow very large.

I'm also thinking of if we should return the orphaned state in the api if it was requested by the root? I guess yes, right? In order to do that we may go to the corresponding orphaned block first and find the first ancestor which is part of canonical chain and replay the state from there...