core, eth: improve delivery speed on header requests

[holiman]

This PR reduces the amount of work we do when answering header queries, e.g. when a peer is syncing from us.

• For some items, e.g block bodies, when we read the rlp-data from database, we plug it directly into the response package. We didn't do that for headers, but instead read headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it in RLP-form as much as possible.
• When a node is syncing from us, it typically requests 192 contiguous headers. On master it has the following effect:

1. For headers not in ancient: 2 db lookups. One for translating hash->number (even though the request is by number), and another for reading by hash (this latter one is sometimes cached).
2. For headers in ancient: 1 file lookup/syscall for translating hash->number (even though the request is by number), and another for reading the header itself. After this, it also performes a hashing of the header, to ensure that the hash is what it expected.
   In this PR, I instead move the logic for "give me a sequence of blocks" into the lower layers, where the database can determine how and what to read from leveldb and/or ancients.

There are basically four types of requests; three of them are improved this way. The fourth, by hash going backwards, is more tricky to optimize. However, since we know that the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash lookups.

The gapped collection can be optimized similarly, as a follow-up, at least in three out of four cases.

[holiman]

This may actually make the performance a bit worse, in the case that a peer is 3-4 blocks behind us, and wants to catch up: requesting a few headers close to our tip (or beyond). Previously, we would have the headers in object-form in a header cache, whereas this PR makes them read from db. I'll look into it

[holiman]

This may actually make the performance a bit worse, in the case that a peer is 3-4 blocks behind us, and wants to catch up:

Pushed a commit to use the headerchain cache better.

[holiman]

Triage discussion: probably safe, but we should hold with this until after the London release.

[holiman]

TODO (not in this PR), add basefee check to sanityCheck

[holiman]

Rebased on top of the eth/66 changes that got merged. This still does not make use of the faster ancient accessors though

[holiman]

Rebased again. Also, tests are added to check the correctness of the split leveldb/ancient loader

[holiman]

This is now rebased on top of #23566, which should go in first

[holiman]

Rebased (Part V)

[holiman]

I tested this a bit on a live node. On my node, I made it

• When the 'new' code path was requested, it did both a old-style lookup and a new-style lookup
• When doing so, it randomized: sometimes doing old style first, sometimes doing new-style first,
• After pulling both answers, it compared the byte output of the two, to see if they matched.

Example:

INFO [11-30|11:28:29.065] Served block headers                     newtime=1.877586ms  oldtime=8.107161ms  order="new version first" reverse=false skip=0 amount=192 origin="{Hash:0x0000000000000000000000000000000000000000000000000000000000000000 Number:8166928}"

I then collected some stats about the performance.

// Old version first
// Total old: 1.420655265s, Total new: 131.683183ms, count 263, avg old: 5.401731ms, avg new: 500.696µs
// New version first
// Total old: 1.440066122s, Total new: 116.201786ms, count 253, avg old: 5.69196ms, avg new: 459.295µs

It doesn't matter a whole lot which handler was first, for serving 192 headers on an intel NUC with SSD, the current handler takes ~5.5ms, the one in this PR takes about 0.5ms, so one order of magnitude faster.

Also, no difference in the two responses have been found so far

[holiman]

After letting it run for a bit longer:

Total old: 7.335648596s, Total new: 709.078148ms, count 1208, avg old: 6.072556ms, avg new: 586.985µs