New overlays feature for doing ad-hoc simulations of existing contracts with modified bytecode

[crebsy]

In this PR I'm introducing so-called overlays for doing ad-hoc state simulations of contracts which replay the state with a modified bytecode for any given block range.

The new feature adds two new RPC methods: overlay_callConstructor and overlay_getLogs

I've also added a few bench tests and a new postman collection which checks both new methods for the WETH contract in original and modified version.

Motivation

Overlays allow you to add your custom logic to already deployed contracts and simulate on top of them. With overlays you can create new view functions, modify existing ones, change field visibility, emit new events and query the historical data of any contract with your modified source code.

Similar commercial products exist in the space and I believe making this feature opensource in erigon will make it easier for everyone to tinker with it and build new cool things on top of it ✨

Usage

1. compile your contract to obtain the new deployment bytecode, append the constructor args at the end, if any
2. call overlay_callConstructor with the result from 1) to receive the runtime bytecode
3. call overlay_getLogs to simulate new logs, passing 2) as state overrides (example)
4. call eth_call to simulate new view function calls passing 2) as state overrides

Ad-hoc query performance & historical back-filling

The current implementation for ad-hoc range queries of overlay_getLogs is optimized so that irrelevant blocks are skipped entirely from the simulation context. This can lead to very fast results of a few minutes or even seconds for simulating the entire historical logs of a contract which makes any infra for back-filling redundant. However, such shortcuts are only effectively working for contracts which are not touching too many blocks of the chain's history. So depending on the contract you're trying to patch, you still might need to create a custom back-filling script which simulates the new logs with e.g. 1000 or 10000 block ranges.

As a reference for a "busy" contract the simulation rate for the Uniswap V2 router was ~ 280 blocks/s on a host with 24 cores. So while small block ranges returned relatively quickly, the simulation of the entire historical logs for this contract took a few hours on a single node.

I'm not collecting any perf metrics yet for this but this could be easily added in the future for tracking better implementations and insights about contract <-> block distribution.

Configuration

1. Enable the ots,overlay api namespaces by adding it to the --http.api flag

2. The implementation of overlay_getLogs is parallelized so it's good practice to increase the --db.read.concurrency to a higher value, e.g. 256 so other erigon goroutines are not getting blocked.

Supported networks

Tested with ethereum and polygon nodes

[crebsy]

Fixed CREATE2 and added some API docs to my PR:
https://github.com/crebsy/erigon/tree/devel/turbo/jsonrpc/overlay

[crebsy]

rebased to current devel d5502a0

[Giulio2002]

are these overlays methods in a spec somewhere?

[Giulio2002]

uhm, shouldn't the requester or service running the node decide the timeout? maybe worth putting under a flag at least?

[crebsy]

good idea, added two new flags for the outer call of overlay_getLogs and the inner call that is replaying a single block.

[Giulio2002]

extra note, please if you do the flag make it a Duration Flag so we don't have this.

[crebsy]

are these overlays methods in a spec somewhere?

Thanks for your review! I've added a few payload examples to my README. overlay_getLogs has the same interface as eth_getLogs plus the state overrides where you send the custom bytecode.

[crebsy]

rebased to devel 4369d4e

[crebsy]

@Giulio2002
The two new cli flags are called:
--rpc.overlay.getlogstimeout (default: 5m)
--rpc.overlay.replayblocktimeout (default: 10s)

Here's an initial README showing how overlays could be used: https://github.com/crebsy/erigon/blob/devel/turbo/jsonrpc/overlay/README.md

Is there anything else I could add ?

[banteg]

i tried it out and overlay_getLogs worked amazing for me

[LukaszRozmej]

Not sure if this is right direction when eth_simulate is being developed in official spec and soon will be released in first clients: ethereum/execution-apis#484
Feel free to join Mondays calls (in EF calendar) to discuss.

[OlegJakushkin]

Dear colleagues, I'm excited to introduce a powerful feature we've been working on-eth_simulate (also known as eth_multicall). This feature, which you can explore here, is now available in prototype form for both Nethermind and Geth, and it's packed with benefits that I believe will pique your interest.

in eth_simulate, we can

• send a block with updated contracts code (so, say you can add events emission or update functions behaviour)
• rerun any blocks on top (if needed, re-run each new block in a loop)

So, in other words, it looks like if you run eth_simulate with each new block data in a loop, you'll get a similar behaviour as with this PR.

Nice features we also have:

• We can do more than one contract
• Mock any accounts and transactions (add funds, add txs)
• We allow overloading of precompiles (e. g. ecRecover)

See other examples and potential use cases in @KillariDev post here

Please come to our weekly discussions by Joininig our Zoom Meeting
https://us02web.zoom.us/j/83932094353?pwd=V2o4NG4yQy9MZFJGY2FKdGU1OFdaZz09

Let's join forces and create a more standardized and useful feature for all. Your insights and expertise are invaluable, and I look forward to seeing you at our weekly discussions. Together, we can make a great difference!

[banteg]

i see this as complementary to eth_simulate. the overlay_getLogs method allows you to replay eth_getLogs with a modified code in a single rpc request. with eth_simulate, i would first need to fetch the original logs, remember the original calls, and then construct what possibly could be a massive payload that explicitly states every log-producing call and preceding calls in their respective blocks that might've affected the state. other than this, i agree with the importance of standardizing this.

[Giulio2002]

I have to kind of agree with Banteg on this one; while I do think standardizing is a good thing, I believe that eth_simulate is not meant for this specific use-case or would be a bad duct-taped solution.

[Giulio2002]

I, just want to point out, that while we merged this, we may or may not remove this endpoint in the future, if we either find it difficult to maintain or if as @LukaszRozmej said - eth_simulate ends up being more usable for this specific usecase. We are not too keen on custom endpoints but it could be a worthwhile experiment to integrate stuff built by the community with a somewhat conservative mindset. since there seems to be great interest in such a feature, we made an exception