EIP1559 issue following changes after 17th April 24

[wawrzek]

I noticed a problem with deploying to a docker-compose based POS network (https://github.com/OffchainLabs/eth-pos-devnet) when geth image is newer than the following tag (which was created 2024-04-17T06:37:18.62946929Z):

ethereum/client-go@sha256:0de486c8adf2f10ab0934b46fbe6c7a2c933eafad3fe4d02460e0e4c04154031

Reading release notes for 1.14.0 I thought that maybe it's a matter of ensuring the network is post-merge one, but it has the variable terminalTotalDifficultyPassed: true set in the genesis.json. Also, the support for pre-merge networks was dropped on 7th March, and the POS DevNet worked fine after that date. (I ran it in few places and found a working POS with image mentioned above).

System information

Geth version: 1.14.0
CL client & version: Prysm 5.0.3
OS & Version: Docker
Commit hash : (if develop)

Expected behaviour

Contracts are deployed

Actual behaviour

Failed to estimate EIP1559 fees

forge create Counter  --private-key 0x2e0834786285daccd064ca17f1654f67b4aef298acbb82cef9ec422fb4975622 --rpc-url=http://localhost:8545
[⠊] Compiling...
No files changed, compilation skipped
Error:
Failed to estimate EIP1559 fees. This chain might not support EIP1559, try adding --legacy to your command.

(BTW. --legacy works fine)

Steps to reproduce the behaviour

Set a POS DevNet network (this branch is a good test base: https://github.com/ivy-net/eth-pos-devnet/tree/wn/lock-down-images)
Deploy a contract, e.g. follow Forge simple example: https://blog.lambdaclass.com/ethereum-development-made-easy-with-foundry/

[karalabe]

Do you have any logs from Geth itself? Startup logs, error logs, everything?

[fjl]

Please run Geth with --verbosity=5 and post the output. It should show the RPC calls being made by forge.

[wawrzek]

Please find the geth log from with the verbosity 5.
geth-1.14.3-v5.log

To make it easier to find the right area of the logs, I also attached the logs from the container running the forge script.
eigenlayer-v5.log

This time I ran the 1.14.3 geth.

It's very easy to repeat the run with my docker-compose (https://github.com/ivy-net/iv1/), so please advise if you need to get any further information (any other version or command line option).

[fjl]

Hmm. This log is not super helpful. It would be better to get the log of geth 1.14.4 which is not working. I have a suspicion it might be related to eth_feeHistory but we won't know until we see that log.

[wawrzek]

To make clear, the problem appears in every version from 1.14.0. So I attached the logs from geth 1.14.0 and 1.14.5, so I'm attaching logs from 1.14.0, 1.14.4 and 1.14.5
geth-1.14.0-v5.log
geth-1.14.4-v5.log
geth-1.14.5-v5.log

I have a docker image from the pre 1.14.0 development period ("ethereum/client-go@sha256:0de486c8adf2f10ab0934b46fbe6c7a2c933eafad3fe4d02460e0e4c04154031") which is fine. Logs from there are below.
geth-1.14.0pre-v5.log

I think it might be related to the #29140 , if you have the sha for the docker image before and after the PR I can try to run them and confirm.

[fjl]

So maybe it's a bug in forge? Thanks for providing these logs. Unfortunately, I don't see any issue sticking out in them. Was kind of expecting to see an RPC error or something, but for now, it looks forge just can't handle one of the responses from us. The messages from forge do not tell where the problem is.

[wawrzek]

Not sure if that any use:| Logs from the forge deploying to geth 1.14.0 (with verbosity=5) forge-vvvvv-geth-1.14.0.log

-v
[...]

• 5: Print execution and setup traces for all tests

[wawrzek]

@fjl - I pinpointed the issue to the #29140. First, I build a docker image for the 1ec7af261223d6dad9370ee8263f86347b190bab. It failed. Then I checked out 1ec7af261223d6dad9370ee8263f86347b190bab^. This image was all fine.

I don't know enough about ethereum to even guess if the change in the PR is right, and forge needs to change something to work with their settings, or maybe it is a bug.

I also wonder how the combo (forge+geth) works in other network? It has to be fine. So maybe the problem is with one of genesis block values? How could I compare the one from devnet to the mainnet or holesky?

[fjl]

In the PR you linked, some additional fields were added to the return value of eth_feeHistory. I strongly suspect this to be an issue with the JSON decoding in forge. Their parser might reject unknown fields in the object.

[wawrzek]

I found related: foundry-rs/foundry#8047

[wawrzek]

I looked more carefully at versioning of foundry and their 'latest' image does not have the fix, and indeed it's not the 'latest'. To get the latest, you need to get the 'nightly' build. I grabbed a recent 'nightly' and deployments works fine.

One more question on the geth side. I see no info about this, potentially braking, change, in the release notes for 1.14.0. Have I missed them? (I haven't touched Ethereum for a long time, so maybe that's the case).

[fjl]

We typically do not announce additions of fields in response objects because we believe adding more fields is a backwards-compatible modification. As the protocol changes, new fields will be added. In this particular case, it was added to the spec (ethereum/execution-apis#486) and then implemented by all clients. I think forge is just a bit too strict with their decoding.

Anyway, thanks for following along patiently, and I will close this issue now.