'tx iteration error' after upgrading to 1.10.0

[adamkoch]

System information

Geth version: v1.9.25, upgrading to v1.10.0
OS & Version: Linux (Ubuntu 20.04.2)
Using docker and docker-compose.

Expected behaviour

No errors in logs and chain syncing fine

Actual behaviour

Lots of repeated errors in logs, showing:

tx iteration error                       error="unexpected EOF",

Steps to reproduce the behaviour

Not sure, but in my case, using docker engine and upgrading from a fully sync'd 1.9.25 geth version to 1.10.0

Logs

INFO [03-07|23:18:08.725] Unindexed transactions                   blocks=0 txs=0   tail=9644286 elapsed="692.231µs",
INFO [03-07|23:18:12.475] Aborting state snapshot generation       root="c3b625…8ad7e3" at="003ca1…dcd450" accounts=111699 slots=115478 storage=14.26MiB elapsed=1m26.492s   eta=25h56m44.563s,
INFO [03-07|23:18:12.475] Resuming state snapshot generation       root="e8cc64…1e191c" at="003ca1…dcd450" accounts=111699 slots=115478 storage=14.26MiB elapsed=1m26.492s   eta=25h56m44.563s,
INFO [03-07|23:18:12.483] Chain reorg detected                     number=11994284 hash="d733f7…fffc3e" drop=1 dropfrom="a544f0…466d48" add=2 addfrom="03ac81…23b944",
INFO [03-07|23:18:12.491] Imported new chain segment               blocks=1 txs=136 mgas=12.401 elapsed=231.060ms   mgasps=53.671  number=11994286 hash="03ac81…23b944" dirty=34.56MiB,
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:12.492] tx iteration error                       error="unexpected EOF",
INFO [03-07|23:18:12.492] Unindexed transactions                   blocks=0 txs=0   tail=9644287 elapsed=1.469ms,
INFO [03-07|23:18:12.806] Deep froze chain segment                 blocks=14 elapsed=55.320ms    number=11904286 hash="f10952…3d4c1e",
INFO [03-07|23:18:17.541] Looking for peers                        peercount=11 tried=5  static=0,
INFO [03-07|23:18:23.440] Generating state snapshot                root="e8cc64…1e191c" in="0042a5…807326" at="06d932…74052e" accounts=122849 slots=138896 storage=16.46MiB elapsed=1m37.457s   eta=26h35m35.461s,
INFO [03-07|23:18:27.542] Looking for peers                        peercount=11 tried=4  static=0,
INFO [03-07|23:18:31.742] Generating state snapshot                root="e8cc64…1e191c" in="0042a5…807326" at="0ee060…ae5ff7" accounts=122849 slots=163055 storage=18.15MiB elapsed=1m45.758s   eta=28h51m29.744s,
INFO [03-07|23:18:35.768] Aborting state snapshot generation       root="e8cc64…1e191c" in="0042a5…807326" at="1305ca…d03bb5" accounts=122849 slots=175390 storage=19.01MiB elapsed=1m49.785s   eta=29h57m25.568s,
INFO [03-07|23:18:35.769] Resuming state snapshot generation       root="7efc07…7eda3d" in="0042a5…807326" at="1305ca…d03bb5" accounts=122849 slots=175390 storage=19.01MiB elapsed=1m49.785s   eta=29h57m25.568s,
INFO [03-07|23:18:35.770] Imported new chain segment               blocks=1  txs=249 mgas=12.498 elapsed=265.570ms   mgasps=47.060  number=11994287 hash="b3fe18…30bc14" dirty=36.39MiB,
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",
WARN [03-07|23:18:35.771] tx iteration error                       error="unexpected EOF",

[adamkoch]

Same issues when trying v1.10.1

[holiman]

Can you do eth.getBlockByNumber(9644287, true) and eth.getBlockByNumber(9644286, true)?

[adamkoch]

I have downgraded my geth back to v1.9.25 for now to keep it running. But I can upgrade it again if needed.

But I'm not sure how to run those commands as I run geth from a docker container. Any pointers to docs?

[adamkoch]

I tried upgrading to 'latest' in docker hub as I saw it contained the change from above.

The errors changed but it still doesn't seem to be working:

INFO [03-22|08:08:27.877] Unindexed transactions                   blocks=0 txs=0   tail=9737441 elapsed=1.698ms
INFO [03-22|08:08:34.804] Generating state snapshot                root="3d34a9…6caecb" in="002afd…0ebda1" at="6c722c…57985f" accounts=80725 slots=70241 storage=8.85MiB   elapsed=1m17.898s   eta=32h57m51.704s
WARN [03-22|08:08:37.297] Checkpoint challenge timed out, dropping id=cf34821586f4db46 conn=dyndial addr=47.107.113.245:30303 type=Geth/v1.9.24-stable-...
ERROR[03-22|08:08:37.297] Ethereum peer removal failed             peer=cf348215 err="peer not registered"
INFO [03-22|08:08:43.461] Generating state snapshot                root="3d34a9…6caecb" in="002fe4…5fc15e" at="217cbe…fe4520" accounts=89911 slots=84147 storage=10.39MiB  elapsed=1m26.554s   eta=32h52m36.534s
INFO [03-22|08:08:51.607] Generating state snapshot                root="3d34a9…6caecb" in="002fe4…5fc15e" at="df44b0…1c45c8" accounts=89911 slots=105899 storage=12.42MiB  elapsed=1m34.700s   eta=35h58m15.49s
INFO [03-22|08:08:58.360] Imported new chain segment               blocks=1 txs=193 mgas=12.526 elapsed=541.784ms   mgasps=23.120  number=12087441 hash="3c07ed…70ac97" dirty=13.27MiB
WARN [03-22|08:08:58.360] Failed to decode block body              block=1 error=EOF
WARN [03-22|08:08:58.360] Failed to decode block body              block=2 error=EOF
WARN [03-22|08:08:58.360] Failed to decode block body              block=3 error=EOF
WARN [03-22|08:08:58.360] Failed to decode block body              block=4 error=EOF
WARN [03-22|08:08:58.360] Failed to decode block body              block=5 error=EOF
WARN [03-22|08:08:58.360] Failed to decode block body              block=6 error=EOF
WARN [03-22|08:08:58.361] Failed to decode block body              block=8 error=EOF
WARN [03-22|08:08:58.361] Failed to decode block body              block=7 error=EOF
INFO [03-22|08:08:58.361] Unindexed transactions                   blocks=0 txs=0   tail=9737442 elapsed="839.685µs"
INFO [03-22|08:08:59.470] Imported new chain segment               blocks=1 txs=189 mgas=12.507 elapsed=85.701ms    mgasps=145.940 number=12087441 hash="3a78ce…d5d436" dirty=15.01MiB
INFO [03-22|08:08:59.623] Generating state snapshot                root="3d34a9…6caecb" at="003878…fab8aa" accounts=105840 slots=114269 storage=13.88MiB  elapsed=1m42.717s   eta=33h5m7.634s
INFO [03-22|08:09:00.706] Aborting state snapshot generation       root="3d34a9…6caecb" at="0039e2…47811d" accounts=108468 slots=114893 storage=14.05MiB  elapsed=1m43.799s   eta=32h37m0.28s
INFO [03-22|08:09:00.706] Resuming state snapshot generation       root="a81f42…bce9b8" at="0039e2…47811d" accounts=108468 slots=114893 storage=14.05MiB  elapsed=1m43.799s   eta=32h37m0.28s
INFO [03-22|08:09:00.709] Chain reorg detected                     number=12087440 hash="0cc476…f1ee9d" drop=1 dropfrom="3c07ed…70ac97" add=2 addfrom="fb6fae…b67b17"

I decided to just clear my geth data dir and let it re-sync the chain again. I went back to stable (v1.10.1) and it seems to be sync'ing fine after clearing my geth data.

[ligi]

Can you provide logs that go a bit more into the history - this could give us some more context to look into the problem.

[karalabe]

One wild guess is that your data files in the ancient folder got empty somehow, but no idea how that could have happened. Any ideas?

[adamkoch]

My Geth is no longer stuck as I decided to just clear all data and let it re-sync from scratch which worked fine. For others that may run into this, that is probably your best bet.

In terms of history, there was nothing particularly interesting about my install:

• I run in Docker on Ubuntu 20.04.2
• It had been running smoothly for several months
• When I saw the new release I tried upgrading by updating the version and recreating the container, when I saw the errors I downgraded again, recreated the container and all was well

Here is my docker-compose in case anything stands out:

  geth:
    image: ethereum/client-go:${GETH_DOCKER_TAG}
    container_name: geth
    restart: always
    hostname: geth
    command: --http --http.port=8545 --http.addr=0.0.0.0 --http.vhosts="*" --allow-insecure-unlock --http.api="eth,net,web3,personal" --nousb --maxpeers 15 --nat none
    stop_grace_period: 1m
    ports:
      - 30303:30303/tcp # P2P port (external)
      - 30303:30303/udp # P2P port (external)
    expose:
      - 8545 # RPC
    volumes:
      - ./data/geth:/root/.ethereum

You are welcome to close this issue if I was the only one who ran into this as I'm all set now.