Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syncing stucked and can't be gracefully shutdown #24159

Closed
bitrocks opened this issue Dec 27, 2021 · 1 comment
Closed

Syncing stucked and can't be gracefully shutdown #24159

bitrocks opened this issue Dec 27, 2021 · 1 comment
Labels

Comments

@bitrocks
Copy link

bitrocks commented Dec 27, 2021

System information

Geth version: 1.10.14-stable
OS & Version: Linux

Expected behaviour

  1. sync to the latest height
  2. gracefully shutdown when kill with SIGTERM signal

Actual behaviour

  1. the node stopped syncing in block height 13842763, (latest: 13885049), also with warn like Failed to decode block body like I reported in keeps showing Failed to decode block body, error=EOF #24158
INFO [12-26|12:33:56.111] Deep froze chain segment                 blocks=554 elapsed=241.616ms number=13,752,279 hash=2e59b3..f96a84
INFO [12-26|12:33:58.431] Imported new chain segment               blocks=73  txs=17086 mgas=1164.054 elapsed=8.387s    mgasps=138.779 number=13,842,299 hash=342930..98e0ef age=5d23h4m   dirty=2.50GiB
INFO [12-26|12:34:06.553] Imported new chain segment               blocks=71  txs=13764 mgas=1099.929 elapsed=8.122s    mgasps=135.415 number=13,842,370 hash=14483f..2641fd age=5d22h49m  dirty=2.50GiB
INFO [12-26|12:34:14.642] Imported new chain segment               blocks=61  txs=11946 mgas=939.153  elapsed=8.089s    mgasps=116.100 number=13,842,431 hash=f9d338..04b378 age=5d22h34m  dirty=2.50GiB
INFO [12-26|12:34:22.731] Imported new chain segment               blocks=63  txs=11259 mgas=945.075  elapsed=8.088s    mgasps=116.838 number=13,842,494 hash=229347..6716fb age=5d22h23m  dirty=2.50GiB
INFO [12-26|12:34:30.758] Imported new chain segment               blocks=64  txs=11538 mgas=1013.797 elapsed=8.027s    mgasps=126.296 number=13,842,558 hash=8b58b7..6b6467 age=5d22h9m   dirty=2.50GiB
INFO [12-26|12:34:38.848] Imported new chain segment               blocks=71  txs=13084 mgas=1109.530 elapsed=8.089s    mgasps=137.161 number=13,842,629 hash=7050b9..e01c99 age=5d21h55m  dirty=2.50GiB
INFO [12-26|12:34:46.864] Imported new chain segment               blocks=68  txs=12332 mgas=1040.302 elapsed=8.016s    mgasps=129.776 number=13,842,697 hash=7bf296..8d08e2 age=5d21h42m  dirty=2.50GiB
INFO [12-26|12:34:54.922] Imported new chain segment               blocks=60  txs=11309 mgas=929.221  elapsed=8.058s    mgasps=115.316 number=13,842,757 hash=f210aa..664ab5 age=5d21h29m  dirty=2.49GiB
INFO [12-26|12:34:55.780] Imported new chain segment               blocks=6   txs=1647  mgas=118.826  elapsed=858.637ms mgasps=138.389 number=13,842,763 hash=608650..5b1793 age=5d21h26m  dirty=2.50GiB
INFO [12-26|12:34:55.787] Downloader queue stats                   receiptTasks=0 blockTasks=32901 itemSize=71.65KiB  throttle=3659
WARN [12-26|12:34:56.108] Failed to decode block body              block=11,305,880 error=EOF
INFO [12-26|12:44:55.200] New local node record                    seq=17107 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=60493 tcp=30303
WARN [12-26|12:45:13.302] Checkpoint challenge timed out, dropping id=e9b288e40fe09256 conn=dyndial addr=212.102.60.119:30303 type=Geth/v1.10.9-stable/...
INFO [12-26|13:06:51.372] New local node record                    seq=17108 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-26|13:12:04.523] New local node record                    seq=17109 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=60493 tcp=30303
INFO [12-26|13:20:20.199] New local node record                    seq=17110 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-26|13:23:30.618] Writing clean trie cache to disk         path=/data/chain-data/eth/geth/triecache threads=1
INFO [12-26|13:23:30.618] Regenerated local transaction journal    transactions=0 accounts=0
INFO [12-26|13:23:31.912] Persisted the clean trie cache           path=/data/chain-data/eth/geth/triecache elapsed=1.294s
  1. I try to gracefully shut it down with SIGTERM signal, but it shows:
WARN [12-27|03:38:37.882] Already shutting down, interrupt more to panic. times=9
INFO [12-27|03:46:35.320] New local node record                    seq=17209 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|03:52:10.772] New local node record                    seq=17210 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-27|03:54:48.611] New local node record                    seq=17211 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|04:02:42.190] New local node record                    seq=17212 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
WARN [12-27|04:02:57.649] Already shutting down, interrupt more to panic. times=8
INFO [12-27|04:05:15.901] New local node record                    seq=17213 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|04:09:01.191] New local node record                    seq=17214 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303

similar issue #21273 , is it some new kind of deadlock?

Steps to reproduce the behaviour

Backtrace

https://gist.github.com/bitrocks/77db0474aa319839520b3808832574f5

When submitting logs: please submit them as text and not screenshots.

@fjl
Copy link
Contributor

fjl commented Jan 4, 2022

This happens because of double locking of freezer.writeLock in function rawdb.ReadCanonicalBodyRLP.

@fjl fjl removed the status:triage label Jan 4, 2022
holiman added a commit to holiman/go-ethereum that referenced this issue Jan 4, 2022
@fjl fjl closed this as completed in 66a908c Jan 4, 2022
sidhujag pushed a commit to syscoin/go-ethereum that referenced this issue Jan 6, 2022
Fixes ethereum#24159

Co-authored-by: Felix Lange <fjl@twurst.com>
AusIV pushed a commit to NoteGio/go-ethereum that referenced this issue Jan 18, 2022
Fixes ethereum#24159

Co-authored-by: Felix Lange <fjl@twurst.com>
JacekGlen pushed a commit to JacekGlen/go-ethereum that referenced this issue May 26, 2022
Fixes ethereum#24159

Co-authored-by: Felix Lange <fjl@twurst.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants