Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: ERROR client: Banning chunk producer for producing invalid chunk chunk_producer=AccountId #12264

Open
msobh13 opened this issue Oct 18, 2024 · 5 comments
Open

Bug: ERROR client: Banning chunk producer for producing invalid chunk chunk_producer=AccountId #12264

msobh13 opened this issue Oct 18, 2024 · 5 comments
Assignees
@telezhnaya
Labels
community Issues created by community investigation required

Comments

@msobh13
Copy link

msobh13 commented Oct 18, 2024

Contact Details

No response

Which network are you using?

mainnet

What happened?

neard (release 2.2.1) (build 2.2.1) (rustc 1.79.0) (protocol 71) (db 40)features: [default, json_rpc, rosetta_rpc]
snapshot used 2024-10-17T23:00:23Z
AMD Ryzen 9 5950X 16-Core Processor . and 64G RAM . NVMe

Mainnet node was downloading blocks noormaly with avg seepd 1.5bps and suddeny i got attached err
near01

ERROR client: Banning chunk producer for producing invalid chunk chunk_producer=AccountId("twinstake.poolv1.near") epoch_id=EpochId(85aoSCmxQ3NqfaguV2X2VSWgtfc3KvJ5nL249svEyPik) chunk_hash=ChunkHash(B5p8QQTFmFW7mD4zdugvwXLTGEmT1QNJ27v2c8HHjUNv)

check the img for more info .

then node stopped downliading blocks for hours

Relevant log output

2024-10-18T12:33:13.887376Z  WARN chain: Failed to validate chunk extra err=InvalidStateRoot prev_block_hash=8xNc8ktKQNEJmc1bHbUYNnz7PMsBYzqsQx5AZqZHhJs block_hash=GKxS2vRrF44pQLCWT3GhsyveUKijsBnX6rM9i2ihhHRL shard_id=5 prev_chunk_height_included=130607511 prev_chunk_extra=V3(ChunkExtraV3 { state_root: EQGZydGjGiZGSaNgH7hX7kfWBubhj4UsBsoVenZzWMkB, outcome_root: 3z9Log6qYoixqaDMT4df9QQn3KoW4iqw4u3QigfF2cpe, validator_proposals: [], gas_used: 9528082444124, gas_limit: 1000000000000000, balance_burnt: 773528463466200000000, congestion_info: V1(CongestionInfoV1 { delayed_receipts_gas: 0, buffered_receipts_gas: 0, receipt_bytes: 0, allowed_shard: 2 }) }) chunk_header=V3(ShardChunkHeaderV3 { inner: V3(ShardChunkHeaderInnerV3 { prev_block_hash: 8xNc8ktKQNEJmc1bHbUYNnz7PMsBYzqsQx5AZqZHhJs, prev_state_root: Fy9sSLp6acj8LcQYX8fKzvrnNYy6LvXdBAndei4HHHQW, prev_outcome_root: 3z9Log6qYoixqaDMT4df9QQn3KoW4iqw4u3QigfF2cpe, encoded_merkle_root: 6rSbz2b2j7a7S6FPnDMwFej4bFH8ofBL1LFr7P3MX2Mr, encoded_length: 2348, height_created: 130607512, shard_id: 5, prev_gas_used: 9528082444124, gas_limit: 1000000000000000, prev_balance_burnt: 773528463466200000000, prev_outgoing_receipts_root: C7hHcsL3yM2QFkrJmLPbsqD429P4hSbE32Exp4Q3TiBJ, tx_root: 8itV8xMGgJEvZJifzKCEkM8VQbmSZnnPFZnbWjEYrnkD, prev_validator_proposals: [], congestion_info: V1(CongestionInfoV1 { delayed_receipts_gas: 0, buffered_receipts_gas: 0, receipt_bytes: 0, allowed_shard: 2 }) }), height_included: 130607512, signature: ed25519:4MX2UhNfmDDBrSeaidVuNVsNk7paFBTa72SHbvvWHAzsKz4gkw4wkMo8AeLNQdsN1UkgQZHURzvgZMDa5RsScQxJ, hash: ChunkHash(7FM3tt33y3uCSEtzCXKCuXxQjDEFmAaF2BeSLtvHuaxx) })
2024-10-18T12:33:13.891257Z ERROR client: Banning chunk producer for producing invalid chunk chunk_producer=AccountId("twinstake.poolv1.near") epoch_id=EpochId(85aoSCmxQ3NqfaguV2X2VSWgtfc3KvJ5nL249svEyPik) chunk_hash=ChunkHash(B5p8QQTFmFW7mD4zdugvwXLTGEmT1QNJ27v2c8HHjUNv)
2024-10-18T12:33:13.933559Z  INFO stats: #130607511 Downloading blocks 43.81% (24492 left; at 130607511) 62 peers ⬇ 10.8 MB/s ⬆ 7.63 MB/s 1.95 bps 681 Tgas/s CPU: 290%, Mem: 12.1 GB
2024-10-18T12:33:23.908705Z  INFO stats: #130607511 Downloading blocks 43.80% (24500 left; at 130607511) 62 peers ⬇ 10.6 MB/s ⬆ 7.83 MB/s 0.00 bps 0 gas/s CPU: 197%, Mem: 11.7 GB
2024-10-18T12:33:24.762157Z ERROR network: Failed to store connection attempt. peer_info=PeerInfo { id: ed25519:H8L2fG8DSJFq6igZHfZvgbQiR4Y8PiDgexBuX4tP4fMz, addr: Some(150.136.181.250:24567), account_id: None }
2024-10-18T12:33:33.910399Z  INFO stats: #130607511 Downloading blocks 43.80% (24508 left; at 130607511) 62 peers ⬇ 10.4 MB/s ⬆ 7.26 MB/s 0.00 bps 0 gas/s CPU: 184%, Mem: 11.4 GB
2024-10-18T12:33:43.910403Z  INFO stats: #130607511 Downloading blocks 43.79% (24515 left; at 130607511) 62 peers ⬇ 10.0 MB/s ⬆ 6.71 MB/s 0.00 bps 0 gas/s CPU: 180%, Mem: 11.4 GB
2024-10-18T12:33:53.914757Z  INFO stats: #130607511 Downloading blocks 43.78% (24523 left; at 130607511) 62 peers ⬇ 9.59 MB/s ⬆ 6.50 MB/s 0.00 bps 0 gas/s CPU: 195%, Mem: 11.3 GB
2024-10-18T12:34:03.914861Z  INFO stats: #130607511 Downloading blocks 43.77% (24530 left; at 130607511) 62 peers ⬇ 8.97 MB/s ⬆ 6.31 MB/s 0.00 bps 0 gas/s CPU: 187%, Mem: 11.4 GB
2024-10-18T12:34:05.425526Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425545Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425548Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425550Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425552Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425555Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425556Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425559Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425562Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425564Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425567Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425569Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425571Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425573Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425793Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:05.425799Z  INFO near_network::peer_manager::connection: peer ed25519:FrbiAcqoV4ZrGXaSgf2faXgT6BSa3jRG1NWYvns8Kyho disconnected, while sending SyncSnapshotHosts
2024-10-18T12:34:13.914905Z  INFO stats: #130607511 Downloading blocks 43.77% (24538 left; at 130607511) 61 peers ⬇ 8.79 MB/s ⬆ 6.78 MB/s 0.00 bps 0 gas/s CPU: 201%, Mem: 11.4 GB
2024-10-18T12:34:23.916643Z  INFO stats: #130607511 Downloading blocks 43.76% (24547 left; at 130607511) 61 peers ⬇ 8.68 MB/s ⬆ 6.85 MB/s 0.00 bps 0 gas/s CPU: 210%, Mem: 11.4 GB
2024-10-18T12:34:33.925078Z  INFO stats: #130607511 Downloading blocks 43.75% (24554 left; at 130607511) 62 peers ⬇ 8.71 MB/s ⬆ 7.18 MB/s 0.00 bps 0 gas/s CPU: 199%, Mem: 11.4 GB
2024-10-18T12:34:43.918535Z  INFO stats: #130607511 Downloading blocks 43.74% (24562 left; at 130607511) 62 peers ⬇ 8.76 MB/s ⬆ 7.41 MB/s 0.00 bps 0 gas/s CPU: 192%, Mem: 11.5 GB
2024-10-18T12:34:53.920354Z  INFO stats: #130607511 Downloading blocks 43.73% (24570 left; at 130607511) 62 peers ⬇ 8.74 MB/s ⬆ 7.59 MB/s 0.00 bps 0 gas/s CPU: 186%, Mem: 11.4 GB
2024-10-18T12:35:03.923096Z  INFO stats: #130607511 Downloading blocks 43.73% (24578 left; at 130607511) 62 peers ⬇ 8.81 MB/s ⬆ 7.44 MB/s 0.00 bps 0 gas/s CPU: 185%, Mem: 11.4 GB
2024-10-18T12:35:13.930524Z  INFO stats: #130607511 Downloading blocks 43.72% (24586 left; at 130607511) 62 peers ⬇ 8.83 MB/s ⬆ 6.23 MB/s 0.00 bps 0 gas/s CPU: 204%, Mem: 11.4 GB
2024-10-18T12:35:23.921941Z  INFO stats: #130607511 Downloading blocks 43.71% (24595 left; at 130607511) 62 peers ⬇ 8.98 MB/s ⬆ 5.34 MB/s 0.00 bps 0 gas/s CPU: 205%, Mem: 11.4 GB
2024-10-18T12:35:33.923969Z  INFO stats: #130607511 Downloading blocks 43.70% (24603 left; at 130607511) 63 peers ⬇ 9.04 MB/s ⬆ 4.88 MB/s 0.00 bps 0 gas/s CPU: 206%, Mem: 11.4 GB
2024-10-18T12:35:43.933057Z  INFO stats: #130607511 Downloading blocks 43.69% (24611 left; at 130607511) 63 peers ⬇ 9.06 MB/s ⬆ 4.71 MB/s 0.00 bps 0 gas/s CPU: 209%, Mem: 11.5 GB
@msobh13 msobh13 added community Issues created by community investigation required labels Oct 18, 2024
@msobh13
Copy link
Author

msobh13 commented Oct 18, 2024

Update .. This is fresh installed server i mean not upgraded from previous near ... new snapshot .. also I have near testnet node running on same machine without issue

@walnut-the-cat
Copy link
Contributor

@msobh13 , could you share which snapshot you used?

@msobh13
Copy link
Author

msobh13 commented Oct 18, 2024

snapshot used 2024-10-17T23:00:23Z

@marcelo-gonzalez
Copy link
Contributor

@msobh13 If you'd like, it would probably be helpful for debugging this further if you took a snapshot of the data directory. Up front, it won't take up any more disk space to do so (as long as $SNAPSHOT_DIR below is in the same filesystem as $NEAR_HOME_DIR), since the snapshot would just be making hard links, and wouldn't copy things. You could do so with this command: neard --home $NEAR_HOME_DIR database make-snapshot --destination $SNAPSHOT_DIR. Up to you of course, but we would appreciate the help in debugging it if you wouldn't mind.

Then later, for actually getting your node working again, you should hopefully be able to get back online by downloading the next new public snapshot and restarting your node

@msobh13
Copy link
Author

msobh13 commented Oct 18, 2024

I am really sorry I didnot see your reply till later while I already deleted the current data and started downloading today snapshot . I will keep this open till i see what happen with today snapshot . I will update you again in 15 hours

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@telezhnaya telezhnaya

Labels
community Issues created by community investigation required
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@telezhnaya @marcelo-gonzalez @msobh13 @walnut-the-cat