sync issue (`block not accepted`) when syncing in to 2.0 dev1 chain

[spoonincode]

It recovered but shouldn't have had this glitch in the first place right?

dev1huh.txt

[heifner]

Probably worth looking into if we can get it to repeat.
Couple of observations from the logs.
The multiple

info  2025-07-01T21:22:47.710 net-0     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.710 net-2     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.710 net-2     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-0     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-3     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-0     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-1     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-1     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-3     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-0     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35
info  2025-07-01T21:22:47.711 net-3     net_plugin.cpp:2073           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 36 to 1035, fhead 131, froot 35

In the log indicates it is is some odd state. It should not keep requesting the same range like that in quick succession.

The

warn  2025-07-01T21:22:47.718 net-1     net_plugin.cpp:2427           rejected_block       ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] block 698 not accepted, closing connection max violations reached`

Is right before the transition to Savanna. We have a known "issue" where blocks received ahead of the transition can't be processed until the transition completes. Live this is not an issue, but syncing it typically rejects them first and then processes them fine after the transition has completed. Fixing this is likely not worth the effort since it does recover and syncing through Savanna consensus upgrade doesn't happen very often.

[spoonincode]

Why 698 though? Isn't the transition at 132?

warn  2025-07-01T21:22:47.718 net-1     net_plugin.cpp:2427           rejected_block       ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] block 698 not accepted, closing connection max violations reached
info  2025-07-01T21:22:47.718 net-1     net_plugin.cpp:1435           _close               ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] closing
warn  2025-07-01T21:22:47.718 net-1     net_plugin.cpp:2056           request_next_chunk   ] Unable to continue syncing at this time
info  2025-07-01T21:22:47.718 net-1     net_plugin.cpp:1933           set_state            ] old state lib catchup becoming in sync
info  2025-07-01T21:22:47.722 nodeos.20 controller.cpp:1489           transition_to_savann ] Transitioning to savanna, IF Genesis Block 90, IF Critical Block 131
info  2025-07-01T21:22:47.723 nodeos.20 controller.cpp:1534           operator()           ] Transition to instant finality happening after block 131, First IF Proper Block 132

But yeah the repeated requests very sus on their own.

[heifner]

Why 698 though? Isn't the transition at 132?

That failed with a max violations reached which is 13 consecutive blocks rejected where there has to be 2 milliseconds between them to be counted. We now process blocks so quickly that 2 milliseconds probably needs to be revisited.

[heifner]

Verified both issues also in release/1.2 branch.

[heifner]

Verified the multiple requests and the rejected block are both the same issue. Syncing past the Savanna transition.

info  2025-07-02T12:59:03.706 net-0     net_plugin.cpp:2054           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 2 to 1001, fhead 1, froot 1
debug 2025-07-02T12:59:04.046 net-0     net_plugin.cpp:3923           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] rejected block 132 00000084bf3aedf56996b2e9c47a521fa1e8790b934a8895d87e43c67944b15a
debug 2025-07-02T12:59:04.046 net-0     net_plugin.cpp:2411           rejected_block       ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] rejected block 132, sending handshake
info  2025-07-02T12:59:04.047 net-2     net_plugin.cpp:2054           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 15 to 1014, fhead 117, froot 14
debug 2025-07-02T12:59:04.048 net-2     net_plugin.cpp:3923           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] rejected block 133 00000085b9036029fd4a99dbdf16694f170f7a955e50830762933ee88c9c5619
debug 2025-07-02T12:59:04.048 net-2     net_plugin.cpp:2411           rejected_block       ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] rejected block 133, sending handshake
info  2025-07-02T12:59:04.049 net-2     net_plugin.cpp:2054           operator()           ] ["p2p.testnet-1.vaulta.com:3444" - 1 52.71.134.149:3444] requesting range 15 to 1014, fhead 117, froot 14