Test Failure: `lib_advance_if_test`

[greg7mdp]

https://github.com/AntelopeIO/spring/actions/runs/8728356303/job/23948345171

[heifner]

Very interesting test failure. The nodes understanding of LIB (213) is ahead of its current understanding of HEAD (148). As it is applying blocks it has switched to a different fork that does not have a block that will become LIB. Although it is likely that the node will switch back the node doesn't know that yet.

Currently log_irreversible calls forkdb.fetch_branch( irreversible_block_id, new_lib_num) and moves LIB if the block has been validated. Instead, we need to fetch the branch from irreversible_block_id to HEAD since blocks could be validated but not be on this nodes understanding of best HEAD.

debug 2024-04-17T20:51:06.408 net-2     controller.cpp:4417           set_if_irreversible_ ] set irreversible block 148: 0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44, old 147: 00000093ab07f33365dea233f310bca8dba0917259ac60e88a9e7b2d043fc742
...
debug 2024-04-17T20:51:06.726 net-2     controller.cpp:4417           set_if_irreversible_ ] set irreversible block 213: 000000d5a766d19c52abb83c1dc26b9bc22b959b4b20a41bb9e6d3a4b3a9007e, old 212: 000000d4cd805f8daf2a272671f9e381b0ca99196a6897687171b5d35cb56609
debug 2024-04-17T20:51:06.726 net-2     net_plugin.cpp:3830           operator()           ] validated block header, broadcasting immediately, connection - 4, blk num = 216, id = 000000d87db9c060fc5908230de92538cd5f979ac92930a0822588c16b07db09
debug 2024-04-17T20:51:06.726 net-2     net_plugin.cpp:3830           operator()           ] validated block header, broadcasting immediately, connection - 5, blk num = 161, id = 000000a18b375d5bfff15d77276977384626398b0849afc77b5332f7210fa484
...
info  2024-04-17T20:51:06.841 nodeos    controller.cpp:3379           log_applied          ] Received block 0bd5dfb6da178186... #148 @ 2024-04-17T20:50:42.000 signed by defproducerd [trxs: 1, lib: 145, net: 144, cpu: 252, elapsed: 447035, time: 894421, latency: 24841 ms]
info  2024-04-17T20:51:06.841 nodeos    controller.cpp:4145           operator()           ] successfully switched fork to new head 000000940bd5dfb6da1781860b2c689f4fef99f5593661b0877d691d69613c37, removed {148:0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44}, applied {148:000000940bd5dfb6da1781860b2c689f4fef99f5593661b0877d691d69613c37}
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:4039           on_irreversible_bloc ] on_irreversible_block, blk num = 146, id = 00000092b716dbfb03a7b814a24f8af973a77f85640e099c3a751885ad3fba6d
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:3263           update_chain_info    ] updating chain info lib 146, fork 148
debug 2024-04-17T20:51:06.841 trace-mx  trace_api_plugin.cpp:155      operator()           ] Waking up to handle lib: 146
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:4039           on_irreversible_bloc ] on_irreversible_block, blk num = 147, id = 00000093ab07f33365dea233f310bca8dba0917259ac60e88a9e7b2d043fc742
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:3263           update_chain_info    ] updating chain info lib 147, fork 148
debug 2024-04-17T20:51:06.841 trace-mx  trace_api_plugin.cpp:155      operator()           ] Waking up to handle lib: 147
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:4039           on_irreversible_bloc ] on_irreversible_block, blk num = 148, id = 0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44
debug 2024-04-17T20:51:06.841 nodeos    net_plugin.cpp:3263           update_chain_info    ] updating chain info lib 148, fork 148
debug 2024-04-17T20:51:06.841 trace-mx  trace_api_plugin.cpp:155      operator()           ] Waking up to handle lib: 148
warn  2024-04-17T20:51:06.841 nodeos    controller.cpp:3954           push_block           ] 3020000 fork_database_exception: Fork database exception
removing the block and its descendants would remove the current head block
    {}
    nodeos  fork_database.cpp:597 remove_impl

error 2024-04-17T20:51:06.842 nodeos    producer_plugin.cpp:737       on_incoming_block    ] Cannot recover from 3020000 fork_database_exception: Fork database exception
removing the block and its descendants would remove the current head block
    {}
    nodeos  fork_database.cpp:597 remove_impl
rethrow
    {}
    nodeos  controller.cpp:3954 push_block
. Shutting down.