I tested after merging https://github.com/ledgerwatch/erigon/pull/10468

[mh0lt]

          I tested after merging https://github.com/ledgerwatch/erigon/pull/10468

There is 2 problems:

1. bor files retire process started before first staged sync cycle finished - see [1/15 Snapshots] MaxTxNums index is in progress

[DBUG] [05-25|03:54:11.676] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:54:11.677] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:54:11.693] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:54:11.693] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:54:11.693] [index] write                            file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:54:11.758] [bor snapshots] Retire Bor Blocks        type=borspans range=54599k-54600k
[DBUG] [05-25|03:54:11.760] [snapshots] Compression start            file=v1-054599-054600-borspans workers=7
[INFO] [05-25|03:54:11.822] [1/15 Snapshots] MaxTxNums index: 0k/56913k
[DBUG] [05-25|03:54:11.828] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borspans.idx.tmp
[DBUG] [05-25|03:54:11.828] [index] calculating                      file=v1-054599-054600-borspans.idx
[DBUG] [05-25|03:54:11.828] [index] write                            file=v1-054599-054600-borspans.idx
[INFO] [05-25|03:54:12.049] [snapshots:bor:retire] Stat              blocks=54600k indices=54600k alloc=22.1GB sys=32.5GB
[DBUG] [05-25|03:54:18.689] [bor.heimdall] Got new milestone         start=57359916 end=57359929
[DBUG] [05-25|03:54:18.689] [bor] current head block behind incoming block=milestone head=0 end block=57359929




[DBUG] [05-25|03:54:29.575] [bor.heimdall] Got new milestone         start=57359916 end=57359929
[DBUG] [05-25|03:54:29.575] [bor] current head block behind incoming block=milestone head=0 end block=57359929
[INFO] [05-25|03:54:30.980] [1/15 Snapshots] MaxTxNums index: 9459k/56913k
[DBUG] [05-25|03:54:41.503] [bor.heimdall] Got new milestone         start=57359916 end=57359929
[DBUG] [05-25|03:54:41.504] [bor] current head block behind incoming block=milestone head=0 end block=57359929
[DBUG] [05-25|03:54:45.563] [bor] current head block behind incoming block=checkpoint head=0 end block=57358501
[DBUG] [05-25|03:54:45.563] [bor.heimdall] Got new checkpoint        start=57357734 end=57358501 rootHash=0xad890e6e750fedf9e8657cb0be2488bc81b490f6d5c587c3b35045116f3073c8
[DBUG] [05-25|03:54:45.563] [bor.heimdall] Failed to whitelist checkpoint err="missing blocks"
[DBUG] [05-25|03:54:45.563] [bor] unable to handle whitelist checkpoint err="missing blocks"
[DBUG] [05-25|03:54:53.562] [bor.heimdall] Got new milestone         start=57359916 end=57359929
[DBUG] [05-25|03:54:53.562] [bor] current head block behind incoming block=milestone head=0 end block=57359929
[INFO] [05-25|03:55:00.980] [1/15 Snapshots] MaxTxNums index: 26398k/56913k
[DBUG] [05-25|03:55:05.384] [p2p] Discovery table                    protocol=67 version=v4 len=162 live=162 unsol=24 ips=248 db=0 reval=912 RPC timeout=25 unsolicited reply=3
[DBUG] [05-25|03:55:05.384] [p2p] Server                             protocol=67 peers=3 trusted=0 inbound=0 closed by remote=69341 too many peers=24 EOF=7
[DBUG] [05-25|03:55:05.384] [p2p] Dial scheduler                     protocol=67 peers=3/33 tried=72306 static=0 i/o timeout=2775 connect: connection reset by peer=3 connect: connection refused=129
[DBUG] [05-25|03:55:05.389] [p2p] Server                             protocol=68 peers=7 trusted=0 inbound=0 EOF=3 closed by remote=65285 too many peers=7
[DBUG] [05-25|03:55:05.389] [p2p] Dial scheduler                     protocol=68 peers=7/33 tried=67671 static=0 connect: connection refused=96 i/o timeout=2262 connect: connection reset by peer=4 connect: network is unreachable=1
[DBUG] [05-25|03:55:05.389] [p2p] Discovery table                    protocol=68 version=v4 len=162 live=162 unsol=38 ips=249 db=0 reval=914 RPC timeout=24 unsolicited reply=5
[DBUG] [05-25|03:55:05.509] [bor.heimdall] Got new milestone         start=57359930 end=57359949
[DBUG] [05-25|03:55:05.509] [bor] current head block behind incoming block=milestone head=0 end block=57359949
[WARN] [05-25|03:55:17.467] [bor.heimdall] an error while fetching   path=/milestone/latest queryParams= attempt=1 err="Get \"https://heimdall-api.polygon.technology/milestone/latest\": read tcp 10.132.15.225:43178->104.18.41.110:443: read: connection reset by peer"
[WARN] [05-25|03:55:17.467] [bor.heimdall] an error while fetching   path=/milestone/lastNoAck queryParams= attempt=1 err="Get \"https://heimdall-api.polygon.technology/milestone/lastNoAck\": read tcp 10.132.15.225:43178->104.18.41.110:443: read: connection reset by peer"
[DBUG] [05-25|03:55:19.544] [bor.heimdall] Got new milestone         start=57359930 end=57359949
[DBUG] [05-25|03:55:19.544] [bor] current head block behind incoming block=milestone head=0 end block=57359949
[DBUG] [05-25|03:55:29.496] [bor.heimdall] Got new milestone         start=57359950 end=57359963
[DBUG] [05-25|03:55:29.496] [bor] current head block behind incoming block=milestone head=0 end block=57359963
[INFO] [05-25|03:55:30.981] [1/15 Snapshots] MaxTxNums index: 43713k/56913k
[DBUG] [05-25|03:55:41.523] [bor.heimdall] Got new milestone         start=57359950 end=57359963
[DBUG] [05-25|03:55:41.523] [bor] current head block behind incoming block=milestone head=0 end block=57359963
[DBUG] [05-25|03:55:53.544] [bor.heimdall] Got new milestone         start=57359950 end=57359963
[DBUG] [05-25|03:55:53.544] [bor] current head block behind incoming block=milestone head=0 end block=57359963
[INFO] [05-25|03:55:53.776] [snapshots:download] Stat                blocks=56914k indices=56914k alloc=26.6GB sys=32.5GB
[INFO] [05-25|03:55:53.778] [snapshots] History Stat                 blocks=56912k txs=4095m txNum2blockNum="2048=49361K,2560=56030K,2592=56471K,2608=56715K,2616=56832K,2620=56900K,2621=56912K" first_history_idx_in_db=0 last_comitment_block=56912765 last_comitment_tx_num=4095312500 alloc=26.6GB sys=32.5GB
[DBUG] [05-25|03:55:53.968] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:55:53.970] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:55:54.170] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:55:54.170] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:54.170] [index] write                            file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:54.387] [bor snapshots] Retire Bor Blocks        type=borspans range=54599k-54600k
[DBUG] [05-25|03:55:54.389] [snapshots] Compression start            file=v1-054599-054600-borspans workers=7
[DBUG] [05-25|03:55:54.484] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borspans.idx.tmp
[DBUG] [05-25|03:55:54.484] [index] calculating                      file=v1-054599-054600-borspans.idx
[DBUG] [05-25|03:55:54.484] [index] write                            file=v1-054599-054600-borspans.idx
[INFO] [05-25|03:55:54.705] [snapshots:bor:retire] Stat              blocks=54600k indices=54600k alloc=26.8GB sys=32.5GB
[DBUG] [05-25|03:55:54.771] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:55:54.772] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:55:54.844] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:55:54.844] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:54.844] [index] write                            file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:54.912] [bor snapshots] Retire Bor Blocks        type=borspans range=54599k-54600k
[DBUG] [05-25|03:55:54.913] [snapshots] Compression start            file=v1-054599-054600-borspans workers=7
[DBUG] [05-25|03:55:55.002] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borspans.idx.tmp
[DBUG] [05-25|03:55:55.002] [index] calculating                      file=v1-054599-054600-borspans.idx
[DBUG] [05-25|03:55:55.002] [index] write                            file=v1-054599-054600-borspans.idx
[INFO] [05-25|03:55:55.009] [1/15 Snapshots] DONE                    in=14m35.589952325s block=56913999
[INFO] [05-25|03:55:55.012] [2/15 Headers] Waiting for headers...    from=56913999
[DBUG] [05-25|03:55:55.016] [2/15 Headers] Requested header          from=56916716 length=192
[INFO] [05-25|03:55:55.025] [2/15 Headers] Processed                 highest=56914101 age=1w4d19h headers=102 in=0.014 blk/sec=7261
[DBUG] [05-25|03:55:55.026] [2/15 Headers] DONE                      in=17.257551ms
[INFO] [05-25|03:55:55.027] [3/15 BorHeimdall] Processing spans...   from=8532 to=8893
[INFO] [05-25|03:55:55.129] [snapshots:bor:retire] Stat              blocks=54600k indices=54600k alloc=27.0GB sys=32.5GB
[DBUG] [05-25|03:55:55.293] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:55:55.293] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:55:55.296] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:55:55.296] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:55.296] [index] write                            file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:55.299] [bor snapshots] Retire Bor Blocks        type=borspans range=54599k-54600k
[DBUG] [05-25|03:55:55.300] [snapshots] Compression start            file=v1-054599-054600-borspans workers=7
[DBUG] [05-25|03:55:55.302] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borspans.idx.tmp
[DBUG] [05-25|03:55:55.302] [index] calculating                      file=v1-054599-054600-borspans.idx
[DBUG] [05-25|03:55:55.302] [index] write                            file=v1-054599-054600-borspans.idx
[INFO] [05-25|03:55:55.421] [snapshots:bor:retire] Stat              blocks=54600k indices=54600k alloc=27.3GB sys=32.5GB
[DBUG] [05-25|03:55:55.490] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:55:55.491] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:55:55.494] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:55:55.494] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:55.494] [index] write                            file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:55.497] [bor snapshots] Retire Bor Blocks        type=borspans range=54599k-54600k
[DBUG] [05-25|03:55:55.497] [snapshots] Compression start            file=v1-054599-054600-borspans workers=7
[DBUG] [05-25|03:55:55.500] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borspans.idx.tmp
[DBUG] [05-25|03:55:55.500] [index] calculating                      file=v1-054599-054600-borspans.idx
[DBUG] [05-25|03:55:55.500] [index] write                            file=v1-054599-054600-borspans.idx
[INFO] [05-25|03:55:55.622] [snapshots:bor:retire] Stat              blocks=54600k indices=54600k alloc=27.6GB sys=32.5GB
[DBUG] [05-25|03:55:55.783] [bor snapshots] Retire Bor Blocks        type=borevents range=54599k-54600k
[DBUG] [05-25|03:55:55.784] [snapshots] Compression start            file=v1-054599-054600-borevents workers=7
[DBUG] [05-25|03:55:55.787] [index] created                          file=/mnt/erigon/snapshots/v1-054599-054600-borevents.idx.tmp
[DBUG] [05-25|03:55:55.787] [index] calculating                      file=v1-054599-054600-borevents.idx
[DBUG] [05-25|03:55:55.787] [index] write                            file=v1-054599-054600-borevents.idx

2. my node has [snapshots:bor] Stat blocks=54600k indices=54600k alloc=14.2GB sys=14.4GB - 54.6M of bor files. but node still created empty v1-054599-054600-borevents file.

Originally posted by @AskAlexSharov in #10299 (comment)

[AskAlexSharov]

tested today on amoy on main - see infinity loop.
I see next thing: node has snapshots/v1-006300-006400-bor* files and using them. But it still creating empty snapshots/v1-006399-006400-bor* files and loop on it.