Warp sync stalls, probably because the peer isn't sending anymore data

[dvdplm]

Sometimes warp sync stalls. The likely cause is that the warp sync peer is not sending any data and no progress is made. It can look like this:

2020-02-27 10:18:07  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:12  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:17  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:22  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:27  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:32  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:37  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:42  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:47  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:52  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:18:57  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:02  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:07  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:12  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:17  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:22  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:27  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:32  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:37  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:42  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:47  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:52  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:19:57  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:02  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:07  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:12  IO Worker #2 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:17  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:22  IO Worker #3 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:27  IO Worker #0 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-02-27 10:20:32  IO Worker #1 INFO import  Syncing snapshot 509/4349        #0   22/25 peers   920 bytes chain 3 MiB db 0 bytes queue 53 KiB sync  RPC:  0 conn,    0 req/s,    0 µs

TODO:

1. Validate the hypothesis
2. Add heuristics to detect the stall and try to find a different peer with the same snapshot and continue from there, and/or restart the search for a snapshot peer.