Scalability - Reconnect Improvements

[poulok]

The goal of this epic is to improve the performance of reconnect such that the scale can scale to hold greater numbers of entities than it does today while supporting 10k TPS.

Misc Improvements

Preview Give feedback

1. Need a JMH benchmark for virtual map reconnects #11347
   1 of 1
   P1 Performance Platform Data Structures Platform Reconnect Platform Virtual Map Test Development +6
   
2. Remove forced flush of the DB after reconnect #9157
   Platform Data Structures Platform Reconnect +2
   
3. Level by level traversal with pauses to wait for confirmations #11425
   +0
   
4. Throttle the teacher based on the size of expectedResponseQueue #11426
   +0
   
5. A new metric for reconnect duration #11431
   Platform Data Structures Platform Reconnect +2
   
6. New reconnect metrics based on ReconnectNodeCount collected stats #11432
   Observability Platform Platform Data Structures Platform Reconnect +4
7. Add metrics support for the new ReconnectBench in swirlds-benchmarks #11745
   +0
   
8. Emulate latencies in ReconnectBench #11809
   +0
   
9. Reuse generated state between iterations in ReconnectBench #11816
   +0
   
10. Virtual root nodes may be synced in parallel during reconnects #12108
    Improvement Performance Platform Platform Data Structures Platform Virtual Map +5
    
11. Improve reusing the generated state in the ReconnectBench JMH test #12194
    Platform +1
12. Fuzz slow I/O delay in ReconnectBench #12343
    +0
    
13. Virtual map reconnect: improve virtual subtree snapshots on the teacher side #13732
    Performance Platform Platform Data Structures Platform Reconnect Platform Virtual Map +5
14. AsyncInputStream: reduce the number of locks when reading from the socket #15234
    Performance Platform Platform Reconnect +3
    
15. Remove timeout from reconnect/rehash Iterators #15167
    Platform Platform Reconnect +2
    

Push to Pull Tasks

Preview Give feedback

1. Switch virtual map reconnects to the pull model #11403
   Performance Platform Data Structures Platform Reconnect Platform Virtual Map +4
   
2. Provide a way to have different reconnect teacher/learner threads implementations #11436
   Platform Data Structures Platform Reconnect Platform Virtual Map +3
   
3. Implement ReconnectNodeCount for pull reconnect model #11618
   Platform Data Structures Platform Reconnect +2
   
4. A new interface for node order traversal policy during pull-based reconnects #11786
   Performance Platform Data Structures Platform Reconnect Platform Virtual Map +4
   
5. Implement a new pull reconnect traversal policy: leaves to root #11889
   Performance Platform Data Structures Platform Reconnect Platform Virtual Map +4
   
6. Enable pull / leaves to root reconnect policy by default #12648
   Performance Platform Platform Data Structures Platform Reconnect Platform Virtual Map +5
7. Analyze and improve performance of pull / leaves to root reconnect policy #12649
   Performance Platform Platform Data Structures Platform Reconnect Platform Virtual Map +5