fix(net): Reduce inbound service overloads and add a timeout

[teor2345]

Motivation

Sometimes Zebra's inbound service gets overloaded and drops a whole lot of connections at the same time.

Close #6911

Complex Code or Requirements

This is concurrent code.

Solution

Timeouts:

• Add a timeout to inbound service requests, there is currently no timeout on those requests. This is a security issue.
• Treat inbound timeouts like queue overloads

Overloads:

• Increase the inbound concurrency limit, to reduce overloads at the cost of slightly more memory
• Reduce the peer broadcast fraction, to reduce network load
• Reduce maximum connection drop probability, to reduce the number of disconnections that happen at the same time

Related changes:

• Document security requirements of inbound peer overload handling

Review

This is an important security fix, it should go in early in the next release.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

Work out which requests are slow (or numerous), and make them faster. This PR adds logging for inbound request timeouts.

[codecov]

Codecov Report

Merging #6950 (79a8afb) into main (329dd71) will decrease coverage by 0.04%.
The diff coverage is 62.06%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6950      +/-   ##
==========================================
- Coverage   77.65%   77.61%   -0.04%     
==========================================
  Files         310      310              
  Lines       41475    41488      +13     
==========================================
- Hits        32207    32202       -5     
- Misses       9268     9286      +18     

[arya2]

This looks good, thank you.

we're downloading blocks in the wrong order (not strict height order) then hitting a concurrency limit

I suspect you were right about the blocks being out of order, increasing MAX_INBOUND_CONCURRENCY should help.