Client Node Request/Fetch Circuit Breaker

[pickypg]

Currently, it's possible to overload a client node during the fetch process by requesting extreme amounts of data. In particular, this is possible to observe by repeating the problem in #11070 (e.g., fetching too many things in an aggregation).

[clintongormley]

This is at least partially solved by:

• Limit request size #17133 - limiting the size of inflight requests
• Add a soft limit on the number of shards that can be queried in a single search request. #17396 - limiting the number of shards in a request to 1000
• Hard limit on from/size #9311 - hard limit on from/size
• Circuit break on aggregation bucket numbers with request breaker #19394 - circuit break on aggregation buckets

It would still be possible to overwhelm a client node by requesting too many enormous documents as hits, but the chances are greatly reduced

[clintongormley]

@metacret rather open a pull request where people can review it properly

[metacret]

@clintongormley it was edited on v1.7.5 branch which is the version we're using. This is the urgent hot fix to prevent our tribe nodes from being crashed by long GC pause or OOM error. I just wanted to ask you to verify how feasibly my idea will work before setting up the official PR because making PR will need huge amount of work including writing all unit tests.

My summarized idea is TransportResponseHandler should keep recording how many bytes it has processed and if that number is over the threshold, MessageChannelHandler should throw an exception before response deserialization to prevent memory allocation. It was quite tricky how to record this data because TransportResponseHandler is allocated on the fly for every shard request. So I made SearchServiceListener to keep how many bytes are processed and that information is being propagated up to TransportResponseHandler.

[metacret]

I couldn't find easy solution to Install the circuit breaker in handleResponse because it was very hard for me find out how to release everything.

[tomcallahan]

Closing as a duplicate of #9310