BigArrays#newByteArray always uses REQUEST breaker

[Tim-Brooks]

When you create a byte array using BigArrays, it always increases the
REQUEST breaker. The REQUEST is described as:

/**
 *  The request breaker tracks memory used for particular requests. This
 *  includes allocations for things like the cardinality aggregation, and
 *  accounting for the number of buckets used in an aggregation request.
 *  Generally the amounts added to this breaker are released after a request
 *  is finished.
 */

However, there is also a IN_FLIGHT_REQUESTS breaker. This breaker is
described as:

/**
 * The in-flight request breaker tracks bytes allocated for reading and
 * writing requests on the network layer.
 */

When we serialize a message for writing to a channel in the http or tcp
transports, we use BigArrays which manipulates the REQUEST breaker.
However, based on the descriptions, it seems like this should be the
IN_FLIGHT_REQUESTS breaker.

[elasticmachine]

Pinging @elastic/es-core-infra

[Tim-Brooks]

@s1monw / @danielmitterdorfer - do you agree this is an issue? Daniel, I'm pinging you because in git is looks like you added the IN_FLIGHT_REQUESTS breaker.

[s1monw]

+1 I agree @dakrone WDYT?

[dakrone]

@tbrooks8 the request breaker was added initially before the in flight requests breaker, it was intended for requests that needed to use bigarrays for doing some sort of calculation (like the cardinality aggregation which uses some byte arrays from BigArrays). The in flight breaker is solely for network layer stuff (it could basically be the NETTY_BREAKER)

Personally I think it's nice to have both, since any resources used and released can use the requests breaker, and the in flight request breaker can be solely for network stuff while having different limits. However, I can see how they are close to each other, so if you want to collapse them into the same breaker that'd be fine too (I'd prefer the "REQUESTS" name myself)

[Tim-Brooks]

@dakrone - I'm not really suggesting collapsing them. I was just pointing out that right now REQUESTS is the circuit breaker we use when we claim bytes for serializing a message to write to the network. It seems based on the descriptions that should be IN_FLIGHT_REQUESTS ?

[dakrone]

@tbrooks8 yeah if it's possible to change just the one when claiming bytes for serializing network messages, I agree it should be changed to IN_FLIGHT_REQUESTS