Maybe we should be able to opt arbitrary requests out of tripping the http request breaker?

[nik9000]

Describe the feature:
I was talking to someone about a problem they were having with a test cluster (#18144) bumping into the http breaker and I asked them to run the tasks list API to see if any tasks were stuck. And it tripped the breaker. Maybe we should opt the tasks list API out of the breaker? Maybe we should allow any request to opt out of the breaker via url parameter and only use it for administrative stuff? I dunno.

[jasontedor]

Relates #17951

[danielmitterdorfer]

I think this is necessary although I want #17951 in before, so we use similar approaches. At least in the first step, I wouldn't go so far as to allow users to opt-out via an API call but just statically opt-out actions that are necessary for inspecting a cluster.

[nik9000]

just statically opt-out actions that are necessary for inspecting a cluster

My reasoning behind just putting it on the API is that we don't have to know that list. I'm not sure we'll know it consistently. We could make it a dynamic setting but then settings update would have to be in the list or else we couldn't change the list when we're in a bad state.

[jasontedor]

I don't think we need to do this. The circuit breaker setting is dynamic. There are actions (e.g., TransportReplicationAction) that should always be exempt from the circuit breaker setting but otherwise if trouble arises the end-user can resize the limit.

[clintongormley]

I'm against adding this as well. Sysadmins would have to scrub this option from every request they receive from users intent on bypassing the circuit breaker.

[clintongormley]

Discussed in FixItFriday - we prefer the whitelist of admin requests, and the ability to dynamically update this settings, to just allowing any request to ignore the circuit breaker limit. Better for sysadmins