Stuck shard causing client nodes to run out of memory

[ppf2]

We had an instance where a shard is stuck during recovery (in initializing state) due to peer node connectivity issues during recovery. When this occurs with concurrent indexing and searching to the cluster, all the client nodes started to run out of memory (adding a new client node does not help for that new client node will also end up running out of memory). We collected the heap dump on OOM and it looks like a lot of heap is used in the netty buffer which doesn't seem right.

[s1monw]

what version is this on?

[ppf2]

1.3.4

[bleskes]

I think I see what's going on. Normally, the client nodes receive a bulk request, splits it in shard bulks and send it on to the node where the primary is hosted. If the primary is not yet ready, the client node will wait for it to start. It does so by registering a listener for a cluster state update. Problem is that by registering the listener, it frees up a bulk thread which happily accepts another bulk request and so forth. We need to find a way to apply some back pressure in this case.

In the past we had the idea of having a central listener for all shard related issues - that would help here too.

An alternative is to limit the amount of "transient" cluster state listener which will throttle this but also other similar requests (like waiting for a master nodes).

[kimchy]

@bleskes I think that potentially adding a circuit breaker on "public" actions, count towards it and down it when we send the response back through the channel might help in a more generic fashion, for most use cases.

[bleskes]

@kimchy yeah, that's another more generic solution. I wonder if we shouldn't just add indexing request to the current request circuit breaker, which is currently only used for big arrays, i.e., per requests aggs. We might want to split it into "bigarrays" and "indexing" sub breakers so it will be easy to monitor. I briefly looked at the code and it seems doable.