Description
We run all queries with timeout=160s
, but I understand this only really bounds the collection phase of the search?
We had someone run a query using aggregations today over billions of records, which brought the cluster down.
The cluster never OOM'd, but it did run into constant GC as the heap got full.
Another query was run again over potentially huge amounts of data. The cluster had indexing disabled as it was recovering from the previous event, and since the query was run it's been at 100% CPU for about 40 minutes now. The query, afaict, is still running. But we have no way of knowing what it is, and no way to kill it other than restarting the entire cluster.
I'd like to request a feature that lists all queries that are currently executing on every node, as well a way to kill them while they're in progress.