Description
openedon Aug 2, 2022
8.4 SNAPSHOT
Original install method (e.g. download page, yum, from source, etc.):
Cluster installed with terraform on elastic cloud.
Describe the bug:
When fleet overdrives the elastic search nodes it get into a bad state
Steps to reproduce:
- Run 25k agents with underpowered es nodes
Expected behavior:
Screenshots (if relevant):
Errors in browser console (if relevant):
Error fetching agents
[es_rejected_execution_exception: [es_rejected_execution_exception] Reason: rejected execution of org.elasticsearch.action.bulk.TransportBulkAction$1/org.elasticsearch.action.ActionListener$RunBeforeActionListener/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.tasks.TaskManager$1{SafelyWrappedActionListener[listener=WrappedActionListener{org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction$$Lambda$8846/0x00000008020a4f58@50921e21}{org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction$$Lambda$8849/0x00000008020a5378@63f4b3b9}]}{Task{id=1055264, type='transport', action='indices:data/write/bulk', description='requests[1], indices[.security-tokens]', parentTask=unset, startTime=1658399095698, startTimeNanos=21774904411368219}}/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$6237/0x0000000801d90e58@61953e36/org.elasticsearch.action.bulk.TransportBulkAction$$Lambda$8017/0x0000000801f9d000@26a0d988 on EsThreadPoolExecutor[name = instance-0000000001/write, queue capacity = 10000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ed3e9a8[Running, pool size = 3, active threads = 3, queued tasks = 10000, completed tasks = 429077]]]: rejected execution of org.elasticsearch.action.bulk.TransportBulkAction$1/org.elasticsearch.action.ActionListener$RunBeforeActionListener/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.tasks.TaskManager$1{SafelyWrappedActionListener[listener=WrappedActionListener{org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction$$Lambda$8846/0x00000008020a4f58@50921e21}{org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction$$Lambda$8849/0x00000008020a5378@63f4b3b9}]}{Task{id=1055264, type='transport', action='indices:data/write/bulk', description='requests[1], indices[.security-tokens]', parentTask=unset, startTime=1658399095698, startTimeNanos=21774904411368219}}/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$6237/0x0000000801d90e58@61953e36/org.elasticsearch.action.bulk.TransportBulkAction$$Lambda$8017/0x0000000801f9d000@26a0d988 on EsThreadPoolExecutor[name = instance-0000000001/write, queue capacity = 10000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ed3e9a8[Running, pool size = 3, active threads = 3, queued tasks = 10000, completed tasks = 429077]]
Provide logs and/or server output (if relevant):
Any additional context: