[CI] WatchBackwardsCompatibilityIT failures

[martijnvg]

Several bwc tests fail, because of connection timeouts. In the es logs of the upgraded node the following fatal error can be found:

[2019-04-03T22:02:01,777][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#4]], exi
ting
java.lang.AssertionError: setIfMatch [3], currentDocTem [1]
        at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6
.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteReplicaRequest.writeTo(TransportReplicationAction.java:1332) ~[elasticsearch
-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:107) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:93) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:140) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler.internalSendMessage(OutboundHandler.java:78) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:70) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:690) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:679) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.access$300(TcpTransport.java:100) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:272) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:626) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$2(SecurityServerTransportInterceptor.java:129) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAfterRewritingAuthentication(SecurityContext.java:111) ~[?:?]

I suspect that this assertion error is caused by watcher trying to update a watch status, because the watcher tests was the first one to fail with connection timeout. Also Watcher's execution service does not take into account nodes before 6.6.0 that don't support if_seq_no and if_primary_term parameters on write requests.

build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/226/console

[elasticmachine]

Pinging @elastic/es-core-features

[martijnvg]

This should be fixed by #40888

[davidkyle]

I have seen a similar error in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/364/console

[2019-05-08T23:06:37,431][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#1]], exiting
java.lang.AssertionError: setIfMatch [630], currentDocTem [1]
	at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]

The error is on the one third upgraded node upgrading from 6.1.0 to 6.7.3, I grabbed relevant logs

rolling-upgrade-logs.zip

[davidkyle]

I suspect https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/366/console and https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/27/console are the same root problem. In both cases a node seems to have died unexpectedly

[droberts195]

There have been more failures of this today (and I suspect every day last week too as nobody was doing test triage during the all hands event).

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/399/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/60/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/398/console

These all manifest themselves in the test log as a failure to kill a process, for example:

Standard error:
kill: (812909): No such process

Then in the one third upgraded cluster node log is the error from the original issue description:

[2019-05-20T13:20:13,714][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#13]], exiting
java.lang.AssertionError: setIfMatch [648], currentDocTem [1]
	at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteReplicaRequest.writeTo(TransportReplicationAction.java:1332) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:107) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:93) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:140) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler.internalSendMessage(OutboundHandler.java:78) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:70) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:690) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:679) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.access$300(TcpTransport.java:100) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:272) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:632) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$2(SecurityServerTransportInterceptor.java:129) ~[?:?]
	at org.elasticsearch.xpack.core.security.SecurityContext.executeAfterRewritingAuthentication(SecurityContext.java:111) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:129) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:542) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:530) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.sendReplicaRequest(TransportReplicationAction.java:1213) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1175) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:165) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:152) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:126) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:488) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:150) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:133) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]