Frequent "I/O reactor terminated abnormally" crashes ES client (due to NoSuchElementException)

[LouisTrezzini]

Java API client version

7.17.28

Java version

java version "21.0.3" 2024-04-16 LTS

Elasticsearch Version

7.17.28

Problem description

Hello,

We are seeing frequent (about once per day) errors where the Apache HTTP client used by the Elasticsearch client stops working due to an I/O reactor terminated abnormally error.

Here is a full stack trace:

I/O dispatch worker terminated abnormally
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by java.util.NoSuchElementException: null
    at java.base/java.util.Collections$1.next(Collections.java:5037)
    at co.elastic.clients.transport.rest_client.MultiBufferEntity.init(MultiBufferEntity.java:65)
    at co.elastic.clients.transport.rest_client.MultiBufferEntity.close(MultiBufferEntity.java:59)
    at org.apache.http.nio.protocol.BasicAsyncRequestProducer.close(BasicAsyncRequestProducer.java:155)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.releaseResources(DefaultClientExchangeHandlerImpl.java:84)
    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:429)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    [8 more]

Impact:

• When this happens, the HTTP client is marked as STOPPED.
• As a result, all subsequent calls to the Elasticsearch client fail with:

 java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED

Additional Context:

• Apache HTTP versions:

org.apache.httpcomponents:httpasyncclient:4.1.4
org.apache.httpcomponents:httpclient:4.5.14
org.apache.httpcomponents:httpcore-nio:4.4.12
org.apache.httpcomponents:httpcore:4.4.16

• We use BulkIngester quite heavily.
• As a temporary workaround, we have a watchdog thread that checks if the HTTP client is still running and recreates the RestClient if needed. (Pseudocode below:)

if (!httpClient.isRunning()) {
    RestClient newClient = createClient();
    // Atomically swap httpClient, using an AtomicReference
}

Thanks for your help!

[l-trotta]

Hello! From checking the code and the stacktrace you provided, I suspect that something "common" such as a TimeoutException is causing an exception in the RestClient code: from the stacktrace we can see the HttpAsyncRequestExecutor failing, then the DefaultClientExchangeHandlerImpl trying to release the resources of the producer, and finally some concurrency issue we didn't account for in MultiBufferEntity is triggering the NoSuchElementException.

I'd like to first identify the initial exception, so that then we can reproduce the issue and analyze what's happening with MultiBufferEntity. Is there any other log you could provide? Setting the log level of the application to see more details could help.

Also:

• Is the java client connected to the server directly or is it behind a proxy?
• How are the ElasticsearchClient and BulkIngester created and configured? (with parameters included, especially timeouts)
• How do items get added to the BulkIngester? (single thread, multithread)

[LouisTrezzini]

Hi!

Thanks for the prompt answer. I will try to collect more logs. In the meantime, here are the answers to your questions:

• Is the java client connected to the server directly or is it behind a proxy?

Yes, our servers are directly connected to ES nodes

• How are the ElasticsearchClient and BulkIngester created and configured? (with parameters included, especially timeouts)

for ElasticsearchClient:

List<HttpHost> httpHosts = getHosts(_hosts);
RestClientBuilder builder = RestClient.builder(httpHosts.toArray(new HttpHost[] {}));
builder.setHttpClientConfigCallback(new HttpClientConfig(_credentials, _localAddressPrefix)); // sets BasicCredentialsProvider and setDefaultRequestConfig(RequestConfig.copy(RequestConfig.DEFAULT).setLocalAddress(a).build())
RestClient httpClient = builder.build();

RestClientOptions options = new RestClientOptions(SafeResponseConsumer.DEFAULT_REQUEST_OPTIONS);
ElasticsearchTransport transport = new RestClientTransport(httpClient, SearchUtils.makeJsonpMapper(), options);

this.httpClient = httpClient;
this.client = new ElasticsearchClient(transport);
this.asyncClient = new ElasticsearchAsyncClient(transport));

for BulkIngester:

BulkIngester.of(b ->
    b
        .client(_asyncClient)
        .flushInterval(5000, TimeUnit.MILLISECONDS)
        .maxOperations(100)
        .maxConcurrentRequests(50)
        .listener(_listener)
);

• How do items get added to the BulkIngester? (single thread, multithread)

it's hard to give a detailed explanation
the bulk ingester lives in a singleton bean in a tomcat/spring web application, so it's naturally multi threaded and ingester.add can be called from multiple threads concurrently
however, I don't think the ingester is under pressure when the issue happens ("I/O reactor terminated abnormally" at 00:24:05.730)

(pending_requests = ingester.pendingRequests() & pending_operations = ingester.pendingOperations())

thanks again for your help!

[LouisTrezzini]

Hi @l-trotta!

I added more logs, and they seem to point out to this exception (most likely) being raised right before the I/O reactor gets shut down:

Connection closed unexpectedly
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146)
    at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
    at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:283)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:1583)

or perhaps this

Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:401)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:434)
    at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231)
    at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    [5 more]

[l-trotta]

Hey @LouisTrezzini, thanks for all the details! So, as suspected the underlying exceptions are networking related, and they are most likely caused by connectivity issues between the client and the Elasticsearch server. This is not something that can be fixed client side, and will probably need to be investigated. That said, the client definitely has a concurrency bug that is triggered by these specific exception, and I've been trying to replicate it with no success. I'll investigate more and update as soon as I have more information!