EofException caused by ClosedChannelException

[brettkail-wk]

Jetty version(s)
Observed in 12.0.6-12.0.10 with ee10.

First observed in March 2024 when we deployed a release of our server that upgraded from 11.0.20 to 12.0.6+ee10. It had never been observed prior to that (at least, I can't find it in our retained error logs from the last year).

Jetty Environment
ee10

Java version/vendor (use: java -version)

openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode, sharing)

OS type/version
Amazon Linux 2

Description
We periodically but rarely (a few times a month) see EofException caused by ClosedChannelException. We more routinely see plain IOException when a client disconnects without reading the response, and we ignore that error. We've considered ignoring this error too, but our understanding is that it should only happen if Jetty closed its side of the connection and then subsequently attempted to write, so it seems like this could be some kind of connection state management issue in Jetty. The relevant filters/servlets are purely synchronous: they do not use startAsync, and they do not use HttpServletRequest/HttpServletResponse from other threads. Recent stack trace from 12.0.10+ee10 (as logged by Logback, which writes the explicit null for null messages).

org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:117)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
	at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.process(HttpConnection.java:842)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
	at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1429)
	at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1263)
	at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:751)
	at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:288)
	at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:205)
	at org.eclipse.jetty.ee10.servlet.HttpOutput.close(HttpOutput.java:562)
	redacted
	at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
	at org.eclipse.jetty.server.Server.handle(Server.java:179)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException: null
	at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:222)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:568)
	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:660)
	at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:111)
	... 52 common frames omitted

How to reproduce?
I don't know. I've tried various things against a local Jetty instance (early client close, delayed client close, slow client write, slow server write, different buffered amounts to trigger an error from .write vs .close, etc.), but the only relevant error I can cause is EofException with IOException: Broken pipe as expected.

[sbordet]

Your stack trace shows that a Servlet Filter called ServletOutputStream.close(), which triggers the write of the response.

Jetty tries to write the response, and finds the connection already closed.

This may have happened due to timeouts concurrent with your application.

Another reason might be previous failures, where a previous filter caused a flush() or similar, which caused the close of the connection, and a subsequent filter shown in the stack trace above finds the connection already closed.

Do you see other errors that may have triggered the close of the connection before the one reported in the stack trace above?

[brettkail-wk]

Do you see other errors that may have triggered the close of the connection before the one reported in the stack trace above?

No, I don't see any other exceptions, errors, or warnings in the logs +/- 5min of when this occurs.

We still see this with relatively low frequency (3 occurrences in the last 30 day).

This may have happened due to timeouts concurrent with your application.

You mean some timeout managed by Jetty?

Another reason might be previous failures, where a previous filter caused a flush() or similar, which caused the close of the connection, and a subsequent filter shown in the stack trace above finds the connection already closed.

I would not expect that. At least one of the servlets writes a large amount of statically generated data and closes the output stream once. There shouldn't be any additional flush() or secondary close().

However, even if that did happen for some reason, we expected Jetty's ServletOutputStream to have a state machine that would know it had already closed the underlying channel and not attempt to continue using it. If our understanding is incorrect, and it's expected that Jetty will continue using channels that it has already internally closed, then we'll ignore this error as harmless. We were hesitant to do that since it seemed like an issue with the state machine.