NPE in GzipHandler from race condition

[mperktold]

Jetty version(s)
12.0.12

Jetty Environment
core

Java version/vendor (use: java -version)
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

OS type/version
Windows 11

Description
In production, we often get the following NPEs:

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.util.compression.DeflaterPool.acquire()" because "this._deflaterPool" is null
 
Cannot invoke "org.eclipse.jetty.util.compression.DeflaterPool.acquire()" because "this._deflaterPool" is null
    java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.util.compression.DeflaterPool.acquire()" because "this._deflaterPool" is null
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.getDeflaterEntry(GzipHandler.java:396)
        at org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.commit(GzipResponseAndCallback.java:238)
        at org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.write(GzipResponseAndCallback.java:132)
        at org.eclipse.jetty.io.content.ContentSinkOutputStream.write(ContentSinkOutputStream.java:54)
        at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
        at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
        at java.base/sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
        at java.base/sun.nio.cs.StreamEncoder.lockedFlush(Unknown Source)
        at java.base/sun.nio.cs.StreamEncoder.flush(Unknown Source)
        at java.base/java.io.OutputStreamWriter.flush(Unknown Source)
        at java.base/java.io.BufferedWriter.implFlush(Unknown Source)
        at java.base/java.io.BufferedWriter.flush(Unknown Source)
		...

Apparently, this comes from a race condition between writing the response and stopping the server.

How to reproduce?
You can use the following snippet together with breakpoints in your IDE to reproduce the issue:

Server server = new Server(8080);
server.setHandler(new GzipHandler(new Handler.Abstract() {
    @Override
    public boolean handle(Request request, Response response, Callback callback) throws Exception {
        try (var out = Content.Sink.asOutputStream(response)) {
            out.write("Hello, Jetty".getBytes(StandardCharsets.UTF_8));
        }
        return true;
    }
}));
server.start();
server.stop();

One breakpoint can be set on the line server.stop() to control when the server will be stopped.

The other one can be set here were the deflater pool is used for writing back the response, and where the NPE is thrown:

jetty.project/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/gzip/GzipResponseAndCallback.java

Line 238 in 8277051

_deflaterEntry = _factory.getDeflaterEntry(request, contentLength);

Now you can use these breakpoints to ensure the following steps are executed in the "right" in order:

1. Make a request. Let it wait in GzipResponseAndCallback.commit.
2. Stop the server.
3. Let GzipResponseAndCallback.commit proceed.

[joakime]

I think we should at least make this more friendly for graceful shutdown for sure. (which is not what the OP shows in his example)

The standard stop behaviors are rather racy towards active requests / connections anyway.

[mperktold]

What would graceful shutdown look like?