Deadlock in jetty server helper startup code

[antheque]

Recently, we updated restlet to the 2.1-RC4 release in our application. The unit tests stopped working. It seems that the jetty server helper has some timing issue which manifests itself when the server is stopped and restarted in a single process. I tried very hard to pinpoint the bug, but failed.

In general we have a server and a client. Each unit test method starts the server, starts the client, does something, checks results, and then stops both the server and the client. Each test method works correctly when tested in isolation, but when I run all tests, the first method works correctly, but the second one results in a deadlock.

Looking at the thread dumps, it seems that the client waits for an answer from the server, but the answer doesn't come. I tried debugging it on the client side. The client sends the request OK, and then halts when waiting for the response. Debugging on the server side revealed that the connection is accepted by jetty (SelectChannelConnector.accept), the channel is registered with the SelectorManager, but nothing happens afterwards. The ServerResource method isn't called. The request disappears somewhere between the SelectorManager and my own application's ServerResources.

I tried looking for leftover state. I stopped the server, performed Enginge.clear(), System.gc and did a heap dump afterwards. It seems that all ports are closed correctly, all threads die correctly, all jetty objects are cleared. AFAIU the issue must lie within the startup code itself.

I've been able to write a minimal application which mimicks what we do. A unit test with two test methods, the first method runs correctly, the second results in a deadlock. All is fine with the built-in server connector, and the ext.simple server connector, so the blame probably lies on jetty.

The jetty connector in restlet 2.0 worked fine on this as well.

The minimal app can be downloaded from my google docs account. Download it and type mvn clean install. On my machine it sometimes halts on the first test method, sometimes on the second. In eclipse the test deadlocks on the second test method once every couple of times. On my app with real test, it seems to hang every single time. These effects are visible on my machine both in Eclipse and when maven is run from the command line.

I'm at a loss right now. I would be very grateful for any help.

https://docs.google.com/open?id=0B3_VP-_aChUzUjZyZ3g4ZG1hNjQ

[thboileau]

That's very annoying, and we share also your disapointment. It seems we also encounter this kind of problems also, since we serve our web pages using the Jetty connector and have experienced strange behaviour.

[antheque]

Thanks for the answer. Could you please elaborate a bit more?

1. Did you try my test project? Does it show the deadlock on your machine(s)?
2. Is there any other github issue related to deadlocks in jetty helper, where some work is being done? Maybe there are comments over there with more info.
3. What problems do you encounter, what strange behaviour do you see?

Right now, we use the jetty helper in production, but the tests work with the built-in connector. I'd like to fix that asap. I spent some time on it but couldn't go any further. Any bit of information on this might help me.

[antheque]

FYI: I noticed the recent commit e142785. Tried my test app with the 2.1 snapshot from the 3rd of May. I still get the deadlocks.

I also noticed that there are three "connector styles" in jetty. I added a line

component.getServers().get(0).getContext().getParameters().set("type", "2");

to my TestServer class. It seems that I get the deadlocks with both NIO connectors, the selecting one and the blocking one. This is both 2.1-RC4 and the 3rd of May snaphost. The BIO one ("3") is OK.

[jlouvel]

Thanks for the detailled report and the test case. We will be looking more closely at it moving forward (for 2.1.0) as we try to stabilize the connectors.

Updated milestone and assignee