Problems with Content-Length using Undertow or Jetty

[marcinc81]

Yesterday I was doing comparison of all 3 servers: Netty, Jetty and Undertow. By the way Undertow is the fastest one, Netty is next, last is Jetty. Really small difference: 7300req/s - 6300req/s. For sure no one will notice this difference in production.

Anyway, I changed server to Undertow as it is the winner and found weird issue. After changing some asset (eg. generated JavaScript from Gulp/Browserify), browser sends this error message:

GET http://localhost:8080/assets/js/forms.js net::ERR_CONTENT_LENGTH_MISMATCH

And indeed, in HTTP header Jooby sets content length different than sent file has in reality. Doesn't help reloading app, start / stop, mvn clean, etc. And this is weird. Probably this issue doesn't apply in a production where assets are static.

The same issue is with Jetty, but in this case this web server logs exception:

[2016-09-07 16:33:29,628]-[jetty task-48] WARN  org.eclipse.jetty.io.AbstractConnection - 
java.lang.NullPointerException: null
    at org.jooby.internal.jetty.JettyResponse.failed(JettyResponse.java:87) ~[jooby-jetty-1.0.0.CR7.jar!/:1.0.0.CR7]
    at org.eclipse.jetty.util.IteratingNestedCallback.onCompleteFailure(IteratingNestedCallback.java:65) ~[jetty-util-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at org.eclipse.jetty.server.HttpOutput$ReadableByteChannelWritingCB.onCompleteFailure(HttpOutput.java:1255) ~[jetty-server-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:401) ~[jetty-util-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:115) ~[jetty-io-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar!/:9.3.9.v20160517]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]

Additionally Jetty has problem with serving one image (png, 1954544 bytes) - sends only first 800kB, and browser displays the same error (content length mismatch).

Switching back to Netty resolves the problem.

Cheers! :)

[jknack]

Can you create a demo app at github so I can look?

Also, did you try the jooby asset module: http://jooby.org/doc/assets/

How did you run the tests? 7300 doesn't seem to fast, but of course depends on what you run there.

We did pretty good at Techempower benchmark. They use wrk

[marcinc81]

I will try to prepare demo app. Yes, I'm using assets module.

I did simple test on my local machine, it was just rendering home page (Jade without cache), resolving session etc. Results are GOOD. PHP7 (Laravel with caches, optimized page) has 650req/s. Similar page. ;)

Netty on my production server has 9600 req/s (Jooby in production mode).

The test was not for checking general performance of Jooby - which is awesome, but for checking which one web server is the fastest one with Jooby. I was using JMeter.

2 months ago I found Jooby on Techempower benchmark - that's why I'm here! :)

[jknack]

glad you're happy with a jooby performance is better than Spring/Play.. just there was a recent discussion in reddit about jooby performance and that was the reason I asked about it.

demo app will great to see and/or reproduce the content-length errors between servers (these type of errors are tricky)

[marcinc81]

Well, the issue with Undertow server is just size of file served via assets module. Didn't check Jetty.

Here's working example. I couldn't replicate issue with JavaScript file, but finally I did it! Just in my case the .js file is quite big, it's Angular app in debug mode: 1.162kB. So the issue is not with "replacing" file by newer version, but with sending it.

I think both web servers have limits.

[jknack]

@marcinc81 thanks a lot for the examples. I was able to reproduce and fix in both servers.

Problem was exactly the same in both 😄

Wasn't not a content-lenght problem (necessarily)... it was related to asynchronous sending of larger files (> 16kb) and closing the connection too early.

Will push the fixes later today, thanks again.