[BUG] Memory leak in Web Service Listener

[ChristopherSchultz]

Describe the bug
Under certain conditions, instances of sun.net.httpserver.HttpConnection are retained forever after the connection has been closed.

To Reproduce
Setup steps (if required). Example:

1. Configure a channel whose source listener is a Web Service Listener with all defaults (e.g. Mirth is accepting the SOAP request and ingesting HL7 messages through its standard service).

Steps to reproduce the behavior:

1. Additional steps (this issue represents a DOS and I would prefer to give details privately; please let me know how best to submit confidential information)

Expected behavior
After connections are closed, they are eventually GC'd.

Actual behavior
Connections pile-up forever until the heap is exhausted.

Environment (please complete the following information):

• OS: Linux kernel 4.14
• OpenJDK 1.8.292-b10, 64-bit
• Mirth Connect 3.9.0, also 3.10.1

Workaround(s)
Re-deploying the channel allows all connections to be cleaned-up.

Additional Information
This appears to be a leak in Java's built-in SOAP server and not actually a problem with Mirth itself. I'm wondering if Mirth can possibly perform some additional setup operations to allow these connections to be cleaned-up properly.

[jonbartels]

Discussion in Slack - https://mirthconnect.slack.com/archives/C02SW0K4D/p1627916984014200

[narupley]

👀

[narupley]

After some research, it appears that the default behavior of that internal Sun ServerImpl class is to hold onto "stuck" connections forever, maybe because there's always a chance that they're not really stuck and the network layer is just taking a long time. But you can override that by setting these in your vmoptions:

-Dsun.net.httpserver.maxReqTime=60
-Dsun.net.httpserver.maxRspTime=60

When at least one of those is set, the ServerImpl will start up a separate thread to periodically (once per second, but that can be tweaked too with sun.net.httpserver.timerMillis) purge any old connections.

This issue could be resolved by having Mirth Connect set those system properties by default, unless they're overridden in VM options.

[ChristopherSchultz]

This solution appears to work-around the problem in dev/test, and it actually looks like it's the official (if undocumented) way to prevent DOS in the built-in HTTP server. All other "real" HTTP servers do this by default. I'll be trying it in production very soon and will be able to confirm that the case-in-the-wild has been resolved as well.

+1 for enabling this by default if the user hasn't already set a a value in *.vmoptions.

[ChristopherSchultz]

After 24 hours of observation, this workaround absolutely prevents the memory leak from getting out of control.

[ChristopherSchultz]

I see this has been fixed in 4.0.0. Was the solution to simply enable these automated clean-up threads, or has the underlying implementation been replaced with Jetty?