Jetty 9 is EOL

[cortlepp]

Jetty 9 is EOL and is no longer being maintained. If I see this correctly it is only being used in the htmlunit-websocket-client in the class org.htmlunit.websocket.JettyWebSocketAdapter. Is there anything concretely blocking an update to Jetty 12 (the latest currently maintained version of Jetty)?

References:

• the commit that introduced htmlunit-websocket-client
• a current CVE in Jetty 9 that will not be resolved due to this version being EOL

[rbri]

Is there anything concretely blocking an update to Jetty 12

HtmlUnit is at JDK8

[niloc132]

Note also that HtmlUnit is only using jetty's websocket client, so most CVEs/etc that apply to Jetty 9 will not impact HtmlUnit.

[cortlepp]

Is there anything concretely blocking an update to Jetty 12

HtmlUnit is at JDK8

How do you intend to move forward in this regard? I understand that some people may still be stuck on JDK8, but the world is moving on and it will only become increasingly difficult to maintain the library. So is there a plan to migrate to a more recent JDK, or how do you want to handle "bitrot" issues like this one, now and in the future?

[cortlepp]

Note also that HtmlUnit is only using jetty's websocket client, so most CVEs/etc that apply to Jetty 9 will not impact HtmlUnit.

Thanks for the hint, but I think it would still be better to tackle the root cause. The vulnerability I cited above is in a URI implementation, it's anybody's best guess whether this affects the parts of the code we are using. Apart from that, a new vulnerability (in code that we are relying on) could pop up anytime, and when we are relying on EOL libs there is nothing we can do except upgrade to a supported version.

[cortlepp]

I just now noticed (thanks to the PR comment by @niloc132) that the real Issue is in htmlunit-websocket-client, and that this is not a submodule of this repo. Would you be open to publishing a new major version of this websocket client if I provided a PR for this (which would then require JDK 17 / Jetty 12) as an official maven artifact that people could then use as a drop-in replacement? I.e. the default could even stay at JDK 8, if that is important.

[cortlepp]

@rbri can you please provide some feedback as to how a suitable solution to this may look to you? After looking a bit into the htmlunit-websocket-client and how it is used I htmlunit it is now clear to me that it will not be possible to just update the htmlunit-websocket-client since htmlunit relies on several Jetty 9 APIs that have since been removed/changed. I see a couple of options moving forward:

1. update htmlunit-websocket-client to JDK 17 + Jetty 12 and adapt the integration in htmlunit, keeping the tight coupling as it is. If we do this, we should be aware that his means we essentially couple ourselves to the Jetty release policy (when it comes to the JDK), i.e. this will not be the last JDK update. I have no problem with that, but I guess that you might.
2. update htmlunit-websocket-client to use okhttp and adapt the integration in htmlunit, keeping the tight coupling as it is (and stay on JDK 8). It does not seem like the okhttp project has concrete plans to move off of JDK8, so this should be viable for some time.
3. properly encapsulate the websocket implementation(s) in htmlunit-websocket-client and only access it in htmlunit via a custom, not implementation-specific API. This would mean removing JettyWebSocketAdapter to htmlunit and re-implementing it (with whatever backend is suitable) in htmlunit-websocket-client. Then we could for example support a legacy websocket client based on Jetty 9 and a modern one based on the JDK/okhttp. This would probably be a bit more cumbersome packaging-wise since we would have to split the implementations into distinct artifacts.

[niloc132]

May I suggest option 4: Java 11+ has websocket support in the JDK as part of JEP 321, so an alternative implementation could be provided without the use of jetty/okhttp at all, and any future changes to those libraries or their dependencies will not put this at risk.

[rbri]

@cortlepp

How do you intend to move forward in this regard? I understand that some people may still be stuck on JDK8, but the world is moving on and it will only become increasingly difficult to maintain the library. So is there a plan to migrate to a more recent JDK, or how do you want to handle "bitrot" issues like this one, now and in the future?

Hi Christan,
i like to start with some clarification

• HtmlUnit is OpenSource - you can do whatever you like to do with this
• i'm doing all the work on this in my spare time (and yes i have a real 40h+ job)
  • i'm doing more than 1500 commits per year on this lib (https://www.linkedin.com/posts/ronald-brill-a93271197_review-2024-these-are-the-number-of-commits-activity-7281947673631318017-dawN?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAC4ii88BJE7tDZcyOhHL8GqKvxAKpkQrM6c)
  • the software is worth 9.000.000$ (https://openhub.net/p/HtmlUnit/estimated_cost)
• currently >17.000 github project depending on this lib

If you like to see something special to be done i can see some options

• be kind to the peoples doing the work this helps a lot to get some priority
• talk to your boss - sponsoring (https://github.com/sponsors/rbri) is the only way to make sure the lib will be maintained in the future
• provide PR's and discuss your improvements

[rbri]

@cortlepp have done a first change of the code to hopefully support your suggestion 3. This is work in progress but i think you can use this already as starting point to create a WebSocket support based on a more recent lib of your choice.

If you like i can create a new repo and you can provide your code.

And of course if you have some suggestions to improve the 'API' we can change that. Having different implementations of the API is always a good way to build something robust.

Looking forward to your suggestions...

[rbri]

@cortlepp

What is on my list to improve the API

• move the jetty stuff to the htmlunit-websocket-client project
• think about using the service interface to load the websocket lib dynamically
• document the way to use a different websocket lib

[cortlepp]

Hey @rbri , thanks for your reply. I do realize that this is an OSS project and that I have no right to ask anything of you and I appreciate the work that you have already put into the project. I only wanted to (and still do) contribute to fix this issue, it was never my intention to get you to do this or to lay any blame on you for this issue. Reading my messages again I can see that my tone was off and sounded confrontational - sorry about that.

That being said, currently I'm on vacation but I will look into your changes as soon as I get back. As soon as I have gotten something to work or i have further questions I'll get back to you.

[rbri]

@cortlepp enjoy your vacation...

[cortlepp]

I've started to work on a re-implementation of the websocket support using the JDK-internal client. Before actually starting I

• modularized htmlunit-websocket-client so we can separate the API from the implementations. I think it would be nice if we could have all implementations in one place, and not in different repositories. Note that I didn't yet cleanly split e.g. the release plugin configuration so that all modules will be released properly.
• pulled the websocket stuff out of htmlunit. Initially I implemented this by passing all parameters individually, going forward it might make sense to factor this out into a builder/config object.

I hope that I'll get to implementing (and integrating) the actual client in the coming days. My current state is here (just if you're curious / want to give early feedback).

[cortlepp]

I have some updates:

• I have an initial jdk implementation in the websocket client and an initial integration for htmlunit
• Unfortunately I must have done something wrong, both when porting the existing jetty implementation and with the new jdk one, because now in both cases the websocket tests fail.

I then realized that in the integration of my employer we actually already have disabled the websocket support in htmlunit, i.e. we never even touch these code paths. And if we don't use it, I can't justify working on this.

TLDR: I will (for now) no longer work on this issue. I will keep the work I have done (which is linked above) published, if anybody else wants to continue where I left off you are welcome to do so.