Repository navigation
WebSocketServer SSL handshake gets stuck when client certificate is used #1500
Description
Activity
I think the fundamental issue with the current code is that SSLSocketChannel2 does not always consume all the bytes it read from the SocketChannel, as sometimes during the handshake, the SSLEngine requires it to run a task, which it schedules asynchronously and then exits - requiring SSLSocketChannel2#read() to be called again to process the remaining data.
However, the worker thread in WebSocketServer only checks SocketChannel#isReadable() and is unaware of remaining data to be processed in SSLSocketChannel2 that was already read from the SocketChannel. Thus, situations arise when SSLSocketChannel2 needs its read() method to be called, but WebSocketServer just won't make that call.
Commits ad3d043 and 89eaf41 doctored around the symptoms of this inherent issue, but failed to fully resolve it.
I suppose the real fix would be to properly layer SSLSocketChannel2 over the SocketChannel and query it for any incoming to be processed, which would then check both its internal buffer as well as the underlying SocketChannel.
Another design issue that makes me wonder: Does SSLSocketChannel2 really need to schedule the SSLEngine tasks to run asynchronously? In the case of SSLSocketChannel2 holding read data to be processed after the task has finished, this would cause the WebSocketServer worker thread to busy-loop waiting for the task to complete. In this case, it would be better to execute the task directly in consumeDelegatedTasks(), as the currently commented-out line task.run(); does.
Describe the bug
When setting up a WebSocketServer with a WebSocketServerFactory that produces ServerSockets which want or need client certificate authentication, the WebSocketServer gets stuck in the handshake when a client with a valid client certificate connects.
What I could figure out so far: The code in SSLSocketChannel2 seems to get stuck, because the last read (presumably yielding the client certificate) leaves SSLEngine in the NEED_TASK state (presumably to validate the certificate), and the task leaves SSLEngine in the NEED_UNWRAP state. But since there is no further data to read, no further call to unwrap() is made, so the handshake never completes. Note that when the client certificate is invalid, the handshake is correctly rejected.
Hacking SSLSocketChannel2 to execute tasks synchronously in consumeDelegatedTasks() and after task execution checking for this state and calling sslEngine#unwrap() then makes the handshake complete, but I don't think that's a suitable solution.
Unfortunately, I don't understand the code well enough to come up with a proper solution.
To Reproduce
Steps to reproduce the behavior:
Example application to reproduce the issue
N/A
Expected behavior
The SSL handshake should successfully complete if the client sends a valid authentication certificate.
Debug log
N/A
Environment(please complete the following information):
Additional context
N/A