-
Notifications
You must be signed in to change notification settings - Fork 522
Fix connection termination issues when using connection filters (#737, #747). #750
Conversation
{ | ||
_outputStream.Write(block.Data.Array, block.Data.Offset, block.Data.Count); | ||
} | ||
catch (Exception e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This exception is lost
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to confirm: do we want to suppress all types of exceptions and prevent future write?
The logged msg may not be accurate as the exceptions according to API could be:
// Exceptions:
// T:System.ArgumentException:
// The sum of offset and count is greater than the buffer length.
//
// T:System.ArgumentNullException:
// buffer is null.
//
// T:System.ArgumentOutOfRangeException:
// offset or count is negative.
//
// T:System.IO.IOException:
// An I/O error occured, such as the specified file cannot be found.
//
// T:System.NotSupportedException:
// The stream does not support writing.
//
// T:System.ObjectDisposedException:
// System.IO.Stream.Write(System.Byte[],System.Int32,System.Int32) was called after
// the stream was closed.
A few more changes to come. |
And not only because I have to fix the build 😛 |
{ | ||
_outputStream.Write(block.Data.Array, block.Data.Offset, block.Data.Count); | ||
} | ||
catch (ObjectDisposedException ex) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't expect an ODE anymore, so we can do just the ConnectionError
fix.
affa092
to
53780e6
Compare
66b844e
to
e7f178e
Compare
Rebased. |
…#747). - If we're done before the client sends a FIN, force a FIN into the raw SocketInput so the task in FileteredStreamAdapter finishes gracefully and we dispose everything in proper order. - If there's an error while writing to a stream (like ObjectDisposedException), log it once and prevent further write attempts. This means the client closed the connection while we were still writing output. - This also fixes a bug related to the point above, where memory blocks were being leaked instead of returned to the pool (because we weren't catching the exception from Write()).
@@ -32,12 +34,17 @@ public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Except | |||
Console.WriteLine($"Log {logLevel}[{eventId}]: {formatter(state, exception)} {exception?.Message}"); | |||
#endif | |||
|
|||
TotalErrorsLogged++; | |||
TotalMessagesLogged++; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just remove this property, and use the now correct TotalErrorsLogged
in the test that was using it.
|
8c450fc
to
f29dd60
Compare
This PR addresses two issues that surface when using connection filters (namely our https connection filter):
#747) If the client closes the connection while we're still writing a response, we see exceptions in the log because we're trying to write to a diposed streams. Specifically with https, if a client sends an https request and closes the connection while we're still writing output, we get an exception from
SslStream.Write()
because the stream has already been disposed.Not capturing that exception is undesired because we end up leaking memory blocks and leaving them up to be collected by the GC, while ideally we should properly return all of them to the memory pool.
This is not a completely unexpected situation, so the fix is just to log that it happened and prevent further attempts to write to the disposed stream.
#737) If a client takes too long to send a FIN and in the mean time we've already consumed all input and written a response, we get an exception when trying to close the connection because we still have code awaiting socket input. This is what is happening in this issue: for whatever reason nginx takes too long to send a FIN and we attempt to close the connection before receiving it. However, the task started by
FilteredStreamAdapter.ReadInput()
is still running and awaiting for input (which given enough time would be the FIN). Since we close the connection before receiving the FIN we see the reported stack trace becauseConnection._rawSocketInput
is disposed and aborts any consumers awaiting on it.The fix is to force a FIN into
Connection._rawSocketInput
anAbort()
method inFilteredStreamAdapter
that sets a flag that we check after the task that copies data from the wrapped stream to the outerSocketInput
finishes. The previous call toIncomingData()
on_rawSocketInput
causes theCopyToAsync
task to end gracefully and by callingFilteredStreamAdapter.Abort()
we ensure the continuation for theCopyToAsync
task will just attempt to abort awaiting consumers but not log any errors.Another change is to return the continuation Task from
FilteredStreamAdapter.ReadInput()
so we can set another continuation on it to dispose otherConnection
members. By disposing those objects on that second continuation we ensure the disposal is really happening after theCopyToAsync
task has finished AND any awaiters onSocketInput
have been aborted.cc @halter73 @yuwaMSFT @Eilon