Skip to content
This repository was archived by the owner on Dec 18, 2018. It is now read-only.

Fix connection termination issues when using connection filters (#737, #747). #750

Merged
merged 1 commit into from
Apr 20, 2016

Conversation

cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Apr 14, 2016

This PR addresses two issues that surface when using connection filters (namely our https connection filter):
#747) If the client closes the connection while we're still writing a response, we see exceptions in the log because we're trying to write to a diposed streams. Specifically with https, if a client sends an https request and closes the connection while we're still writing output, we get an exception from SslStream.Write() because the stream has already been disposed.

Not capturing that exception is undesired because we end up leaking memory blocks and leaving them up to be collected by the GC, while ideally we should properly return all of them to the memory pool.

This is not a completely unexpected situation, so the fix is just to log that it happened and prevent further attempts to write to the disposed stream.
#737) If a client takes too long to send a FIN and in the mean time we've already consumed all input and written a response, we get an exception when trying to close the connection because we still have code awaiting socket input. This is what is happening in this issue: for whatever reason nginx takes too long to send a FIN and we attempt to close the connection before receiving it. However, the task started by FilteredStreamAdapter.ReadInput() is still running and awaiting for input (which given enough time would be the FIN). Since we close the connection before receiving the FIN we see the reported stack trace because Connection._rawSocketInput is disposed and aborts any consumers awaiting on it.

The fix is to force a FIN into Connection._rawSocketInput an Abort() method in FilteredStreamAdapter that sets a flag that we check after the task that copies data from the wrapped stream to the outer SocketInput finishes. The previous call to IncomingData() on _rawSocketInput causes the CopyToAsync task to end gracefully and by calling FilteredStreamAdapter.Abort() we ensure the continuation for the CopyToAsync task will just attempt to abort awaiting consumers but not log any errors.

Another change is to return the continuation Task from FilteredStreamAdapter.ReadInput() so we can set another continuation on it to dispose other Connection members. By disposing those objects on that second continuation we ensure the disposal is really happening after the CopyToAsync task has finished AND any awaiters on SocketInput have been aborted.

cc @halter73 @yuwaMSFT @Eilon

{
_outputStream.Write(block.Data.Array, block.Data.Offset, block.Data.Count);
}
catch (Exception e)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exception is lost

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: do we want to suppress all types of exceptions and prevent future write?
The logged msg may not be accurate as the exceptions according to API could be:
// Exceptions:
// T:System.ArgumentException:
// The sum of offset and count is greater than the buffer length.
//
// T:System.ArgumentNullException:
// buffer is null.
//
// T:System.ArgumentOutOfRangeException:
// offset or count is negative.
//
// T:System.IO.IOException:
// An I/O error occured, such as the specified file cannot be found.
//
// T:System.NotSupportedException:
// The stream does not support writing.
//
// T:System.ObjectDisposedException:
// System.IO.Stream.Write(System.Byte[],System.Int32,System.Int32) was called after
// the stream was closed.

@cesarblum
Copy link
Contributor Author

A few more changes to come.

@cesarblum
Copy link
Contributor Author

And not only because I have to fix the build 😛

{
_outputStream.Write(block.Data.Array, block.Data.Offset, block.Data.Count);
}
catch (ObjectDisposedException ex)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't expect an ODE anymore, so we can do just the ConnectionError fix.

@cesarblum cesarblum force-pushed the cesarbs/dont-write-to-disposed-stream branch from affa092 to 53780e6 Compare April 18, 2016 18:13
@cesarblum
Copy link
Contributor Author

@Eilon @halter73 Updated first comment in this PR explaining the issues and fixes.

@cesarblum cesarblum force-pushed the cesarbs/dont-write-to-disposed-stream branch from 66b844e to e7f178e Compare April 19, 2016 23:22
@cesarblum
Copy link
Contributor Author

Rebased.

…#747).

- If we're done before the client sends a FIN, force a FIN into the raw
  SocketInput so the task in FileteredStreamAdapter finishes gracefully
  and we dispose everything in proper order.
- If there's an error while writing to a stream (like ObjectDisposedException),
  log it once and prevent further write attempts. This means the client closed
  the connection while we were still writing output.
- This also fixes a bug related to the point above, where memory blocks were
  being leaked instead of returned to the pool (because we weren't catching
  the exception from Write()).
@@ -32,12 +34,17 @@ public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Except
Console.WriteLine($"Log {logLevel}[{eventId}]: {formatter(state, exception)} {exception?.Message}");
#endif

TotalErrorsLogged++;
TotalMessagesLogged++;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just remove this property, and use the now correct TotalErrorsLogged in the test that was using it.

@halter73
Copy link
Member

:shipit:

@cesarblum cesarblum force-pushed the cesarbs/dont-write-to-disposed-stream branch from 8c450fc to f29dd60 Compare April 20, 2016 21:55
@cesarblum cesarblum changed the title Prevent writes to disposed output stream (#747). Fix connection termination issues when using connection filters (#737, #747). Apr 20, 2016
@cesarblum cesarblum merged commit f29dd60 into release Apr 20, 2016
@cesarblum cesarblum deleted the cesarbs/dont-write-to-disposed-stream branch April 20, 2016 22:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants