WCF client will hung if consume NettcpBinding service with Async then sync

[ZhaodongTian]

Repro :

1. Create WCF NettcpBinding target with 481
2. Create WCF client target with .netcore6.0 or higher
3. Call the servcie from client project like below:
   Console.WriteLine($"Get async result:{await client.HelloAsync("helloworld")}");
   Console.WriteLine($"Get sync result:{client.Hello("helloworld")}");

if the client target with full framework481, issue not repro

Expected Results:

we can get both the msg in the output

Actual Results
only get the Async result output

Timeout Exception:

Unhandled exception. System.TimeoutException: This request operation sent to net.tcp://localhost:8089/nettcp did not receive a reply within the configured timeout (00:04:59.9980591). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
at System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(MethodCall methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(MethodInfo targetMethod, Object[] args)
--- End of stack trace from previous location ---
at System.Reflection.DispatchProxyGenerator.Invoke(Object[] args)
at generatedProxy_1.Hello(String )
at NetCoreClient.Program.Main(String[] args) in F:\NetTcpIssue\NetCoreClient\Program.cs:line 20

NetTcpIssue.zip

[mconnew]

I've root caused the problem, and this is an interesting one. It should be a small fix and there's an immediate workaround you can do.
When doing a sync call, as long as you don't have any other outstanding calls, it will attempt to retrieve the reply message inline on the same thread, which blocks (which is okay with a sync method call). When you make an async call, it registers the request message in a correlator and kicks off a background async pump which receives messages. When a new incoming message is received, the relates to field is used to lookup the outstanding request in the correlator and if there is, notify it of the reply message. The bug is that the thread which calls to notify that a reply has been received ends up executing the completion of the async service call. The next method call (the synchronous call) is still executing on the same thread as the async pump was executing on so it's not able to start the next loop iteration to receive the next message. The fix is to complete the outstanding request asynchronously so that the pump can continue on to receive the next message.

The simple workaround is to await Task.Yield(); after the async call so that the async message pump gets its thread back and can kick off the next receive loop. Here's what that would look like in your repro app:

            Console.WriteLine($"Get sync result:{client.Hello("helloworld")}");
            Console.WriteLine($"Get async result:{await client.HelloAsync("helloworld")}");
            await Task.Yield();
            Console.WriteLine($"Get sync result:{client.Hello("helloworld")}");

The sync call is able to complete because the async pump is able to continue waiting for the next message when you yield the thread.

[bjornbouetsmith]

@mconnew I seem to still be having this issue - not in NET6, but when a NET48 client is doing this exact pattern.

i.e. first call async method, then call sync method.

And inserting a Task.Yield() fixes the issue as suggested.

Did the fix not get ported to the netframework of the client?

[mconnew]

@bjornbouetsmith, the implementation is a lot different on .NET Framework so a back port isn't simply a case of copy/paste. Are you sure this bug exists there for Task based methods? I verified this problem exists for APM methods on .NET Framework which is why I haven't fixed APM methods in .NET as that's not a regression. I thought Task's were fine in .NET Framework which is why I fixed it in .NET, as it's a regression.
Can you provide a call stack when the thread is blocked? The key thing that you'll see if the bug exists there is further up the call stack from the blocked sync call you'll see either SyncDuplexRequest.GotReply or AsyncDuplexRequest.GotReply which is waiting to return. This is what the await Task.Yield() does, allows the call stack to unwind.

[bjornbouetsmith]

@bjornbouetsmith, the implementation is a lot different on .NET Framework so a back port isn't simply a case of copy/paste. Are you sure this bug exists there for Task based methods?

@mconnew

This is the stack trace:

System.ServiceModel.Internals.dll!System.Runtime.TimeoutHelper.WaitOne
mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Line 251
mscorlib.dll!System.Threading.WaitHandle.WaitOne(System.TimeSpan timeout, bool exitContext) Line 187
System.ServiceModel.Internals.dll!System.Runtime.TimeoutHelper.WaitOne(System.Threading.WaitHandle waitHandle, System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply(System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call(string action, bool oneway, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation, object[] ins, object[] outs, System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage methodCall, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage message)
mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(ref System.Runtime.Remoting.Proxies.MessageData msgData, int type) Line 590
Client.exe!Client.Program.Main(string[] args) Line 24
[Resuming Async Method]
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 515
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 486
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Line 36
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Line 142
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0() Line 158
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask) Line 200
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations() Line 3147
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 305
mscorlib.dll!System.Threading.Tasks.TaskFactory<System.__Canon>.FromAsyncCoreLogic(System.IAsyncResult iar, System.Func<System.IAsyncResult, System.__Canon> endFunction, System.Action<System.IAsyncResult> endAction, System.Threading.Tasks.Task<System.__Canon> promise, bool requiresSynchronization) Line 385
mscorlib.dll!System.Threading.Tasks.TaskFactory<string>.FromAsyncImpl.AnonymousMethod__0(System.IAsyncResult iar) Line 865
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(System.IAsyncResult result, bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.SendCallback(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DuplexChannelBinder.AsyncDuplexRequest.Done(bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DuplexChannelBinder.AsyncDuplexRequest.GotReply(System.ServiceModel.Channels.Message reply)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DuplexChannelBinder.HandleRequestAsReplyCore(System.ServiceModel.Channels.Message message)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequestAsReply(System.ServiceModel.Channels.RequestContext request)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext request, System.ServiceModel.OperationContext currentOperationContext)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult result)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result)
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)
System.ServiceModel.dll!System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(object state)
System.ServiceModel.dll!System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(object state)
System.dll!System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs e) Line 8788
System.dll!System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(System.Net.Sockets.SocketError socketError, int bytesTransferred, System.Net.Sockets.SocketFlags flags) Line 10024
System.dll!System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) Line 10039
mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) Line 51

Scenario is:
Create a WCF Server with both a sync and async task based method:

    [ServiceContract]
    public interface IHello
    {
        [OperationContract(Name ="Hello")]
        string HelloWorld(string who);

        [OperationContract(Name = "HelloAsync")]
        public Task<string> HelloWorldAsync(string who);
    }

Create client using built in NET48 assemblies from System.ServiceModel

static async Task Main(string[] args)
{
    Console.WriteLine("Press enter when service is up");
    Console.ReadLine();

    var factory = new ChannelFactory<IHello>(new NetTcpBinding(SecurityMode.None), new EndpointAddress("net.tcp://localhost:8090/hello"));

    var channel = factory.CreateChannel();

    var comm = ((ICommunicationObject)channel);
    Console.WriteLine(comm.State);
    comm.Open();

    var result = await channel.HelloWorldAsync("Testing async 1");
    Console.WriteLine(result);

    result = channel.HelloWorld("Testing sync");
    Console.WriteLine(result);
    result = await channel.HelloWorldAsync("Testing async 2");

    Console.WriteLine(result);
}

The result = channel.HelloWorld("Testing sync"); never returns and hangs in the strack trace above

Adding Task.Yield() after the first HelloWorldAsync fixes the issue.

Also same happens if you remove the explicit reference in the csproj to System.ServiceMode and e.g. reference the nuget package "System.ServiceMode.NetTcp" (4.9.0) from this repo - since that uses the same System.ServiceModel from the framework