Intermittent error: An attempt was made to access a socket in a way forbidden by its access permissions

[JohnElliffe]

We are getting the following issue intermittently:

An invocation attempt failed. Retries remaining: 5.
Exception:
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Jering.Javascript.NodeJS.HttpNodeJSService.TryInvokeAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.TryInvokeCoreAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)

We have a .Net Core 3.1 web application running on an Azure Windows App Service Server - Web sockets are Off, NodeJs version 14.15.0.
As well as hosting our website this server also uses JeringTech Javascript.NodeJS to run Javascript to create PDFs.
Currently it runs well for around 5 days and then throws the error above. We need to restart the app server to start things up again. The microsoft nodeservices package gave the same behaviour.
Our configuration is simple - the Startup.cs file contains:
services.AddNodeJS();
services.Configure(options => options.NumRetries = 5);

[JeremyTCD]

Hey thanks for reporting the issue!

At present the library uses HTTP requests to communicate with Node.js.
With this approach there've been several issues with intermittent socket problems.
Having taken a look at these issues, and considering that they occur with Microsoft's INodeService, I think there are likely issues in .Net/Node.js framework logic. These issues are beyond what we can realistically fix.

Short of updating the library to use some alternative protocol like websockets or grpc, the best workaround for now is to improve our retry logic. You mention that you have to restart the application after a failed invocation.

An invocation attempt failed. Retries remaining: 5.

Does the program crash/hang after this? Or does it count down retries till all are exhausted? If it crashes, are any further exceptions logged?

Edit: I'm updating the library to use HTTP/2 by default. Might solve socket issues since HTTP/2 multiplexes parallel requests over the same connection.

[JohnElliffe]

Thanks for replying Jeremy.

We have Retries set to 5. The node js service tries 5 times, logging the exception each time and then stops as expected.
Subsequent calls to this function also try 5 times and stop -- until we restart the app server and then everything is OK till the next time.
Would there be any point in increasing the retry number?

We will try using MultiProcess but I am worried this will hide the problem rather than fixing it.

[JeremyTCD]

Sure, socket issues have been the bane of this library, am happy to get additional information that could help resolve them.

Anyway I updated my comment:

I'm updating the library to use HTTP/2 by default. Might solve socket issues since HTTP/2 multiplexes parallel requests over the same connection.

I suspect switching to HTTP/2 has a non-negligible chance of fixing socket issues. Now that .Net core 3 supports HTTP/2 it's at least worth a try.

Will publish a new package today, if issues persist after this change we can look more closely at retries.

[JohnElliffe]

Thanks Jeremy, we will give the new package a try when it is ready.

[JeremyTCD]

Hey I've published 6.0.0-beta.0. It uses HTTP/2 to communicate with node. Let me know how it goes!

[JohnElliffe]

We have deployed 6.0.0-beta.0 to our Dev and Test environments. I will keep an eye on it over the coming week.

[JeremyTCD]

Hi @JohnElliffe have you had any trouble with 6.0.0-beta.0?

[JohnElliffe]

Hi @JeremyTCD, I was just about to let you know that we haven't had a problem with 6.0.0-beta.0 when we received:

An invocation attempt failed. Retries remaining: 8.
Exception:
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Jering.Javascript.NodeJS.HttpNodeJSService.TryInvokeAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.TryInvokeCoreAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)

After killing the Node service processing begins again as normal.

I note that this call passes through some Microsoft.Extensions.Http.Logging methods - Could our problems be related to logging? We use console.log to log within the JavaScript and Azure logging from C#.

[JeremyTCD]

Thanks for the information!

I don't think it's logging.

This library communicates with Node.js over a HTTP connection. The Node.js process starts a HTTP server on a port of its choice and the C# process sends messages to it. The SocketException you're getting suggests that the port we're using to communicate with Node.js is either in-use by another process or blocked (could be by your firewall or OS).

I'm shooting in the dark here, but these seem like plausible causes of your exception:

• The Node.js server died and was restarted on a port that is blocked.
• Something is changing port permissions on your system, i.e. Node.js server didn't die but the port it was using got blocked.

I'll try 2 things:

• Log port used by Node.js server when we connect/reconnect. This way you can look into what is affecting the port.
• As you mentioned "After killing the Node service processing begins again as normal.". I'll build this into the library to improve its robustness, i.e. auto-restart the Node.js process on SocketException.

[JohnElliffe]

Thanks, they'll be a couple of useful improvements.
From my experience, if you are able to kill the Node process the next TryInvokeCoreAsync invocation will create a new one.

[JeremyTCD]

Hey I've published 6.0.0-beta.2. I've added a new option, OutOfProcessNodeJSServiceOptions.NumProcessRetries. Basically if retries in the original NodeJS process fail, a new process is created and invocations are retried there, so you don't need to manually kill NodeJS processes:

Javascript.NodeJS/src/NodeJS/NodeJSServiceImplementations/OutOfProcess/OutOfProcessNodeJSServiceOptions.cs

Lines 27 to 39 in b96c2f9

	/// <summary>
	/// <para>The number of times we create a new NodeJS process to retry an invocation.</para>
	/// <para>Invocations are retried <see cref="NumRetries"/> times in the existing NodeJS process. Once <b>existing process retries</b> are exhausted,
	/// if any <b>process retries</b> remain, a new NodeJS process is created and invocations are tried <see cref="NumRetries"/> times in the new process.</para>
	/// <para>For example, consider the situation where <see cref="NumRetries"/> and this value are both 1. An invocation is first tried in the existing process.
	/// If it fails it is retried in the same process. If it fails again, a new process is created and the invocation is retried there once. In total, the
	/// invocation is attempted 3 times.</para>
	/// <para>If set to a negative value, new NodeJS processes are recreated for retries indefinitely.</para>
	/// <para>If the module source of an invocation is an unseekable stream, the invocation is not retried.
	/// If you require retries for such streams, copy their contents to a <see cref="MemoryStream"/>.</para>
	/// <para>Defaults to 1.</para>
	/// </summary>
	public int NumProcessRetries { get; set; } = 1;

Also, on connecting to a NodeJS process, port number is now logged at LogLevel.Information level.

Let me know how it goes!

[JohnElliffe]

Great, thanks Jeremy, I give the new version a try.

[JohnElliffe]

Hi Jeremy, Your latest change works well.
This morning we had the error invoking Node. After the number of re-tries was exhausted a connection was made on a new port and the request was processed successfully.
Thank you very much for all your help.