RedisConnectionException: No connection is available to service this operation

[ConnorL33t]

Hi.

Recently updated our server with StackExchange.Redis v2.0.600.
About a day or two into running the library I started receiving thousands of these errors

RedisConnectionException: No connection is available to service this operation; It was not possible to connect to the redis server(s). To create a disconnected multiplexer, disable AbortOnConnectFail. ConnectTimeout; IOCP: (Busy=45,Free=955,Min=1000,Max=1000), WORKER: (Busy=43,Free=32724,Min=100,Max=32767), Local-CPU: n/a

Immediate issues were resolved by a web app restart, shortly after I reverted back to an older version of StackExchange.Redis.

Running Azure Redis Cache, ~20 client connections during these errors, ~120KiB throughput. 100Mb cache size. Redis Resource health shows "good"

[mgravell]

Hmmm. Ok. I can't get much from that error message, but it does tell me ways that I can improve that error message, i.e. track why a connection has become unusable and report it. Will hack something in but: it might need one round of better logging to know where to look for the actual fix.

…

On Thu, 11 Apr 2019, 19:25 Connor Lee, ***@***.***> wrote: Hi. Recently updated our server with StackExchange.Redis v2.0.600. About a day or two into running the library I started receiving thousands of these errors RedisConnectionException: No connection is available to service this operation; It was not possible to connect to the redis server(s). To create a disconnected multiplexer, disable AbortOnConnectFail. ConnectTimeout; IOCP: (Busy=45,Free=955,Min=1000,Max=1000), WORKER: (Busy=43,Free=32724,Min=100,Max=32767), Local-CPU: n/a Immediate issues were resolved by a web app restart, shortly after I reverted back to an older version of StackExchange.Redis. Running Azure Redis Cache, ~20 client connections during these errors, ~120KiB throughput. 100Mb cache size. Redis Resource health shows "good" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1120>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsCoxWZnlo02E625T8zPhdi7Giky9ks5vf34AgaJpZM4cqjO7> .

[ConnorL33t]

I understand, I'll update and try to get a better error message when it's on nuget.

[sksk571]

We are also getting these errors on Azure Redis. Application restart fixes them temporarily, but then they are starting again. We are on 2.0.571

[Tadimsky]

We're also seeing this error pop up recently with the v2 library.
We have to restart our services in order to mitigate the issue, but it seems to come back after an hour or so.

Did anyone make any progress on figuring out what's going wrong?

[bartdk-be]

Same error. Turning on/off the AbortOnConnectFail setting has no impact.
Any ideas what is the main issue here ?

[trpalmer]

Possibly related in 2.0.601.3402
AWS ElasticCache Redis Version 2.8.23
1,100 connections during this time with a 64 GB cache and ~35MB throughput
Process gets in this state and requires a restart

RedisConnectionException: No connection is available to service this operation: SETEX ****; 
SocketClosed (ReadEndOfStream, last-recv: 0) on ***:6379/Interactive, Idle/MarkProcessed, 
last: HSET, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 46s ago, keep-alive: 60s, 
state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, 
last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), 
WORKER: (Busy=3,Free=32764,Min=4,Max=32767), Local-CPU: n/a

[Styrna]

Any update on the issue?

We are also experiencing similar exceptions after updating to v2

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: HMSET ...; SocketClosed on .../Interactive, Idle/MarkProcessed, last: HMSET, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 55s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402; IOCP: (Busy=1,Free=999,Min=200,Max=1000), WORKER: (Busy=5,Free=32762,Min=200,Max=32767), Local-CPU: n/a

[jglinsek]

We are also experiencing these quite regularly in our Azure Cloud Services. Using version 2.0.571.

[Styrna]

Today we downgraded our dll to 2.0.5 and we are not experiencing those errors anymore

[jglinsek]

@Styrna what version specifically did you go to? As noted, we are seeing it with 2.0.571 but I see versions 2.0.505, 2.0.513 and 2.0.519 out there. Also there are some later versions 2.0.588 and 2.0.593.

[Styrna]

2.0.519.65453 is working fine for us.

[moshegutman]

Same issue for me on 2.0.601

No connection is available to service this operation: GET ...; An existing connection was forcibly closed by the remote host; IOCP: (Busy=1,Free=999,Min=2,Max=1000), WORKER: (Busy=3,Free=8188,Min=2,Max=8191), Local-CPU: n/a SocketFailure on ..., Idle/Faulted, last: GET, origin: ReadFromPipe, outstanding: 10, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402 Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. An existing connection was forcibly closed by the remote host

Not sure if this helps:

StackExchange.Redis.RedisConnectionException:
...
System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Security.SslStreamInternal+<<FillBufferAsync>g__InternalFillBufferAsync|38_0>d`1.MoveNext (System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Security.SslStreamInternal+<ReadAsyncInternal>d__34`1.MoveNext (System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Pipelines.Sockets.Unofficial.StreamConnection+AsyncStreamPipe+<CopyFromStreamToReadPipe>d__13.MoveNext (Pipelines.Sockets.Unofficial, Version=1.0.0.0, Culture=neutral, PublicKeyToken=42ea0a778e13fbe2Pipelines.Sockets.Unofficial, Version=1.0.0.0, Culture=neutral, PublicKeyToken=42ea0a778e13fbe2: C:\Code\Pipelines.Sockets.Unofficial\src\Pipelines.Sockets.Unofficial\StreamConnection.AsyncStreamPipe.csPipelines.Sockets.Unofficial, Version=1.0.0.0, Culture=neutral, PublicKeyToken=42ea0a778e13fbe2: 97)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe+DefaultPipeReader.GetResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at StackExchange.Redis.PhysicalConnection+<ReadFromPipe>d__110.MoveNext (StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46: C:\projects\stackexchange-redis\src\StackExchange.Redis\PhysicalConnection.csStackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46: 1475)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException:

[stas-vi]

Hi,

We've upgraded from 2.0.513 due to the performance issue, but now there is "No connection is available to service this operation" errors.

Any progress regarding this issue?

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: HSET MessageQueue:ProcessingListsData; IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=71,Free=1976,Min=100,Max=2047), Local-CPU: n/a at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, RedisValue hashField, RedisValue value, When when, CommandFlags flags)

[MaximKojineMojio]

Hi,

We have a similar issue with 2.0.571. There are thousands of No connection is available to service this operation in our Production environment (so it really affects Production). The issue could be resolved by restarting the service (node).

Running Azure Redis Cache, Client host: Azure VM. Clients and Server are located in the same region.

RedisConnectionException: Message: No connection is available to service this operation: XXX WWWWW; It was not possible to connect to the redis server(s). To create a disconnected multiplexer, disable AbortOnConnectFail. ConnectTimeout; IOCP: (Busy=0,Free=3000,Min=500,Max=3000), WORKER: (Busy=5,Free=2995,Min=500,Max=3000), Local-CPU: n/a

[tejasavora]

Hi,

We have a similar issue with 2.0.571. There are thousands of No connection is available to service this operation in our Production environment (so it really affects Production). The issue could be resolved by restarting the service (node).

Running Azure Redis Cache, Client host: Azure VM. Clients and Server are located in the same region.

RedisConnectionException: Message: No connection is available to service this operation: XXX WWWWW; It was not possible to connect to the redis server(s). To create a disconnected multiplexer, disable AbortOnConnectFail. ConnectTimeout; IOCP: (Busy=0,Free=3000,Min=500,Max=3000), WORKER: (Busy=5,Free=2995,Min=500,Max=3000), Local-CPU: n/a

@mgravell This is causing a huge issue for our production servers and processing. We cannot take latest Reid’s package as it has severe performance issue and now we are experiencing this issue consistently. This is completely unacceptable - as according to you, it seems more of a logging issue - I understand that you need to log from exact location - as to why connections are being dropped, but still, this is not an acceptable behavior - specially when this package is recommended by MS Azure team. We have 8 different production instances and we are experiencing this issue on all production system. Out of all these instances, largest one is a shared deployment with 6 x 13GB Redis. Off course we are trying to handle all these connection issues in our pipeline, however, doing so is causing massive delays in the system. cc @MaximKojine