Reading is not allowed after reader was completed

[nathanmascitelli]

Hi,

We are getting the following exception intermittently in production:

at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoReadingAllowed()
at System.IO.Pipelines.Pipe.AdvanceReader(SequencePosition& consumed, SequencePosition& examined)
at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed, SequencePosition examined)
at StackExchange.Redis.PhysicalConnection.ReadFromPipe() in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 1850

When this happens its alway after we have called IDatabase.StreamAddAsync.

Looking at PhysicalConnection I'm wondering if there is a race condition between ReadFromPipe and something calling Dispose?

Does that make sense? I'm pretty sure this class is used by multiple threads and so I think the above is posslbe. Looking at the exception handler it seems like we don't even really want to report this exception?

So to sum up:

• Is this error happening due to the described race condition?
• Do we even want to report this exception up to the caller?

We have seen this happen on multiple OS, dotnet, and package versions:

• Ubuntu 20.04 and 22.04
• Dotnet 7.0.17 and 8.0.3
• StackExchange.Redis 2.7.20 and 2.7.33

Let me know if you need more info.

Thanks!

[NickCraver]

It sounds like the connection is being used post-disposal here (at which point we shouldn't be sending any more commands) - is that accurate?

[nathanmascitelli]

@NickCraver so the application has one instance of IConnectionMultiplexer which is used by multiple threads. At some point one of the threads will dispose the connection (either the app is shutting down or some configruation has changed) yes. When that happens any other threads thats currently trying to use the IConnectionMultiplexer gets this error, always when trying to write a value to Redis.

In the screenshot above it looks like we actualy handle this OK when reading but if isReading is false then we will always report an exception. Maybe we just need to change that part?

[nathanmascitelli]

Any thoughts on my last comment?

[mukhamedyarov]

Hello, I am getting the same exceptions. Are there any updates on this?

[NickCraver]

Overall here: either do not dispose the multiplexer if you're still using it or stop using it before disposal - this advice is universal to all disposals in .NET. When you dispose an object, that's very intentionally when we cleanup resources and shut things down - if something in in-flight, that's an ill-advised thing to do.

[nathanmascitelli]

@NickCraver in general I agree with you, and ultimatly we updated our code to correctly recognize that this exception is equivalent to an ObjectDisposedException and act accordingly. However, I still wonder if the catch block should be tweaked so that regardless of if a read is in progress or not we consistanltly report and exception or not if we are in ReadFromPipe and the pipe is disposed. It seems odd to me (though maybe I'm just dumb) that sometimes we would report and exception and sometimes not.

[isacruzramos]

hi, we are hitting exact same issue of trying to use a disposed multiplexer, are there any guidelines on how to properly identify when the multiplexer was disposed?

[jianze-mao]

Hi, we are also getting the same error when trying to read cache. Is there any update on the fix?

[mgravell]

Define "fix" in this context. Ultimately, for the reader to be disposed, either the multiplexer was disposed, or a connection died.

Connections do indeed occasionally die - that's a reality of sockets. We try to recover gracefully and reconnect. If the problem is that we did something ungraceful or gave a raw exception: fine, we can probably improve that by presenting a cleaner "oops, something bad happened in transit" exception, but: still ultimately an exception. If we didn't recover at all and got soft-locked: please say, so we can be clear.

If the multiplexer was used after disposing: don't do that. If it was disposed while something was in flight, and the in flight operation caused a viscous side effect: please be explicit.

So: more detail please. Rather than just saying "same error": what exactly happened, and what exactly was your doing at the time? Details matter, sorry.

[fstugren]

I see the same error in our services, and the connection multiplexer is not disposed. The service uses a shared Redis cache client with a connection multiplexer that lives for the duration of the application. When those "Reading is not allowed after reader was completed" exceptions happen the application is not shutting down - so it’s not a case of the service shutdown disposing the connection multiplexer while some requests are still processing.

The error is generally isolated and the service eventually recovers. Sometimes there's a burst of errors like this one, sometimes just one. Eventually, it appears that the Redis client successfully reconnects internally, and normal operations resume.

Is there a recommendation on what to do on the client side? We could catch this specific exception and ignore it, but the problem is that each time we run a "Get" operation that fails with this error it takes 3-5 seconds. Is there anything else the client can use to make smarter decisions?