SignalR Redis Cluster scale-out: Message throughput scalability

[VilleKylmamaa]

Hello,

This issue is concerning the scalability of running a Redis Cluster as a backplane for SignalR.

Here is the official documentation: https://learn.microsoft.com/en-us/aspnet/core/signalr/redis-backplane?view=aspnetcore-7.0

I noticed that the documentation regarding Redis Clustering was changed last month from:

Redis Clustering is a method for achieving high availability by using multiple Redis servers. Clustering isn't officially supported, but it might work.

to

Redis Clustering is a method for achieving high availability by using multiple Redis servers. Clustering is supported without any code modifications to the app.

While the Redis Cluster of course enables high availability, usually its purpose is also to enable horizontal scaling (for example in caching context).

Was throughput scalability testing of the backplane run with the Cluster, and is there an implementation to make the throughput scale out? I bring concern to this because Redis did Pub/Sub benchmarks with a standard Cluster and the results were that the throughput decreases with each additional node in the Cluster (source video, and source slides). If not yet implemented, horizontal scaling of the Redis backplane might be possible with the Sharded Pub/Sub.

If there is no testing or implementation, should this be mentioned in the documentation? Because if so, clustering without any code modifications will result in negative scaling of message throughput in the Cluster with each additional node.

[davidfowl]

Can you clarify what the request is here? We don’t do anything(AFAIK) to make redis cluster scale better/worse that what it already does OOTB.

We didn’t change the backplane implementation to take advantage of new sharded redis pubsub commmands. That is definitely something to look into..

[VilleKylmamaa]

The requests would be:

1. Main request. I believe that since Redis Cluster is mentioned as being supported, the documentation should also indicate to the client what are the implications of running the backplane in cluster mode. I am presuming the implications for Pub/Sub in a Redis Cluster OOTB to be that having more nodes improves availability, but there is an important trade-off in that more nodes decreases message throughput of the backplane (because all messages have to be sent to each node).

For a high availability setup with the current SignalR and Redis backplane OOTB, it might be more optimal to have an active-passive single instance Redis server setup as opposed to a cluster. That is, you use a single instance as the backplane, and in a fail-over a different single instance is promoted from passive to active, and thus there is no message propagation to N number of nodes.

2. Wishful thinking request. Feature suggestion: Support horizontal scaling in a Redis Cluster in order to achieve an increased throughput of messages as the number of nodes increases. Sharded Pub/Sub likely being a key in the implementation.

[BrennanConroy]

We looked at sharded pub/sub when checking if clustering works, but we have to wait until the redis library we're using supports it (StackExchange/StackExchange.Redis#2055).

1. the documentation should also indicate

The docs are open source 😃 feel free to send a PR to update them, or file an issue on the docs repo with some recommended content.

[mgravell]

Some relevant notes:

• at the server, in old-style pub/sub, messages are broadcast horizontally (so you can subscribe to any node), which may account for some perceived throughput impact for clusters with more shards
• the library can make its own determination; SE.Redis currently chooses (arbitrarily) to use the same shard approach as with keys, meaning that channels are routed between servers, with the intent of splitting the final distribution of messages to different channels between multiple nodes
• but as a consequence, if you only use one channel, this may have the catastrophic effect of making one server do all the work!
• this doesn't stop the underlying server broadcast behaviour, however
• we (the lib maintainers) would be happy to consider adding APIs for SSUBSCRIBE/SPUBLISH support, which is the enforced sharding and routing without the horizontal broadcast effect
• note this still doesn't improve the situation if you are only using one channel
• note that this precludes using wildcards; there is no sharded equivalent to PSUBSCRIBE, so you can't subscribe to glob-based channel patterns
• SSUBSCRIBE/SPUBLISH are dependent on redis 7; I don't know (but will find out) whether azure redis cache currently supports redis 7
• for purely pub/sub scenarios, I would agree that it may be preferable to use non-cluster redis; in most pub/sub scenarios I've looked at, a single redis node is more than capable of keeping up with a large number of redis clients (sharding is primarily useful to increase the total memory capacity of the database by distributing it between multiple servers; memory capacity is not an issue for pub/sub)

[BrennanConroy]

Docs were updated, thanks @VilleKylmamaa!
Closing the issue. We will likely want to revist when SE.Redis adds SSUBSCRIBE and SPUBLISH.