Redis backplane using connection string doesn't seem to identify the redis endpoints

[efreeman518]

.Net 9, latest C#

Have not determined if this is a bug, but watching the VS debug output with FusionCache configured with Redis backplane using the Redis connection string shows the Redis sdk complaining about no endpoints.

Instead of using a connection string, I tried building StackExchange.Redis.ConfigurationOptions, explicitly defining the 'Endpoints' which seems to be working. Code below.

Researching - I came across this article
https://markmcgookin.com/2020/01/15/azure-redis-cache-no-endpoints-specified-error-in-dotnet-core/

config.GetSection(cacheInstance.RedisConfigurationSection).Bind(redisConfiguration);
var redisConfigurationOptions = new StackExchange.Redis.ConfigurationOptions
{
    EndPoints =
            {
                redisConfiguration.EndpointUrl,
                redisConfiguration.Port
            },
    Password = redisConfiguration.Password,
    Ssl = true,
    AbortOnConnectFail = false,
    ChannelPrefix = new StackExchange.Redis.RedisChannel(cacheInstance.BackplaneChannelName, StackExchange.Redis.RedisChannel.PatternMode.Auto)

};
//var connectionString = config.GetConnectionString(cacheInstance.RedisName);
fcBuilder
    .WithDistributedCache(new RedisCache(new RedisCacheOptions() 
    { 
        //Configuration = connectionString,  
        ConfigurationOptions = redisConfigurationOptions
    }))
    
    .WithBackplane(new RedisBackplane(new RedisBackplaneOptions
    {
        //Configuration = connectionString,
        ConfigurationOptions = redisConfigurationOptions
    }));

public class RedisConfiguration
{
    public string EndpointUrl { get; set; } = null!;
    public string Port { get; set; } = null!;
    public string Password { get; set; } = null!;
}

[jodydonetti]

Mmh, this is really strange: this is the related line, and I normally use Redis via either the connection string or the multiplexer factory.

Can you share the (redacted) connection string you're using?

[efreeman518]

//Redis Cache ----------------
"Redis1": "xyz.redis.cache.windows.net:6380,password=x..y..z..,ssl=True,abortConnect=False",

[efreeman518]

no specific endpoints defined in the connection string, I was assuming a default based on the url & port

[jodydonetti]

Ah, it's on Azure: did you allow connections to it? Depending from where you are connectin from, you may need to enable vnets, specific ips, etc.

Also, only the backplane seems not to work or also the distributed cache?

[efreeman518]

Yes Azure, public endpoint so no vnets or ip whitelist configured. Distributed cache seems to work fine, it's only when configuring the backplane the errors occur.

…

On Wed, Nov 27, 2024, 5:41 PM Jody Donetti ***@***.***> wrote: Ah, it's on Azure: did you allow connections to it? Depending from where you are connectin from, you may need to enable vnets, specific ips, etc. Also, only the backplane seems not to work or also the distributed cache? — Reply to this email directly, view it on GitHub <#335 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3S3W7DCQBOKKJSALTAPIT2CZDBVAVCNFSM6AAAAABSTUPR3CVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZZHE4DKNI> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[jodydonetti]

Hi @efreeman518 , any news on this?

[efreeman518]

no updates at the moment, will take some time to research as other tasks are taking priority.

[tankerkiller125]

This still seems to be an issue with connecting to Azure Redis Cache even with the latest version.

[efreeman518]

Update:
The connection string (with FusionCache backplane) works running Redis local in docker (Aspire).
FusionCache using Azure Redis still has a problem, so I created this settings class and config to separate the connecting string elements, this approach seems to work:

public class RedisConfiguration
{
    public string EndpointUrl { get; set; } = null!;
    public int Port { get; set; }
    public string? Password { get; set; }
}

with config that looks like:

  "RedisConfiguration": {
    "EndpointUrl": "your-redis.redis.cache.windows.net",
    "Port": 6380
    "Password": "..."
  },

then when registering services:

private static void ConfigureFusionCacheRedis(IFusionCacheBuilder fcBuilder, IConfiguration config, CacheSettings cacheSettingsInstance)
{
    // possible use the redis connection string
    if (!string.IsNullOrEmpty(cacheSettingsInstance.RedisConnectionStringName))
    {
        var redisConnectionString = config.GetConnectionString(cacheSettingsInstance.RedisConnectionStringName);
        fcBuilder
            .WithDistributedCache(new RedisCache(new RedisCacheOptions()
            {
                Configuration = redisConnectionString
            }))
            .WithBackplane(new RedisBackplane(new RedisBackplaneOptions
            {
                Configuration = redisConnectionString
            }));
    }
    //Azure seems to have a backplane issue when using redis connection string, so optionally parse into explicit config options
    else
    {
        RedisConfiguration redisConfigFusion = new();
        config.GetSection(cacheSettingsInstance.RedisConfigurationSection!).Bind(redisConfigFusion);

        if (redisConfigFusion.EndpointUrl != null)
        {
            var redisConfigurationOptions = CreateRedisConfigurationOptions(redisConfigFusion, cacheSettingsInstance.BackplaneChannelName);

            fcBuilder
                .WithDistributedCache(new RedisCache(new RedisCacheOptions()
                {
                    ConfigurationOptions = redisConfigurationOptions
                }))
                .WithBackplane(new RedisBackplane(new RedisBackplaneOptions
                {
                    ConfigurationOptions = redisConfigurationOptions
                }));
        }
        else
            throw new InvalidOperationException("Redis not configured.");
    }
}

[tankerkiller125]

What's strange about it is that I'm actually not even providing FusionCache the actual redis connection string, I'm passing a redis ConnectionMultiplexerFactory

cache.WithBackplane(new RedisBackplane(new RedisBackplaneOptions()
    {
        ConnectionMultiplexerFactory = () => Task.FromResult<IConnectionMultiplexer>(mux)
    }));

But I'll give the ConfigurationOptions thing a try, hopefully I have some luck with that.

[jodydonetti]

Hi all, thanks for the update: since it's having problems even when directly passing the IConnectionMultiplexer, it looks like something not related to FusionCache (but I may be wrong).

Let me know if this is a solution, thanks!

[tankerkiller125]

So I tried passing the ConfigurationOptions and I'm still running into same issue (at least I think it's the same overall issue as the original this discussion started with, as the error doesn't happen when the backplane is turned off)

ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6DQ K=OurCacheKey): [DC] an error occurred while getting entry from distributed
      StackExchange.Redis.RedisConnectionException: The message timed out in the backlog attempting to send because no connection became available (5000ms) - Last C
onnection Exception: UnableToConnect on "redis-ourcacheservice-location.redis.cache.windows.net:6380/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsyn
c, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.8
.31.52602, command=HMGET, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, last-in: 0, cur-in: 0, sync-ops: 3, async-ops: 5, serverEndpoint:
 "redis-ourcacheservice-location.redis.cache.windows.net:6380, conn-sec: n/a, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: HOSTNAMESERVER(SE.Redis-v2.8.31.52602), IOC
P: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=12,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=2397,Timers=10), v: 2.8.31.52602 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
       ---> StackExchange.Redis.RedisConnectionException: UnableToConnect on "redis-ourcacheservice-location.redis.cache.windows.net:6380/Interactive, Initializing/NotStarted,
 last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.8.31.52602
         --- End of inner exception stack trace ---
         at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefreshAsync(String key, Boolean getData, CancellationToken token)
         at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token)
         at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.<>c__DisplayClass16_0`1.<<TryGetEntryAsync>b__0>d.MoveNext() in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Async.cs:line 162
      --- End of stack trace from previous location ---
         at ZiggyCreatures.Caching.Fusion.Internals.RunUtils.RunAsyncFuncWithTimeoutAsync[TResult](Func`2 asyncFunc, TimeSpan timeout, Boolean cancelIfTimeout, Action`1 timedOutTaskProcessor, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/RunUtils.cs:line 39
         at ZiggyCreatures.Caching.Fusion.Internals.Distributed.DistributedCacheAccessor.TryGetEntryAsync[TValue](String operationId, String key, FusionCacheEntryOp
tions options, Boolean hasFallbackValue, Nullable`1 timeout, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/Internals/Distributed/DistributedCacheAccessor_Async.cs:line 161
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6DQ K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6DR K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6DS K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6DT K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6EA K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
fail: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6ER K=OurCacheKey): stopped auto-recovery because of an error after 1 processed items
warn: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6ES K=OurCacheKey): [DC] distributed cache activated again
warn: ZiggyCreatures.Caching.Fusion.FusionCache[0]
      FUSION [N=FusionCache I=0HNFCLC4UQ68F] (O=0HNFCLC4UQ6ES K=OurCacheKey): [DC] distributed cache temporarily de-activated for 00:00:30

I should note just in case, that the actual cache itself can communicate with redis just fine and is in fact saving keys, getting keys, etc. with the backplane turned off, there's only an issue when the backplane is turned on.

[jodydonetti]

Mmh, looking at the exception details I see:

[DC] an error occurred while getting entry from distributed
StackExchange.Redis.RedisConnectionException: The message timed out in the backlog attempting to send because no connection became available (5000ms) -
Last Connection Exception: UnableToConnect on "redis-ourcacheservice-location.redis.cache.windows.net:6380/Interactive,
Initializing/NotStarted,
last: NONE,
origin: BeginConnectAsync,
outstanding: 0,
last-read: 0s ago,
last-write: 0s ago,
keep-alive: 60s,
state: Connecting,
mgr: 10 of 10 available,
last-heartbeat: never,
global: 0s ago,
v: 2.8.31.52602,
command=HMGET,
timeout: 5000,
inst: 0,
qu: 0,
qs: 0,
aw: False,
bw: CheckingForTimeout,
last-in: 0,
cur-in: 0,
sync-ops: 3,
async-ops: 5,
serverEndpoint: "redis-ourcacheservice-location.redis.cache.windows.net:6380,
conn-sec: n/a,
aoc: 0,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: HOSTNAMESERVER(SE.Redis-v2.8.31.52602),
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=1,Free=32766,Min=12,Max=32767),
POOL: (Threads=5,QueuedItems=0,CompletedItems=2397,Timers=10),
v: 2.8.31.52602

(Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

I think we should take a look here as mentioned in the last line, to understand what is going on with the connection handling.