Kubernetes Redis Cluster Returning MOVED

[mjquinlan2000]

Hello,

Recently, the in-memory cache for my application has become too large and it is causing my Kubernetes pods to be OOM killed. I suspect that the memory usage is also impacting my app performance. I am trying to use the Redis adapter as a caching solution with a Redis cluster I have set up on my k8s cluster. It's a StatefulSet with 6 nodes. I have verified that the cluster is set up properly and I am currently using it to cache some temporary location data for the apps.

The issue I am having is that when I try to write to the cache, it returns the error

** (Redix.Error) MOVED 866 10.8.17.53:6379
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter/command.ex:103: NebulexRedisAdapter.Command.handle_command_response/1
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:480: NebulexRedisAdapter.put/6
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:475: anonymous fn/7 in NebulexRedisAdapter.put/6
    (telemetry 1.0.0) /build/services/api/deps/telemetry/src/telemetry.erl:293: :telemetry.span/3
    (nebulex 2.3.1) lib/nebulex/cache/entry.ex:42: Nebulex.Cache.Entry.put/4

When I read from the cache, I get no such error.

Here is my configuration using all 6 Redis hostnames as possible master nodes.

config :my_app, MyApp.Cache,
  mode: :redis_cluster,
  master_nodes: [
    [
      host: "redis-cluster-0.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-1.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-2.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-3.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-4.redis-cluster-headless"
    ],
    [
      host: "redis-cluster-5.redis-cluster-headless"
    ]
  ],
  conn_opts: []

Any help would be greatly appreciated. Thank you

[cabol]

Hi! Well, this kind of issue is a bit tricky, usually, when I get it, it is because of something wrong with the configuration, perhaps wrong master nodes, etc. Try to validate the configuration, for example, from one of the instances if you can run MyApp.Cache.command(["CLUSTER", "SLOTS"]), and using the redis-cli you can also maybe run the same command and validate this config, it should be the same, just to discard any config issue. In the same way, you can use the redis-cli to connect to one of the nodes and try to run some read and write commands, you should see the redirected messages, validate the redirected node for a specific key with the config you got from MyApp.Cache.command(["CLUSTER", "SLOTS"]). This is for validating the config/setup is correct.

Additionally, could you provide more info to reproduce it? For example, if you can create a repo with a docker-compose or something to reproduce the issue would be great. Stay tuned!

[mjquinlan2000]

Hi @cabol,

I ended up implementing a multilevel cache with a redis fallback to a standalone redis instance as a bandaid for the situation. It seems to be working fine, but as users/data grow I'll have to address this soon. The redis image I am using in production is the one provided by bitnami. They provide a docker-compose file that I've been trying to augment to run locally:

version: '2'
services:
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-0:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-1:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-1:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-2:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-2:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-3:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-3:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-4:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-4:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-5:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-5:/bitnami/redis/data
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDISCLI_AUTH=bitnami'
      - 'REDIS_CLUSTER_REPLICAS=1'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
      - 'REDIS_CLUSTER_CREATOR=yes'

volumes:
  redis-cluster_data-0:
    driver: local
  redis-cluster_data-1:
    driver: local
  redis-cluster_data-2:
    driver: local
  redis-cluster_data-3:
    driver: local
  redis-cluster_data-4:
    driver: local
  redis-cluster_data-5:
    driver: local

It's hard to replicate what's going on with Kubernetes in the local environment unless you do something like spin up a minikube instance which is something I'm considering.

PS:

Are you tied to Redix? It's unfortunate that Redix does not support clustering. I found that there is an erlang library eredis_cluster which is being well maintained by Nordix. I don't know if it would be a heavy lift to swap out the current implementation with this one, but it's definitely something I could help out with when I get some free time. I think it would be beneficial to take the responsibility of connecting to/handling commands with redis and focusing on the adapter pattern only.

[cabol]

Hey @mjquinlan2000 !!

The redis image I am using in production is the one provided by bitnami. They provide a docker-compose file that I've been trying to augment to run locally

Will take a look at it, see if I can try to reproduce it, at least will try the Redis cache directly with the docket-compose, without the k8s.

Are you tied to Redix? It's unfortunate that Redix does not support clustering. I found that there is an erlang library eredis_cluster which is being well maintained by Nordix. I don't know if it would be a heavy lift to swap out the current implementation with this one, but it's definitely something I could help out with when I get some free time. I think it would be beneficial to take the responsibility of connecting to/handling commands with redis and focusing on the adapter pattern only.

Yeah, currently the adapter uses Redix, it can be changed of course, but that means re-write pretty much the whole adapter. What can be done is another Redis adapter but using eredis_cluster, which could be easier. But I will take a look at the eredis_cluster anyways. BTW, have you tried eredis_cluster with the k8s setup you have? does it work ok?

[mjquinlan2000]

@cabol regarding eredis_cluster, I have not tried it against k8s. I stumbled upon it when I was looking for other options and found it listed on the Redis Website here as a "recommended" library. Redix is not listed as "recommended". There is also an eredis library for the standalone version of Redis. It's also maintained by Nordix and is a dependency of eredis_cluster.

[cabol]

Hey @mjquinlan2000 👋 !!

It is been a long time but just wanted to let you know I've been working on several improvements for the Redis Cluster management. I created this issue #49 and also the implementation PR. So, whenever you can please try it out, it is on the master branch at the moment (but will push a minor release 2.3.0 soon). Thanks and I stay tuned!

[mjquinlan2000]

@cabol thank you! I was just discussing this at work and now I saw your reply. I have pulled the master branch into one of our integration pipelines and I'll run some tests.

[mjquinlan2000]

Hey @cabol, I was just testing this and it's in a much better state than when I previously opened this issue. However, we are still seeing MOVED errors when the redis cluster changes or when we deploy updates to the redis cluster:

** (Redix.Error) MOVED 866 10.79.0.218:6379
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter/command.ex:167: NebulexRedisAdapter.Command.handle_command_response/2
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:588: NebulexRedisAdapter.put/6
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:577: anonymous fn/7 in NebulexRedisAdapter.put/6
    (telemetry 1.2.1) /source/services/api/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
    (nebulex 2.4.2) lib/nebulex/cache/entry.ex:42: Nebulex.Cache.Entry.put/4
    (nebulex 2.4.2) lib/nebulex/adapters/multilevel.ex:560: anonymous fn/4 in Nebulex.Adapters.Multilevel.eval/3
    (elixir 1.13.4) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
    (nebulex 2.4.2) lib/nebulex/adapters/multilevel.ex:351: Nebulex.Adapters.Multilevel.put/6

The Redis docs explain why this happens (key rebalancing) and how clients should act accordingly. I think that when a MOVED redirect is returned, the the client should first send the request along to the redirect endpoint, update the hashslot for that key, and then rebuild the client-side hashslot mappings. I believe that is done somewhere on initialization in this library. Maybe you're already handling it somewhere else. I'm going to try and walk the code today.

I think it's really close!

I just walked your code and it looks sound to me. Then I noticed that I had set a configuration variable incorrectly for the mode: in my vars. I will be testing again soon

[mjquinlan2000]

Ok @cabol I'm no longer getting the MOVED errors.

I am still having an issue reconnecting to the cluster when a rollout operation is performed on the k8s cluster

i.e. kubectl rollout restart statefulset redis-cluster

The error I'm seeing is:

** (Redix.ConnectionError) the connection to Redis is closed
    (redix 1.2.1) lib/redix.ex:628: Redix.command!/3
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter/redis_cluster.ex:62: anonymous fn/7 in NebulexRedisAdapter.RedisCluster.exec!/5
    (elixir 1.13.4) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:857: NebulexRedisAdapter.execute_query/3
    (nebulex_redis_adapter 2.2.0) lib/nebulex_redis_adapter.ex:761: anonymous fn/5 in NebulexRedisAdapter.execute/4
    (telemetry 1.2.1) /source/services/api/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
    (nebulex 2.4.2) lib/nebulex/adapters/multilevel.ex:452: anonymous fn/6 in Nebulex.Adapters.Multilevel.execute/4
    (elixir 1.13.4) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3

I tracked down the pid with recon and it appears to be related to a Redix.Connection being disconnected in the Pool Registry. Here is the info from :recon.info/1

[
  meta: [
    registered_name: [],
    dictionary: [
      "$ancestors": [#PID<0.10370.0>, MyApp.Cache.Redis.DynamicSupervisor,
       #PID<0.6833.0>, MyApp.Cache.Redis.Supervisor, MyApp.Cache.Redis,
       MyApp.Cache.Supervisor, MyApp.Cache, MyApp.Supervisor,
       #PID<0.6711.0>],
      "$initial_call": {Redix.Connection, :init, 1}
    ],
    group_leader: #PID<0.6710.0>,
    status: :waiting
  ],
  signals: [
    links: [#PID<0.10370.0>, #PID<0.6832.0>],
    monitors: [],
    monitored_by: [],
    trap_exit: false
  ],
  location: [
    initial_call: {:proc_lib, :init_p, 5},
    current_stacktrace: [
      {:gen_statem, :loop_receive, 3, [file: 'gen_statem.erl', line: 1259]},
      {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
    ]
  ],
  memory_used: [
    memory: 26984,
    message_queue_len: 0,
    heap_size: 2586,
    total_heap_size: 3246,
    garbage_collection: [
      max_heap_size: %{error_logger: true, kill: true, size: 0},
      min_bin_vheap_size: 46422,
      min_heap_size: 233,
      fullsweep_after: 65535,
      minor_gcs: 6
    ]
  ],
  work: [reductions: 4031]
]

The state of the pid from :recon.get_state/1 is:

{:disconnected,
 %Redix.Connection{
   backoff_current: 30000,
   client_reply: :on,
   connected_address: nil,
   counter: 1,
   opts: [
     socket_opts: [],
     ssl: false,
     sync_connect: false,
     backoff_initial: 500,
     backoff_max: 30000,
     exit_on_disconnection: false,
     timeout: 5000,
     name: {:via, Registry,
      {MyApp.Cache.Redis.Registry, {{:cluster_shards, 10923, 16383}, 1}}},
     host: '10.115.2.75',
     port: 6379
   ],
   socket: #Port<0.2094>,
   socket_owner: #PID<0.11031.0>,
   table: #Reference<0.2186346826.846331908.256919>,
   transport: :gen_tcp
 }}

I'm trying to figure out a way to better monitor these connections and reconnect to the cluster, but if you run NebulexRedisAdapter.RedisCluster.ConfigManager.setup_shards(MyApp.Cache.Redis) it reconnects. However, I am thinking that there's probably a better way to just monitor for when the connection goes down and then reconnect. Let me know if you see anything here that could help me.

This is an issue for us as we will periodically be upgrading and/or restarting the redis cluster.

Thank you for the help!

[mjquinlan2000]

After some further investigation I have found that the issue is likely tied to the fact that the ip address of the redis node changes when the k8s pod is recreated. At this point, the Redix.Connection is lost and will never reconnect because it has the wrong IP address. I tried experimenting with the option exit_on_disconnect: true, but it brings down the ConfigManager when I kill master nodes and I can't find it anymore via GenServer.whereis/1.

Another interesting thing that happens: when I kill any pod running a master node, I get the same error as in the previous comment. However, if I try to run NebulexRedisAdapter.RedisCluster.ConfigManager.setup_shards(MyApp.Cache.Redis) while the pod is coming up, it doesn't seem to fix the problem. I don't know if this has something to do with the way that redis clustering works or not.

[mjquinlan2000]

@cabol I created a repository to show how you can reproduce this here: https://github.com/mjquinlan2000/nbulex_redis_adapter_test