[stable/redis] Sentinel helm upgrades cause CrashLoopBackOff for last slave

[mdeggies]

Describe the bug
A clear and concise description of what the bug is.

When we upgrade anything in our sentinel chart using helm upgrade, the last slave's pod becomes unhealthy. In the below repro example, I do a deployment and then upgrade the deployment by adding resource/request limits. This causes the following error: Back-off 5m0s restarting failed container=sentinel pod=michele-test-redis-slave-2_default(d9383410-8d49-11e9-b7f3-42010ab404ba): CrashLoopBackOff.

Version of Helm and Kubernetes:

$ helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

K8s cluster version: 1.12.6-gke.11

Which chart:

stable/redis

What happened:

The last slave's sentinel container becomes unhealthy, causing the pod to enter CrashLoopBackOff mode instead of Running mode.

What you expected to happen:

Upgrading the chart via helm upgrade should do a seamless upgrade.

How to reproduce it (as minimally and precisely as possible):

• Copy the following into chart.yaml:

cluster:
  slaveCount: 3

password: test

metrics:
  enabled: false

image:
  tag: 5.0.5-debian-9-r14

sentinel:
  enabled: true
  masterSet: "michele-test-redis-master"
  image:
    tag: 5.0.5-debian-9-r14
  # resources:
  #   requests:
  #     memory: 128Mi
  #     cpu: 100m
  #   limits:
  #     memory: 128Mi

master:
  persistence:
    enabled: false
  nodeSelector:
    service: "ads-shared-tools"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "ads-shared-tools"
      effect: "NoSchedule"

slave:
  persistence:
    enabled: false
  nodeSelector:
    service: "ads-shared-tools"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "ads-shared-tools"
      effect: "NoSchedule"

• Run helm install --name michele-test -f chart.yaml stable/redis

• See that everything works fine. All pods are running, etc

• Edit chart.yaml with the following, noting that the only difference between these charts is the addition of resource/request limits:

cluster:
  slaveCount: 3

password: test

metrics:
  enabled: false

image:
  tag: 5.0.5-debian-9-r14

sentinel:
  enabled: true
  masterSet: "michele-test-redis-master"
  image:
    tag: 5.0.5-debian-9-r14
  resources:
    requests:
      memory: 128Mi
      cpu: 100m
    limits:
      memory: 128Mi

master:
  persistence:
    enabled: false
  nodeSelector:
    service: "ads-shared-tools"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "ads-shared-tools"
      effect: "NoSchedule"

slave:
  persistence:
    enabled: false
  nodeSelector:
    service: "ads-shared-tools"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "ads-shared-tools"
      effect: "NoSchedule"

• Upgrade the chart with helm upgrade michele-test -f chart.yaml stable/redis to break things

• See that the last slave's sentinel container is in CrashLoopBackOff mode (pics below show the service and pod affected):

Anything else we need to know:

[mdeggies]

Update: Just as an fyi, I checked and do not see any issues when doing the same upgrade in cluster mode.

[andresbono]

Hi @mdeggies, we can reproduce the issue that you are describing. Just to confirm, can you share with us the logs of the sentinel container that is crashing? You can execute:

kubectl logs michele-test-redis-slave-2 sentinel

[mdeggies]

Yeah, here are the logs from sentinel michele-test-redis-slave-2:

$ kubectl logs michele-test-redis-slave-2 sentinel
Getting information about current running sentinels
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

sentinel known-sentinel michele-test-redis-master 10.104.36.55 26379 9eee1c38364e43ca1446d8819077c05991d6f403
sentinel known-sentinel michele-test-redis-master 10.104.36.54 26379 766d717d2e0f3524892040d101ab8911b73f9ce3

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 4
>>> 'sentinel monitor michele-test-redis-master michele-test-redis-master-0.michele-test-redis-headless.default.svc.cluster.local 6379 2'
Can't resolve master instance hostname.
sentinel known-sentinel michele-test-redis-master 10.104.93.52 26379 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3

kubectl logs michele-test-redis-slave-1 sentinel

$ kubectl logs michele-test-redis-slave-1 sentinel
Getting information about current running sentinels
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

sentinel known-sentinel michele-test-redis-master 10.104.93.52 26379 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc31:X 13 Jun 2019 16:50:59.071 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 13 Jun 2019 16:50:59.071 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 13 Jun 2019 16:50:59.071 # Configuration loaded
1:X 13 Jun 2019 16:50:59.073 * Running mode=sentinel, port=26379.
1:X 13 Jun 2019 16:50:59.073 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 13 Jun 2019 16:50:59.077 # Sentinel ID is e70f5b7441b0e8e589a1073329e6986a6382e576
1:X 13 Jun 2019 16:50:59.077 # +monitor master michele-test-redis-master 10.104.36.54 6379 quorum 2
1:X 13 Jun 2019 16:50:59.080 * +slave slave 10.104.93.52:6379 10.104.93.52 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:50:59.082 * +slave slave 10.104.26.92:6379 10.104.26.92 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:00.154 * +sentinel sentinel 766d717d2e0f3524892040d101ab8911b73f9ce3 10.104.36.54 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:09.114 * +slave slave 10.104.36.55:6379 10.104.36.55 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:10.025 * +sentinel sentinel 9eee1c38364e43ca1446d8819077c05991d6f403 10.104.36.55 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:19.222 * +fix-slave-config slave 10.104.26.92:6379 10.104.26.92 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.177 # +sdown master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.177 # +sdown sentinel 766d717d2e0f3524892040d101ab8911b73f9ce3 10.104.36.54 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.239 # +odown master michele-test-redis-master 10.104.36.54 6379 #quorum 2/2
1:X 13 Jun 2019 16:52:23.239 # +new-epoch 1
1:X 13 Jun 2019 16:52:23.239 # +try-failover master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.243 # +vote-for-leader e70f5b7441b0e8e589a1073329e6986a6382e576 1
1:X 13 Jun 2019 16:52:23.243 # +sdown sentinel 9eee1c38364e43ca1446d8819077c05991d6f403 10.104.36.55 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.254 # 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 voted for e70f5b7441b0e8e589a1073329e6986a6382e576 1
1:X 13 Jun 2019 16:52:23.371 # +sdown slave 10.104.36.55:6379 10.104.36.55 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:34.098 # -failover-abort-not-elected master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:34.156 # Next failover delay: I will not start a failover before Thu Jun 13 16:53:00 2019
1:X 13 Jun 2019 16:52:59.976 # +new-epoch 2
1:X 13 Jun 2019 16:52:59.978 # +vote-for-leader 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 2
1:X 13 Jun 2019 16:53:00.037 # Next failover delay: I will not start a failover before Thu Jun 13 16:53:36 2019

kubectl logs michele-test-redis-slave-0 sentinel

$ kubectl logs michele-test-redis-slave-0 sentinel
Getting information about current running sentinels
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1:X 13 Jun 2019 16:50:54.039 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 13 Jun 2019 16:50:54.039 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 13 Jun 2019 16:50:54.039 # Configuration loaded
1:X 13 Jun 2019 16:50:54.041 * Running mode=sentinel, port=26379.
1:X 13 Jun 2019 16:50:54.041 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 13 Jun 2019 16:50:54.044 # Sentinel ID is 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3
1:X 13 Jun 2019 16:50:54.044 # +monitor master michele-test-redis-master 10.104.36.54 6379 quorum 2
1:X 13 Jun 2019 16:50:54.047 * +slave slave 10.104.93.52:6379 10.104.93.52 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:50:54.148 * +sentinel sentinel 766d717d2e0f3524892040d101ab8911b73f9ce3 10.104.36.54 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:01.087 * +sentinel sentinel e70f5b7441b0e8e589a1073329e6986a6382e576 10.104.26.92 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:04.104 * +slave slave 10.104.26.92:6379 10.104.26.92 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:10.025 * +sentinel sentinel 9eee1c38364e43ca1446d8819077c05991d6f403 10.104.36.55 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:14.186 * +slave slave 10.104.36.55:6379 10.104.36.55 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:14.277 * +fix-slave-config slave 10.104.93.52:6379 10.104.93.52 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:51:36.650 * +sentinel sentinel bbfcb75265f3c0afaf95310b12f748634b3e5f2a 10.104.36.57 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.167 # +sdown master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.168 # +sdown sentinel 766d717d2e0f3524892040d101ab8911b73f9ce3 10.104.36.54 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.168 # +sdown sentinel 9eee1c38364e43ca1446d8819077c05991d6f403 10.104.36.55 26379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:23.249 # +new-epoch 1
1:X 13 Jun 2019 16:52:23.253 # +vote-for-leader e70f5b7441b0e8e589a1073329e6986a6382e576 1
1:X 13 Jun 2019 16:52:23.327 # +sdown slave 10.104.36.55:6379 10.104.36.55 6379 @ michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:24.303 # +odown master michele-test-redis-master 10.104.36.54 6379 #quorum 2/2
1:X 13 Jun 2019 16:52:24.303 # Next failover delay: I will not start a failover before Thu Jun 13 16:52:59 2019
1:X 13 Jun 2019 16:52:59.966 # +new-epoch 2
1:X 13 Jun 2019 16:52:59.966 # +try-failover master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:52:59.970 # +vote-for-leader 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 2
1:X 13 Jun 2019 16:52:59.979 # e70f5b7441b0e8e589a1073329e6986a6382e576 voted for 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 2
1:X 13 Jun 2019 16:53:10.631 # -failover-abort-not-elected master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:53:10.688 # Next failover delay: I will not start a failover before Thu Jun 13 16:53:36 2019
1:X 13 Jun 2019 16:53:36.216 # +new-epoch 3
1:X 13 Jun 2019 16:53:36.219 # +vote-for-leader e70f5b7441b0e8e589a1073329e6986a6382e576 3
1:X 13 Jun 2019 16:53:36.279 # Next failover delay: I will not start a failover before Thu Jun 13 16:54:12 2019
1:X 13 Jun 2019 16:54:12.440 # +new-epoch 4
1:X 13 Jun 2019 16:54:12.440 # +try-failover master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:54:12.444 # +vote-for-leader 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 4
1:X 13 Jun 2019 16:54:12.454 # e70f5b7441b0e8e589a1073329e6986a6382e576 voted for 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 4
1:X 13 Jun 2019 16:54:22.762 # -failover-abort-not-elected master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:54:22.818 # Next failover delay: I will not start a failover before Thu Jun 13 16:54:48 2019
1:X 13 Jun 2019 16:54:48.793 # +new-epoch 5
1:X 13 Jun 2019 16:54:48.793 # +try-failover master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:54:48.796 # +vote-for-leader 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 5
1:X 13 Jun 2019 16:54:48.822 # e70f5b7441b0e8e589a1073329e6986a6382e576 voted for 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc3 5
1:X 13 Jun 2019 16:54:59.179 # -failover-abort-not-elected master michele-test-redis-master 10.104.36.54 6379
1:X 13 Jun 2019 16:54:59.235 # Next failover delay: I will not start a failover before Thu Jun 13 16:55:25 2019

kubectl logs michele-test-redis-master-0 sentinel

$ kubectl logs michele-test-redis-master-0 sentinel
Getting information about current running sentinels
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

sentinel known-sentinel michele-test-redis-master 10.104.36.55 26379 9eee1c38364e43ca1446d8819077c05991d6f403
sentinel known-sentinel michele-test-redis-master 10.104.36.54 26379 766d717d2e0f3524892040d101ab8911b73f9ce3
sentinel known-sentinel michele-test-redis-master 10.104.93.52 26379 7d3b60b9ba6b7d4f7f7f3db36d8ff68a69c8efc31:X 13 Jun 2019 16:51:34.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 13 Jun 2019 16:51:34.631 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 13 Jun 2019 16:51:34.631 # Configuration loaded
1:X 13 Jun 2019 16:51:34.633 * Running mode=sentinel, port=26379.
1:X 13 Jun 2019 16:51:34.633 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 13 Jun 2019 16:51:34.637 # Sentinel ID is bbfcb75265f3c0afaf95310b12f748634b3e5f2a
1:X 13 Jun 2019 16:51:34.637 # +monitor master michele-test-redis-master 10.104.36.57 6379 quorum 2
1:X 13 Jun 2019 16:51:44.710 * +slave slave 10.104.36.56:6379 10.104.36.56 6379 @ michele-test-redis-master 10.104.36.57 6379
1:X 13 Jun 2019 16:52:04.841 * +fix-slave-config slave 10.104.36.56:6379 10.104.36.56 6379 @ michele-test-redis-master 10.104.36.57 6379
1:X 13 Jun 2019 16:52:23.602 # +new-epoch 1
1:X 13 Jun 2019 16:52:34.645 # +sdown sentinel 766d717d2e0f3524892040d101ab8911b73f9ce3 10.104.36.54 26379 @ michele-test-redis-master 10.104.36.57 6379
1:X 13 Jun 2019 16:52:34.645 # +sdown sentinel 9eee1c38364e43ca1446d8819077c05991d6f403 10.104.36.55 26379 @ michele-test-redis-master 10.104.36.57 6379
1:X 13 Jun 2019 16:53:00.297 # +new-epoch 2
1:X 13 Jun 2019 16:53:36.842 # +new-epoch 3
1:X 13 Jun 2019 16:54:13.500 # +new-epoch 4
1:X 13 Jun 2019 16:54:50.001 # +new-epoch 5
1:X 13 Jun 2019 16:55:26.742 # +new-epoch 6
1:X 13 Jun 2019 16:56:03.535 # +new-epoch 7
1:X 13 Jun 2019 16:56:38.231 # +new-epoch 8

[andresbono]

Thanks for the information. We will investigate the issue and we will get back to you.

[sameersbn]

I found that in this case, the master and slave gets re-created at the same time and since the sentinel configuration is not persisted, the slave goes into a crashloop because we bungle up the config by trying to manually discover the sentinels running in the cluster. The redis-sentinel auto-discovers slaves running in the cluster and this manual discovery is not required.

I've submitted the fix for this issue in #15005.

ref:

• https://redis.io/topics/sentinel#configuring-sentinel

edit:

• added references