[elasticsearch] New readiness probes causing full cluster-restart #631
Description
Chart version: 7.7.0
Kubernetes version: 1.17.1
Kubernetes provider: On-prem
Helm Version: 3.2.0
helm get release output
Output of helm get release
USER VALUES
esConfig:
elasticsearch.yml: |
action.auto_create_index: "-hd-*"
esJavaOpts: -Xmx5g -Xms5g
esMajorVersion: 6
image: "private-image-based-on-official-oss:6.4.2"
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: some-credentials
imageTag: 6.4.2
ingress:
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
nginx.ingress.kubernetes.io/auth-secret: dev-elasticsearch-ingress-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/proxy-body-size: 60m
enabled: true
hosts:
- ""
path: /
tls:
- hosts:
- ""
lifecycle:
postStart:
exec:
command:
- bash
- -c
- |
#!/bin/bash
cd /usr/share/elasticsearch/plugins/xxx
/opt/jdk-10.0.2/bin/jar -cf config.jar config.cfg
chmod 777 config.jar
persistence:
enabled: true
podSecurityPolicy:
create: false
rbac:
create: true
resources:
limits:
cpu: 2000m
memory: 8Gi
requests:
cpu: 200m
memory: 8Gi
sysctlInitContainer:
enabled: false
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: rook-ceph-cephfs
Describe the bug:
I updated the chart to the newest version 7.7.0 and expected that the three-elastic nodes are updated one after another, waiting until the cluster is green again. (The most recent restarted pod was not ready, until the cluster was green again in the past). Now, the pod became ready after a few minutes and kubernetes moved on too quickly, so the cluster was red and not down.
Steps to reproduce:
- Update the chart-installation to 7.7.0
Expected behavior:
The readiness probe is working as expected, and mark the pod as not ready, until the cluster is green again.
Any additional context:
I did the update of my release often in the past without such problems, but always today with the new version.