[elasticsearch] Keystore integration

Closes: #90

Adds a kubernetes native way to add strings and files to the
Elasticsearch keystore.

Previously you needed to manually create the keystore and upload
it as a secret. There were a couple of issues with this approach.

1. The Elasticsearch keystore has an internal version for the format. If
this is changed it meant needing to recreate each keystore again.

2. If you wanted to add a single new value it meant recreating the
entire keystore again

elasticsearch/README.md

@@ -111,6 +111,7 @@ helm install --name elasticsearch elastic/elasticsearch --set imageTag=7.3.0
 | `schedulerName`               | Name of the [alternate scheduler](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods)                                                                                                                                                                          | `nil`                                                                                                                     |
 | `masterTerminationFix`        | A workaround needed for Elasticsearch < 7.2 to prevent master status being lost during restarts [#63](https://github.com/elastic/helm-charts/issues/63)                                                                                                                                                                    | `false`                                                                                                                   |
 | `lifecycle`                   | Allows you to add lifecycle configuration. See [values.yaml](./values.yaml) for an example of the formatting.                                                                                                                                                                                                              | `{}`                                                                                                                      |
+| `keystore`                    | Allows you map Kubernetes secrets into the keystore. See the [config example](/elasticsearch/examples/config/values.yaml) and [how to use the keystore](#how-to-use-the-keystore)                                                                                                                                          | `[]`                                                                                                                      |
 
 ## Try it out
 
@@ -171,18 +172,55 @@ There are a couple reasons we recommend this.
 
 #### How to use the keystore?
 
-1. Create a Kubernetes secret containing the [keystore](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html)
-   ```
-   $ kubectl create secret generic elasticsearch-keystore --from-file=./elasticsearch.keystore
-   ```
-2. Mount it into the container via `secretMounts`
-   ```
-   secretMounts:
-   - name: elasticsearch-keystore
-     secretName: elasticsearch-keystore
-     path: /usr/share/elasticsearch/config/elasticsearch.keystore
-     subPath: elasticsearch.keystore
-   ```
+
+##### Basic example
+
+Create the secret, the key name needs to be the keystore key path. In this example we will create a secret from a file and from a literal string.
+
+```
+kubectl create secret generic encryption_key --from-file=xpack.watcher.encryption_key=./watcher_encryption_key
+kubectl create secret generic slack_hook --from-literal=xpack.notification.slack.account.monitoring.secure_url='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd'
+```
+
+To add these secrets to the keystore:
+```
+keystore:
+  - secretName: encryption_key
+  - secretName: slack_hook
+```
+
+##### Multiple keys
+
+All keys in the secret will be added to the keystore. To create the previous example in one secret you could also do:
+
+```
+kubectl create secret generic keystore_secrets --from-file=xpack.watcher.encryption_key=./watcher_encryption_key --from-literal=xpack.notification.slack.account.monitoring.secure_url='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd'
+```
+
+```
+keystore:
+  - secretName: keystore_secrets
+```
+
+##### Custom paths and keys
+
+If you are using these secrets for other applications (besides the Elasticsearch keystore) then it is also possible to specify the keystore path and which keys you want to add. Everything specified under each `keystore` item will be passed through to the `volumeMounts` section for [mounting the secret](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets). In this example we will only add the `slack_hook` key from a secret that also has other keys. Our secret looks like this:
+
+```
+kubectl create secret generic slack_secrets --from-literal=slack_channel='#general' --from-literal=slack_hook='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd'
+```
+
+We only want to add the `slack_hook` key to the keystore at path `xpack.notification.slack.account.monitoring.secure_url`.
+
+```
+keystore:
+  - secretName: slack_secrets
+    items:
+    - key: slack_hook
+      path: xpack.notification.slack.account.monitoring.secure_url
+```
+
+You can also take a look at the [config example](/elasticsearch/examples/config/) which is used as part of the automated testing pipeline.
 
 #### How to enable snapshotting?
 

elasticsearch/examples/config/Makefile

@@ -0,0 +1,19 @@
+default: test
+include ../../../helpers/examples.mk
+
+RELEASE := helm-es-config
+
+install:
+	helm upgrade --wait --timeout=600 --install $(RELEASE) --values ./values.yaml ../../ ; \
+
+secrets:
+	kubectl delete secret elastic-config-credentials elastic-config-secret elastic-config-slack elastic-config-custom-path || true
+	kubectl create secret generic elastic-config-credentials --from-literal=password=changeme --from-literal=username=elastic
+	kubectl create secret generic elastic-config-slack --from-literal=xpack.notification.slack.account.monitoring.secure_url='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd'
+	kubectl create secret generic elastic-config-secret --from-file=xpack.watcher.encryption_key=./watcher_encryption_key
+	kubectl create secret generic elastic-config-custom-path --from-literal=slack_url='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd' --from-literal=thing_i_don_tcare_about=test
+
+test: secrets install goss
+
+purge:
+	helm del --purge $(RELEASE)

elasticsearch/examples/config/README.md

@@ -0,0 +1,3 @@
+# Config
+
+An example testing suite for testing some of the optional features of this chart.

elasticsearch/examples/config/test/goss.yaml

@@ -0,0 +1,26 @@
+http:
+  http://localhost:9200/_cluster/health:
+    status: 200
+    timeout: 2000
+    body:
+      - 'green'
+      - '"number_of_nodes":1'
+      - '"number_of_data_nodes":1'
+
+  http://localhost:9200:
+    status: 200
+    timeout: 2000
+    body:
+      - '"cluster_name" : "config"'
+      - '"name" : "config-master-0"'
+      - 'You Know, for Search'
+
+command:
+  "elasticsearch-keystore list":
+    exit-status: 0
+    stdout:
+      - keystore.seed
+      - bootstrap.password
+      - xpack.notification.slack.account.monitoring.secure_url
+      - xpack.notification.slack.account.otheraccount.secure_url
+      - xpack.watcher.encryption_key

elasticsearch/examples/config/values.yaml

@@ -0,0 +1,31 @@
+---
+
+clusterName: "config"
+replicas: 1
+
+extraEnvs:
+  - name: ELASTIC_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: elastic-credentials
+        key: password
+  - name: ELASTIC_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: elastic-credentials
+        key: username
+
+# This is just a dummy file to make sure that
+# the keystore can be mounted at the same time
+# as a custom elasticsearch.yml
+esConfig:
+  elasticsearch.yml: |
+    path.data: /usr/share/elasticsearch/data
+
+keystore:
+  - secretName: elastic-config-secret
+  - secretName: elastic-config-slack
+  - secretName: elastic-config-custom-path
+    items:
+    - key: slack_url
+      path: xpack.notification.slack.account.otheraccount.secure_url

elasticsearch/examples/config/watcher_encryption_key

@@ -0,0 +1 @@
+supersecret

elasticsearch/templates/statefulset.yaml

@@ -111,6 +111,14 @@ spec:
           configMap:
             name: {{ template "uname" . }}-config
         {{- end }}
+{{ if .Values.keystore }}
+        - name: keystore
+          emptyDir: {}
+{{ end }}
+        {{- range .Values.keystore }}
+        - name: keystore-{{ .secretName }}
+          secret: {{ toYaml . | nindent 12 }}
+        {{- end }}
       {{- if .Values.extraVolumes }}
 {{ tpl .Values.extraVolumes . | indent 6 }}
       {{- end }}
@@ -129,6 +137,38 @@ spec:
         resources:
 {{ toYaml .Values.initResources | indent 10 }}
       {{- end }}
+{{ if .Values.keystore }}
+      - name: keystore
+        image: "{{ .Values.image }}:{{ .Values.imageTag }}"
+        command:
+        - sh
+        - -c
+        - |
+          #!/usr/bin/env bash
+          set -euo pipefail
+
+          elasticsearch-keystore create
+
+          for i in /tmp/keystoreSecrets/*/*; do
+            key=$(basename $i)
+            echo "Adding file $i to keystore key $key"
+            elasticsearch-keystore add-file "$key" "$i"
+          done
+
+          # Add the bootstrap password since otherwise the Elasticsearch entrypoint tries to do this on startup
+          [ ! -z "$ELASTIC_PASSWORD" ] && echo $ELASTIC_PASSWORD | elasticsearch-keystore add -x bootstrap.password
+
+          cp -a /usr/share/elasticsearch/config/elasticsearch.keystore /tmp/keystore/
+        env: {{ toYaml .Values.extraEnvs | nindent 10 }}
+        resources: {{ toYaml .Values.initResources | nindent 10 }}
+        volumeMounts:
+          - name: keystore
+            mountPath: /tmp/keystore
+          {{- range .Values.keystore }}
+          - name: keystore-{{ .secretName }}
+            mountPath: /tmp/keystoreSecrets/{{ .secretName }}
+          {{- end }}
+{{ end }}
       {{- if .Values.extraInitContainers }}
 {{ tpl .Values.extraInitContainers . | indent 6 }}
       {{- end }}
@@ -219,6 +259,11 @@ spec:
           - name: "{{ template "uname" . }}"
             mountPath: /usr/share/elasticsearch/data
           {{- end }}
+{{ if .Values.keystore }}
+          - name: keystore
+            mountPath: /usr/share/elasticsearch/config/elasticsearch.keystore
+            subPath: elasticsearch.keystore
+{{ end }}
           {{- range .Values.secretMounts }}
           - name: {{ .name }}
             mountPath: {{ .path }}

elasticsearch/tests/elasticsearch_test.py

@@ -793,3 +793,109 @@ def test_adding_pod_labels():
 '''
     r = helm_template(config)
     assert r['statefulset'][uname]['metadata']['labels']['app.kubernetes.io/name'] == 'elasticsearch'
+
+def test_keystore_enable():
+    config = ''
+
+    r = helm_template(config)
+    s = r['statefulset'][uname]['spec']['template']['spec']
+
+    assert s['volumes'] == None
+
+    config = '''
+keystore:
+  - secretName: test
+    '''
+
+    r = helm_template(config)
+    s = r['statefulset'][uname]['spec']['template']['spec']
+
+    assert {'name': 'keystore', 'emptyDir': {}} in s['volumes']
+
+def test_keystore_init_container():
+    config = ''
+
+    r = helm_template(config)
+    i = r['statefulset'][uname]['spec']['template']['spec']['initContainers'][-1]
+
+    assert i['name'] != 'keystore'
+
+    config = '''
+keystore:
+  - secretName: test
+    '''
+
+    r = helm_template(config)
+    i = r['statefulset'][uname]['spec']['template']['spec']['initContainers'][-1]
+
+    assert i['name'] == 'keystore'
+
+def test_keystore_mount():
+    config = '''
+keystore:
+  - secretName: test
+'''
+
+    r = helm_template(config)
+    s = r['statefulset'][uname]['spec']['template']['spec']
+    assert s['containers'][0]['volumeMounts'][-1] == {
+        'mountPath': '/usr/share/elasticsearch/config/elasticsearch.keystore',
+        'subPath': 'elasticsearch.keystore',
+        'name': 'keystore'
+    }
+
+def test_keystore_init_volume_mounts():
+    config = '''
+keystore:
+  - secretName: test
+  - secretName: test-with-custom-path
+    items:
+    - key: slack_url
+      path: xpack.notification.slack.account.otheraccount.secure_url
+'''
+    r = helm_template(config)
+    s = r['statefulset'][uname]['spec']['template']['spec']
+    assert s['initContainers'][-1]['volumeMounts'] == [
+            {
+              'mountPath': '/tmp/keystore',
+              'name': 'keystore'
+            },
+            {
+              'mountPath': '/tmp/keystoreSecrets/test',
+              'name': 'keystore-test'
+            },
+            {
+              'mountPath': '/tmp/keystoreSecrets/test-with-custom-path',
+              'name': 'keystore-test-with-custom-path'
+            }
+    ]
+
+def test_keystore_volumes():
+    config = '''
+keystore:
+  - secretName: test
+  - secretName: test-with-custom-path
+    items:
+    - key: slack_url
+      path: xpack.notification.slack.account.otheraccount.secure_url
+'''
+    r = helm_template(config)
+    s = r['statefulset'][uname]['spec']['template']['spec']
+
+    assert {
+             'name': 'keystore-test',
+             'secret': {
+                 'secretName': 'test'
+             }
+           } in s['volumes']
+
+    assert {
+             'name': 'keystore-test-with-custom-path',
+             'secret': {
+                 'secretName': 'test-with-custom-path',
+                 'items': [{
+                     'key': 'slack_url',
+                     'path': 'xpack.notification.slack.account.otheraccount.secure_url'
+                 }]
+             }
+           } in s['volumes']

elasticsearch/values.yaml

@@ -210,3 +210,5 @@ lifecycle: {}
 
 sysctlInitContainer:
   enabled: true
+
+keystore: []

helpers/matrix.yml

@@ -5,6 +5,7 @@ CHART:
   - metricbeat
 ES_SUITE:
   - default
+  - config
   - multi
   - oss
   - security