How to configure snapshots with curator

[syst0m]

Hi,

trying to come up with the best solution to snapshot an indices subset.
Using ElasticSearch (6.2.2) from this repo on Google Kubernetes Engine.

Failed to find any useful examples on how to do this with ES on GKE.
Found PV snapshotting (k8s >v1.8), but I don't need all indices, so no need to snapshot entire volume.

Would use curator snapshot mechanism to store to Google Cloud bucket.

Any tips on how to go about this?

Thanks in advance!

[pires]

This repo docs do mention how to schedule Curator jobs. You just need to learn Curator enough, in order to create a configuration that will deliver what you need.

The definition of the action_file.yaml is quite self-explaining for simple set-ups. For more advanced configuration options, please consult the Curator Documentation.

[syst0m]

This repo docs do mention how to schedule Curator jobs. You just need to learn Curator enough, in order to create a configuration that will deliver what you need.

I already have existing snapshot & delete old indices configs, which I need to adapt.
Existing config uses environment variables passed from the shell (i.e. unique snapshot name generated by using time, date commands).
Not sure how to implement those here, they were populated via script scheduled in a cronjob and then passed to the curator jobs.

[pires]

Use https://github.com/pires/kubernetes-elasticsearch-cluster/blob/master/es-curator-config.yaml, maybe?

[syst0m]

I should probably elaborate a bit.
This is the old bash script scheduled in cron:

#!/bin/bash
rm logs/backup_indices.log
export DAYS=40
export SNAPSHOT_NAME=snapshot-$(date -d -${DAYS}days +%Y-%m-%d)
echo "Creating snapshot for $(date -d -${DAYS}days +%Y-%m-%d)" >> logs/backup_indices.log
/usr/local/bin/curator --config /root/configs/backup_config.yml /root/actions/backup_action.yml
#curator --config configs/backup_config.yml --dry-run actions/backup_action.yml
mail -s "Move Elasticsearch Indices Report (New)" logs@company.com < logs/backup_indices.log

And the backup_actions.yml is using $DAYS & $SNAPSHOT_NAME set by the shell script:

actions:
1:
action: snapshot
options:
repository: s3_repository
name: ${SNAPSHOT_NAME:snapshot-%Y-%m-%d}
continue_if_exception: false
filters:
- filtertype: age
source: name
direction: older
timestring: '%Y-%m-%d'
unit: days
unit_count: ${DAYS}
2:
action: delete_indices
options:
continue_if_exception: false
filters:
- filtertype: age
source: name
direction: older
timestring: '%Y-%m-%d'
unit: days
unit_count: ${DAYS}

How would you go about using these shell env vars in docker/k8s?

[pires]

Maybe I'm missing something but can't you simply set env vars in the podSpec such as https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/?

[syst0m]

$SNAPSHOT_NAME is dynamically generated during each script execution, by using the linux "date" command "date -d -${DAYS}days +%Y-%m-%d)".

I don't think that this is a viable approach for setting a k8s container env var?

[syst0m]

Additionaly, any tips on how to configure the elasticsearch keystore (which appears to be a requirement for configuring both GCS & AWS S3 access to store snapshots)?

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs-usage.html

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-client.html

[syst0m]

@pires I will try to build my own image based on yours, the expanded image will create the "elasticsearch.keystore" keyfile, as described here:

https://www.elastic.co/guide/en/elasticsearch/reference/6.3/secure-settings.html

That way all my containers will already have the keystore configuration and gcs key used to authenticate to the gcs bucket used for snapshots.

I tried creating the file, from a shell in the client node.
looks like it's a binary file:

bash-4.4# cat /elasticsearch/config/elasticsearch.keystore 
��������0��0���T*�H��I0��E���0���12�PBE�*�H��
�              *�H��

��r0p�
�     *�H��

���a�_0]0)�
��H��
���0���A�1���Time 15294058407850>0!0��ӫ��+�����:�_:��!�x-�x�5&�������m�����������՟X��'������(���Mp�

Dockerfile:

FROM quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1

# inital setup
USER elasticsearch
COPY  gcs.client.default.credentials_file /tmp/

# fix elasticsearch-keystore failing with mktemp: Invalid argument
RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX

# create keystore file and import the google cloud storage credentials key
RUN /elasticsearch/bin/elasticsearch-keystore create
RUN /elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file


CMD

What does your image use for CMD?

[syst0m]

I found your Dockerfile here, and modified mine accordingly:
https://github.com/pires/docker-elasticsearch/blob/6.2.2_1/Dockerfile

Dockerfile:

FROM quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1

# inital setup
USER elasticsearch
COPY  gcs.client.default.credentials_file /tmp/

ENV PATH /elasticsearch/bin:$PATH

WORKDIR /elasticsearch

# Copy configuration
COPY config /elasticsearch/config

# Copy run script
COPY run.sh /

# Set environment variables defaults
ENV ES_JAVA_OPTS "-Xms512m -Xmx512m"
ENV CLUSTER_NAME elasticsearch-default
ENV NODE_MASTER true
ENV NODE_DATA true
ENV NODE_INGEST true
ENV HTTP_ENABLE true
ENV NETWORK_HOST _site_
ENV HTTP_CORS_ENABLE true
ENV HTTP_CORS_ALLOW_ORIGIN *
ENV NUMBER_OF_MASTERS 1
ENV MAX_LOCAL_STORAGE_NODES 1
ENV SHARD_ALLOCATION_AWARENESS ""
ENV SHARD_ALLOCATION_AWARENESS_ATTR ""
ENV MEMORY_LOCK true
ENV REPO_LOCATIONS []

# Volume for Elasticsearch data
VOLUME ["/data"]

# fix elasticsearch-keystore failing with mktemp: Invalid argument & create+populate keystore
RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX` && /elasticsearch/bin/elasticsearch-keystore create
RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX` && /elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file


CMD ["/run.sh"]

Build is failing with:

Sending build context to Docker daemon  15.87kB
Step 1/26 : FROM quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1
 ---> 6bc60f1c2953
Step 2/26 : USER elasticsearch
 ---> Using cache
 ---> d099f041a3be
Step 3/26 : COPY  gcs.client.default.credentials_file /tmp/
 ---> Using cache
 ---> edca7b0a21e0
Step 4/26 : ENV PATH /elasticsearch/bin:$PATH
 ---> Running in c88d0b4c1925
Removing intermediate container c88d0b4c1925
 ---> 654cf5eb84ce
Step 5/26 : WORKDIR /elasticsearch
Removing intermediate container e2b811733c47
 ---> 656a5a3786ee
Step 6/26 : COPY config /elasticsearch/config
 ---> d47fe83962d8
Step 7/26 : COPY run.sh /
 ---> 2888df120fd8
Step 8/26 : ENV ES_JAVA_OPTS "-Xms512m -Xmx512m"
 ---> Running in 6cb592601fe8
Removing intermediate container 6cb592601fe8
 ---> c6f39956481c
Step 9/26 : ENV CLUSTER_NAME elasticsearch-default
 ---> Running in 8a7610bf1d1e
Removing intermediate container 8a7610bf1d1e
 ---> d1dd59c6be54
Step 10/26 : ENV NODE_MASTER true
 ---> Running in bb500e435804
Removing intermediate container bb500e435804
 ---> 432e04ce7b2e
Step 11/26 : ENV NODE_DATA true
 ---> Running in a5c19bf07fec
Removing intermediate container a5c19bf07fec
 ---> 2a24f5f585e0
Step 12/26 : ENV NODE_INGEST true
 ---> Running in 7c001fcaa068
Removing intermediate container 7c001fcaa068
 ---> 9bafa994e712
Step 13/26 : ENV HTTP_ENABLE true
 ---> Running in 97d301ad25fd
Removing intermediate container 97d301ad25fd
 ---> 3094beebb028
Step 14/26 : ENV NETWORK_HOST _site_
 ---> Running in 81d5c147a527
Removing intermediate container 81d5c147a527
 ---> e7880443646f
Step 15/26 : ENV HTTP_CORS_ENABLE true
 ---> Running in b65be6498b03
Removing intermediate container b65be6498b03
 ---> 1e1930900b53
Step 16/26 : ENV HTTP_CORS_ALLOW_ORIGIN *
 ---> Running in 389b1c9758d9
Removing intermediate container 389b1c9758d9
 ---> 3be74de80145
Step 17/26 : ENV NUMBER_OF_MASTERS 1
 ---> Running in 8b5ef389ad7f
Removing intermediate container 8b5ef389ad7f
 ---> e04f5db12aa5
Step 18/26 : ENV MAX_LOCAL_STORAGE_NODES 1
 ---> Running in fc3cdf679c0d
Removing intermediate container fc3cdf679c0d
 ---> 40ac5fdb6119
Step 19/26 : ENV SHARD_ALLOCATION_AWARENESS ""
 ---> Running in d9b62faa9cda
Removing intermediate container d9b62faa9cda
 ---> 2bc975f25f15
Step 20/26 : ENV SHARD_ALLOCATION_AWARENESS_ATTR ""
 ---> Running in 7dc6bbb595c9
Removing intermediate container 7dc6bbb595c9
 ---> e64ea469f33d
Step 21/26 : ENV MEMORY_LOCK true
 ---> Running in 50f6ef779f66
Removing intermediate container 50f6ef779f66
 ---> bbd8d99f553e
Step 22/26 : ENV REPO_LOCATIONS []
 ---> Running in 7f4289969020
Removing intermediate container 7f4289969020
 ---> 7b5f62485b45
Step 23/26 : VOLUME ["/data"]
 ---> Running in 2d5ee4e70381
Removing intermediate container 2d5ee4e70381
 ---> 46c035e8d167
Step 24/26 : RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX` && /elasticsearch/bin/elasticsearch-keystore create
 ---> Running in 445d31647926
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'NODE_NAME'
	at org.elasticsearch.common.settings.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:116)
	at org.elasticsearch.common.settings.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:69)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1258)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1208)
	at org.elasticsearch.node.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:127)
	at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:106)
	at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)

[syst0m]

Ok, I managed to build the image.
I then created a canary deployment for a canary client node.
It is failing thusly:

> kubectl logs es-client-canary-598c8f54f7-s47ms
sh: error setting limit: Operation not permitted
-> Downloading repository-gcs from elastic
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.net.URLPermission http://www.googleapis.com/* *:
* java.net.URLPermission https://www.googleapis.com/* *:
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-gcs
[2018-06-20T09:15:40,612][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Out of memory
[2018-06-20T09:15:40,615][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-06-20T09:15:40,615][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-06-20T09:15:40,616][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2018-06-20T09:15:40,616][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-06-20T09:15:40,905][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-client-canary-598c8f54f7-s47ms] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' ([])
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.2.jar:6.2.2]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' ([])
	at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /elasticsearch/[]
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
	at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
	at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
	at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 6 more

Here's my final Dockerfile of the new image:

FROM quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1

# inital setup
COPY gcs.client.default.credentials_file /tmp/

ENV PATH /elasticsearch/bin:$PATH

WORKDIR /elasticsearch

# Copy configuration
COPY config /elasticsearch/config

# Copy run script
COPY run.sh /

# Set environment variables defaults
ENV ES_JAVA_OPTS "-Xms512m -Xmx512m"
ENV CLUSTER_NAME elasticsearch-default
ENV NODE_MASTER true
ENV NODE_DATA true
ENV NODE_INGEST true
ENV HTTP_ENABLE true
ENV NETWORK_HOST _site_
ENV HTTP_CORS_ENABLE true
ENV HTTP_CORS_ALLOW_ORIGIN *
ENV NUMBER_OF_MASTERS 1
ENV MAX_LOCAL_STORAGE_NODES 1
ENV SHARD_ALLOCATION_AWARENESS ""
ENV SHARD_ALLOCATION_AWARENESS_ATTR ""
ENV MEMORY_LOCK true
ENV REPO_LOCATIONS []

# Volume for Elasticsearch data
VOLUME ["/data"]

# fix "Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'NODE_NAME' when running elasticsearch-keystore
ENV NODE_NAME=""

# fix elasticsearch-keystore failing with mktemp: Invalid argument & create+populate keystore
RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX` && /elasticsearch/bin/elasticsearch-keystore create
RUN export ES_TMPDIR=`mktemp -d -t elasticsearch.XXXXXXXX` && /elasticsearch/bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /tmp/gcs.client.default.credentials_file
RUN chown elasticsearch:elasticsearch /elasticsearch/config/elasticsearch.keystore

USER elasticsearch

CMD ["/run.sh"]

Any clues on how to fix this, @pires ?
Any help here would be greatly appreciated.

Thanks in advance!

[mat1010]

Looks like /elasticsearch is not accesible for the user elasticsearch. What about chaging your chown?

- RUN chown elasticsearch:elasticsearch /elasticsearch/config/elasticsearch.keystore
+ RUN chown -R elasticsearch:elasticsearch /elasticsearch

Additionally: Have you ensured /data has the proper permissions / owner as well?