Permissions prevent running on OpenShift

[agc93]

This is essentially the same issue as #49 in a different contex. Let me know if you'd prefer to resurrect that discussion instead.

• Docker image used: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
• Operating System: CentOS 7.3 (running OpenShift 3.6/Kubernetes 1.6.1)
• Steps to reproduce:

• Create a new pod/deployment using the docker.elastic.co/elasticsearch/elasticsearch:5.5.2 image
• Pod will fail on creation with /bin/bash: bin/es-docker: Permission denied

Bug Description

The permissions on /usr/share/elasticsearch/bin/es-docker (and parent directories) only allow read and execute for the hard-coded elasticsearch user. This prevents the container from even starting (Permission denied for bin/es-docker file). Even fixing that specific issue, using something like the below, just results in different permissions problems down the line.

FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.2
USER root
RUN chgrp -R 0 /usr/share/elasticsearch && chmod -R g+rx /usr/share/elasticsearch/
USER elasticsearch

Is there no way to run the elasticsearch container on platforms with runtime user IDs?

[dliappis]

@agc93 This has been discussed before and some general guideline bullets can be seen in the bold section of this comment.

If you can't use docker volumes (see the link above), the best approach would be to adjust the gid of the elasticsearch on your host for the gid:1000 (e.g. with chgrp -R 1000 and chmod -R g+rw).

Failing that, you can create a customized docker image as shown in the Dockerfile, where you can pass the UID/GID during build time.

Note that Elasticsearch bootstrap checks will fail if you try to run as root (reasoning here), so you can't use UID/GID: 0. You will find that some Docker images run the initial CMD as root and mutate the UID/GID of the bind mounted dir to grant write privileges to the elasticsearch user; we feel that mutating host directories from the container breaks the isolation promise of containerization and can lead to unexpected surprises.

[agc93]

Thanks @dliappis !

I'm already using a volume for the data directory, but as shown in the original comment, the issue is that even the entrypoint script cannot be run by a user other than elasticsearch/1000. This is the case for the config directory as well.

Using OpenShift (and some other Kubernetes platforms) the user ID is only known at runtime instead of at build time making the Dockerfile approach somewhat difficult. I'll give that a shot, but that can be a bit tricky.

Agreed re the host mutation being a bit dodgy :)

[dliappis]

You are welcome @agc93 !

For the k8s case we have a similar open issue, #90.

After searching a bit, my understanding is that it's now possible to use the securityContext definition in the Pod.spec as shown here: kubernetes/kubernetes#2630 (comment); also check this comment. This should be possible after k8s ver 1.7.

You can also use an init container workaround, as shown here.

Finally there is a third alternative, building your own custom image:

1. Use the following Dockerfile to grant some sudo rights and chgrp -R 1000 the data dir:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.0
USER root
RUN yum install -y sudo && \
    yum clean all && \
    echo "elasticsearch ALL= NOPASSWD: /usr/bin/chgrp -R 1000 /usr/share/elasticsearch/data" > /etc/sudoers.d/elasticsearch && \
    chmod 0440 /etc/sudoers.d/elasticsearch
USER elasticsearch
# Modify permissions of /usr/share/elasticsearch/data before we start elasticsearch
RUN sed -i '$isudo /usr/bin/chgrp -R 1000 /usr/share/elasticsearch/data' /usr/share/elasticsearch/bin/es-docker

2. Build your customized image (e.g. I used docker build -t mutating_es .).

I tested this using a user dockertest with uid/gid 1001, with no sudo rights what-so-ever (but access to the docker group):

$ sudo su - dockertests

$ id
uid=1001(dockertests) gid=1001(dockertests) groups=1001(dockertests),975(docker)

$ cd es-issue-114/
$ ls
$ mkdir myesdata
$ ls -ldn myesdata/
drwxrwxr-x 2 1001 1001 4096 Sep 18 17:46 myesdata/

$ docker run --rm -d -p 9200:9200 -v $PWD/myesdata:/usr/share/elasticsearch/data mutating_es:latest
e76037de0f49cb09473f36592c2479664a80078ffaab150230cc00a1c2d6b01a

Now if we check the perms of the host data dir, we can see the group ownership has changed:

$ ls -ldn myesdata/
drwxrwxr-x 3 1001 1000 4096 Sep 18 17:46 myesdata/

[dockertests@MS-7885 es-issue-114]$ curl -u elastic:changeme localhost:9200 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   328  100   328    0     0   2121      0 --:--:-- --:--:-- --:--:--  2129
{
  "name": "2fPKgAy",
  "cluster_name": "docker-cluster",
  "cluster_uuid": "SzCBbtybRqmMGkTeZl3Y5A",
  "version": {
    "number": "5.6.0",
    "build_hash": "781a835",
    "build_date": "2017-09-07T03:09:58.087Z",
    "build_snapshot": false,
    "lucene_version": "6.6.0"
  },
  "tagline": "You Know, for Search"
}

[agc93]

Thanks for the info @dliappis, and sorry for taking so long closing the loop on this one (been out of the country).

I've currently got it running successfully using my own image using the following Dockerfile:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.2
USER root
RUN chgrp -R 0 /usr/share/elasticsearch && \
    chmod -Rf g+rwx /usr/share/elasticsearch
USER elasticsearch

Would you guys accept a PR to the official image with this change in group ownership?

---

A bit of background

So the issue being hit here is not strictly related to the data directory, and more importantly isn't related to the user who starts the container (i.e. dockertests in your example).

Openshift uses semi-random high UIDs as the actual container user (i.e. overriding the value of USER). It is possible to disable this using securityContext, but is not always recommended and not always possible (GKE/OpenShift Online for example). Additionally, using scc inside templates gets pretty unwieldy and can break portability.

The one predictable point is that the high UIDs are (automatically) members of the root group (ID 0) which is why the solution above works in OpenShift but the out-of-box image doesn't (everything is hardcoded to elasticsearch:elasticsearch / 1000:1000)

[dliappis]

Thanks for the update @agc93 .

I'd be interested to hear on which occasions securityContext is not recommended, if you have some links. In our experience most docker images that use a preconfigured user, such as database images, consul etc. get executed with the securityContext, to override the group for bind mounted volumes.

However, the random UIDs that overrides the USER definition in the image's Dockerfile seems unique to Openshift. We had a look at GKE and it seems that, if specified, the USER that CMD/Entrypoint runs as, is not affected.

Your offer for a PR is highly valued thanks! I think we wouldn't want to grant +x to all files under /usr/share/elasticsearch though (referring to: chmod -Rf g+rwx /usr/share/elasticsearch). I have been planning to PR the sudo solution for granting access to bind mounts in my earlier comment, through a startup option (env var). However, this will not solve the problem you are describing with Openshift which, by taking over the USER with a random high UID, doesn't even let the CMD wrapper script run in the first place.

Let me think about the options a little more and get back here soon.

cc @tylerjl

[agc93]

Thanks @dliappis !

The recommendation against securityContext I've seen is mostly more a case of "if you use these, it's best to also change the security options". With OpenShift (and Kubernetes' new RBAC, which is based on OpenShift's), the out-of-box security has quite a clear division between privileged and unprivileged and not much middle-ground. To effectively add securityContext to the mix often means adjusting these as well.

That being said, the document I had read recommending against it outright has been updated, so I suspect this has changed in more recent k8s/openshift releases.

We had a look at GKE and it seems that, if specified, the USER that CMD/Entrypoint runs as, is not affected.

Huh, good to know! 😄 Must just be OpenShift doing this then (thought I had my local k8s doing this once, must have set it up wrong).

Agreed, the +x might be a little overzealous, but it was for a PoC 😆

Let me know how you want to proceed!

[dliappis]

Closed by #125

[dliappis]

Hey @agc93 the recently released 6.0.0-rc2 contains #125 , do you think you could give it a try in Openshift to seen if it works as expected?