Permission denied on bin/es-docker if ran with user remapping

[jensdebruijn]

• docker info output:
  Containers: 85 Running: 0 Paused: 0 Stopped: 85 Images: 336 Server Version: 17.04.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/1000.1000/aufs Backing Filesystem: extfs Dirs: 372 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73 runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228 init version: 949e6fa Security Options: apparmor seccomp Profile: default userns Kernel Version: 4.8.0-46-generic Operating System: Ubuntu 16.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 31.33GiB Name: workBuntu ID: 5N25:YF6I:3CKB:YMIE:G4QQ:QYDD:PTVZ:TXP4:7YMH:IKR2:Q4XX:K4FO Docker Root Dir: /var/lib/docker/1000.1000 Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support

• docker-compose version:
  docker-compose version 1.12.0, build b31ff33

• Host OS and version:
  Ubuntu 16,04

Bug description

I am running elasticsearch with user remapping for docker as such:

https://blog.yadutaf.fr/2016/04/14/docker-for-your-users-introducing-user-namespace/

Running the image with docker-compose gives me the following error:

/bin/bash: bin/es-docker: Permission denied
elasticsearch exited with code 126

[dliappis]

@jens1245 Could you please provide your exact docker-compose file (or docker run command?)?

[jensdebruijn]

Certainly!:

version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.0
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
cap_add:
- IPC_LOCK
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet

volumes:
esdata:
driver: local

networks:
esnet:
driver: bridge

[dliappis]

@jens1245 I reformatted your docker-compose.yml below for readability, but other than that this will work fine.

You are using a named volume called esdata.
If you haven't used that volume before this should work out of the box.
If you have used the same esdata: storage volume before, say with a different image and hence having dirs with different ownerships, perhaps ES won't be able to write there.
If you do not mind losing the data currently stored in esdata:, can you try wiping it away with:

docker-compose down -v

(-v will delete the volume)
and then start it again with:

docker-compose up

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.3.0
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    mem_limit: 1g
    cap_add:
      - IPC_LOCK
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

volumes:
  esdata:
    driver: local

networks:
  esnet:
    driver: bridge

[jensdebruijn]

I used:

docker-compose down -v

and rebuilt the image, but I still get the same error.

Did you set the user mapping?

{
        "userns-remap": "yadutaf"
}

in /etc/docker/daemon.json?

[dliappis]

@jens1245 No I didn't set the userns-remap option. Would you care describing what are you trying to achieve with this option, for your use case? The docker ES image, defaults to a specific non-privileged user (elasticsearch uid:1000, gid: 1000) inside the container, with no sudo access. The ES wrapper script /bin/es-docker starts using this user, so this limits the attack surface, if it's security that has you looking at the userns-remap option.

[jensdebruijn]

I set the user mapping for another container. This way it is easy to share data with the host and at the same time limits the attack surface. Thus, in this case, it is not so much for Elasticsearch but for for overall security / usability. Unfortunately, I do not know of a way to use user remapping only for some containers.

Considering the feature is quite new, but received enthusiastically, it would be nice if the image works for this user case.

[jegade]

I'm running in the same problem. I modified the uid/gid because i remap uid 1000 to root, which is great for other containers, but failed for elasticsearch. 1000 because i own the uid on the development maschine and can share code into the container.

I build my own Dockerfile and change the uid/gid to some other and everything works for now

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
# Switch to root for install
USER root
#Hack
RUN echo http://dl-2.alpinelinux.org/alpine/edge/community/ >> /etc/apk/repositories
RUN apk add --no-cache shadow && \ 
 usermod -u 555  elasticsearch && \
 groupmod -g 555 elasticsearch && \
	find . -user  1000 -exec chown -h 555 {} \; && \
	find . -group 1000 -exec chgrp -h 555 {} \; && \
	usermod -g 555 elasticsearch

# Switch back 
USER elasticsearch 

The other way, was to change my user id on the host, then 1000 will map not to 0

[jegade]

Another temporary workaround is to write your own Dockerfile, based on the history from elasticsearch

FROM alpine

RUN apk --no-cache  add bash curl openjdk8 openssl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
RUN adduser -D -u 555 -h /usr/share/elasticsearch elasticsearch                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             

ARG ELASTIC_VERSION
ARG ES_DOWNLOAD_URL
ARG ES_JAVA_OPTS

ENV ELASTIC_VERSION=5.3.0 
ENV ES_DOWNLOAD_URL=https://artifacts.elastic.co/downloads/elasticsearch 
ENV PATH /usr/share/elasticsearch/bin:$PATH
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk

WORKDIR /usr/share/elasticsearch

# Download/extract defined ES version. busybox tar can't strip leading dir.
RUN wget ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    EXPECTED_SHA=$(wget -O - ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz.sha1) && \
    test $EXPECTED_SHA == $(sha1sum elasticsearch-${ELASTIC_VERSION}.tar.gz | awk '{print $1}') && \
    tar zxf elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    chown -R elasticsearch:elasticsearch elasticsearch-${ELASTIC_VERSION} && \
    mv elasticsearch-${ELASTIC_VERSION}/* . && \
    rmdir elasticsearch-${ELASTIC_VERSION} && \
    rm elasticsearch-${ELASTIC_VERSION}.tar.gz

RUN set -ex && for esdirs in config data logs; do \
        mkdir -p "$esdirs"; \
        chown -R elasticsearch:elasticsearch "$esdirs"; \
    done

# Install Plugin
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin list
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

# Remove x-pack 'https://github.com/elastic/elasticsearch-docker/issues/35#issuecomment-285912424
# RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove  x-pack
RUN rm -rf plugins/x-pack


USER elasticsearch

COPY elasticsearch.yml config/
COPY log4j2.properties config/
COPY bin/es-docker bin/es-docker

USER root
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml config/log4j2.properties bin/es-docker && \
    chmod 0750 bin/es-docker

USER elasticsearch
CMD ["/bin/bash", "bin/es-docker"]

[jensdebruijn]

Thank you so much!

I've slightly changed the Docker file you provided for my purpose (with X-pack) and it all works now:

FROM alpine

RUN apk --no-cache  add bash curl openjdk8 openssl
RUN adduser -D -u 555 -h /usr/share/elasticsearch elasticsearch

ARG ELASTIC_VERSION
ARG ES_DOWNLOAD_URL
ARG ES_JAVA_OPTS

ENV PATH /usr/share/elasticsearch/bin:$PATH
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk

WORKDIR /usr/share/elasticsearch

# Download/extract defined ES version. busybox tar can't strip leading dir.
RUN wget ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    EXPECTED_SHA=$(wget -O - ${ES_DOWNLOAD_URL}/elasticsearch-${ELASTIC_VERSION}.tar.gz.sha1) && \
    test $EXPECTED_SHA == $(sha1sum elasticsearch-${ELASTIC_VERSION}.tar.gz | awk '{print $1}') && \
    tar zxf elasticsearch-${ELASTIC_VERSION}.tar.gz && \
    chown -R elasticsearch:elasticsearch elasticsearch-${ELASTIC_VERSION} && \
    mv elasticsearch-${ELASTIC_VERSION}/* . && \
    rmdir elasticsearch-${ELASTIC_VERSION} && \
    rm elasticsearch-${ELASTIC_VERSION}.tar.gz

RUN set -ex && for esdirs in config data logs; do \
        mkdir -p "$esdirs"; \
        chown -R elasticsearch:elasticsearch "$esdirs"; \
    done

USER elasticsearch

# Install xpack
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch x-pack

COPY elasticsearch.yml config/
COPY log4j2.properties config/
COPY bin/es-docker bin/es-docker

USER root
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml config/log4j2.properties bin/es-docker && \
    chmod 0750 bin/es-docker

USER elasticsearch
CMD ["/bin/bash", "bin/es-docker"]

[dliappis]

Another temporary workaround is to write your own Dockerfile, based on the history from elasticsearch

@jegade You do not need to use the docker history command to see how the image has been built, you can always refer to the Dockerfile used in the repo.

Additionally, while creating your customized image, you may also use the existing image as a base and add your own customizations to reduce repetition. One example is provided in the documentation here.

[jegade]

@dliappis sorry, the refered Dockerfile does not contains the whole history. This is based on FROM docker.elastic.co/elasticsearch/elasticsearch-alpine-base:latest - where the USER is applied.

Changing the uid for the existing image does not work for me. If you know a solution, this would be great.

[dliappis]

This is based on FROM docker.elastic.co/elasticsearch/elasticsearch-alpine-base:latest - where the USER is applied.

That is correct. This image is also in a public repo and you can find the Dockerfile here.

One strategy for changing the uid/gid in a dynamic way, when you build the image, is to use build args.
Here is an example:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN sed -r -i "s/^elasticsearch:x:\d+:\d+:(.*)\$/elasticsearch:x:$CUSTOM_UID:$CUSTOM_GID:\1/" /etc/passwd && \
    sed -r -i "s/^elasticsearch:x:\d+:(.*)\$/elasticsearch:x:$CUSTOM_GID:\1/" /etc/group && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch

For example, in a fresh VM where I have a user with uid=1001, gid=1001, I built a customized image with:

$ id
uid=1001(elastic) gid=1001(elastic) groups=1001(elastic),1002(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ docker build --build-arg CUSTOM_UID=$(id -u) --build-arg CUSTOM_GID=$(id -g) -t myes .

This results in a working container:

[elastic@fedora25vbox ~]$ docker run --rm myes
[2017-04-20T14:28:11,704][INFO ][o.e.n.Node               ] [] initializing ...
[2017-04-20T14:28:12,835][INFO ][o.e.e.NodeEnvironment    ] [pRZ41B3] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:0-8792783-ab2ae70e9d0911287fa77759bf0d62dcd73e613a03303d38f9e09327bc52ef9c)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [unknown], types [xfs]

[anshitabharti]

@dliappis if i try the same with 5.4.1 version i still face Permission denied on bin/es-docker
but in 5.3.0 this works. Any changes to be done in the Dockerfile ?

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.1
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN sed -r -i "s/^elasticsearch:x:\d+:\d+:(.*)\$/elasticsearch:x:$CUSTOM_UID:$CUSTOM_GID:\1/" /etc/passwd && \
    sed -r -i "s/^elasticsearch:x:\d+:(.*)\$/elasticsearch:x:$CUSTOM_GID:\1/" /etc/group && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch

[dliappis]

@anshitabharti Sure; starting with 5.4.0, the images are based on centos:7, so the /etc/{passwd,group} format is a bit different, but on the plus side it's easier to change the uid/gid like so:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.1
ARG CUSTOM_UID
ARG CUSTOM_GID
USER root
RUN groupmod -g $CUSTOM_UID elasticsearch && \
    usermod -u $CUSTOM_GID elasticsearch && \
    chown -R $CUSTOM_UID:$CUSTOM_GID /usr/share/elasticsearch
USER elasticsearch