Child docker image failed on 4 or more layers

[prehor]

I found out that if my child Docker image has 4 or more file layers then Elasticsearch terminates with the following error:

# docker run elasticsearch-test:5.2.2
[2017-03-26T21:46:54,902][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-26T21:46:54,948][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:260) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:239) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more
Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_92-internal]
	at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:340) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:108) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:227) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more

There is my Dockerfile for elasticsearch-test image:

# Failed on 4 or more layers
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2

# Failed on 4 or more layers
# FROM elasticsearch:5.2.2-alpine

# Never failed
# FROM elasticsearch:5.2.2

RUN echo test 0 > ftest0
RUN echo test 1 > ftest1
# RUN echo test 2 > ftest2
# RUN echo test 3 > ftest3
# RUN echo test 4 > ftest4
# RUN echo test 5 > ftest5
# RUN echo test 6 > ftest6

COPY test0 dtest0/
COPY test1 dtest1/
# COPY test2 dtest2/
# COPY test3 dtest3/
# COPY test4 dtest4/
# COPY test5 dtest5/
# COPY test6 dtest6/

Elasticsearch failed on any arbitrary combination of 4 or more RUN or COPY layers.

All folders testN contains file with the same name. File contains approximately 15 random characters.

Out of curiosity I have tried images from Docker Hub - Alpine based image exhibits the same behavior but Debian based image seems that it is not affected.

I'm running on Docker for macOS 1.13.1 (15353) and OS X 10.11.6:

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 158
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: srwwkecxby41p6pv153ywy6y0
 Is Manager: true
 ClusterID: fdmz6u2uni02t5ozsxnkgdrkg
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.65.2
 Manager Addresses:
  192.168.65.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.8-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.787 GiB
Name: moby
ID: 3ODG:GSIF:YOIL:LAXH:DB32:YYZZ:PIIX:A5XB:WXR2:QZ3C:EVZA:NZ6A
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 41
 Goroutines: 136
 System Time: 2017-03-26T21:35:01.318890699Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

# uname -a
Darwin Marvin.local 15.6.0 Darwin Kernel Version 15.6.0: Mon Jan  9 23:07:29 PST 2017; root:xnu-3248.60.11.2.1~1/RELEASE_X86_64 x86_64

[dliappis]

I haven't been able to reproduce this on Linux[1].

Have you considered cleaning up stopped+dangling containers+images? You can use the new docker system df / docker system prune commands as explained here.

Also note that the default user in the image is elasticsearch and your COPY commands will retain whatever uid:gid the local file has on your host OS. To change the ownership you'd to add something like:

USER root
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/dtest0
USER elasticsearch

[1]

$ cat Dockerfile 
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2

RUN echo test 0 >ftest0
COPY test0 dtest0/

$ docker build . -t myes
Sending build context to Docker daemon 3.072 kB
Step 1/3 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
 ---> 89315294a341
Step 2/3 : RUN echo test 0 >ftest0
 ---> Using cache
 ---> df2f7444499a
Step 3/3 : COPY test0 dtest0/
 ---> d03df0cde01c
Removing intermediate container 9ca833ab10b1
Successfully built d03df0cde01c


$ docker run --rm myes
[2017-03-27T05:59:17,316][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-27T05:59:17,386][INFO ][o.e.e.NodeEnvironment    ] [ILEy7pa] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:1-9175146-d64fce121a2b42163502576e5ee0dd4359485ff4fb33ba434c7618b48925c71c)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [possibly], types [xfs]
[2017-03-27T05:59:17,387][INFO ][o.e.e.NodeEnvironment    ] [ILEy7pa] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-03-27T05:59:17,387][INFO ][o.e.n.Node               ] node name [ILEy7pa] derived from node ID [ILEy7pahST2f_EFzlATSew]; set [node.name] to override
[2017-03-27T05:59:17,389][INFO ][o.e.n.Node               ] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.9.10-100.fc24.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [aggs-matrix-stats]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [ingest-common]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [lang-expression]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [lang-groovy]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [lang-mustache]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [lang-painless]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [percolator]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [reindex]
[2017-03-27T05:59:18,411][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [transport-netty3]
[2017-03-27T05:59:18,412][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded module [transport-netty4]
[2017-03-27T05:59:18,412][INFO ][o.e.p.PluginsService     ] [ILEy7pa] loaded plugin [x-pack]
[2017-03-27T05:59:18,801][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-03-27T05:59:20,400][INFO ][o.e.n.Node               ] initialized
[2017-03-27T05:59:20,400][INFO ][o.e.n.Node               ] [ILEy7pa] starting ...
[2017-03-27T05:59:20,554][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 0e:6e:ea:e5:d9:e9:f4:ca
[2017-03-27T05:59:20,595][INFO ][o.e.t.TransportService   ] [ILEy7pa] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2017-03-27T05:59:20,599][INFO ][o.e.b.BootstrapChecks    ] [ILEy7pa] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-03-27T05:59:23,655][INFO ][o.e.c.s.ClusterService   ] [ILEy7pa] new_master {ILEy7pa}{ILEy7pahST2f_EFzlATSew}{Z_j7Z_UXT3CydFdxNPeb-w}{172.17.0.2}{172.17.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-03-27T05:59:23,680][INFO ][o.e.h.HttpServer         ] [ILEy7pa] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2017-03-27T05:59:23,680][INFO ][o.e.n.Node               ] [ILEy7pa] started
[2017-03-27T05:59:23,743][INFO ][o.e.g.GatewayService     ] [ILEy7pa] recovered [0] indices into cluster_state
[2017-03-27T05:59:24,066][INFO ][o.e.l.LicenseService     ] [ILEy7pa] license [c85286fa-da68-4b5c-ae2c-df08757dddea] mode [trial] - valid
[2017-03-27T05:59:30,594][INFO ][o.e.c.m.MetaDataCreateIndexService] [ILEy7pa] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [logstash, _default_, node, cluster_info, kibana]
[2017-03-27T05:59:30,699][INFO ][o.e.c.m.MetaDataCreateIndexService] [ILEy7pa] [.monitoring-es-2-2017.03.27] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [_default_, node, shards, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats]
[2017-03-27T05:59:30,867][INFO ][o.e.c.m.MetaDataMappingService] [ILEy7pa] [.monitoring-es-2-2017.03.27/vKBUK49EQVma5V022VaVAA] update_mapping [cluster_stats]
[2017-03-27T05:59:30,916][INFO ][o.e.c.m.MetaDataMappingService] [ILEy7pa] [.monitoring-es-2-2017.03.27/vKBUK49EQVma5V022VaVAA] update_mapping [node_stats]
[2017-03-27T05:59:41,003][INFO ][o.e.c.m.MetaDataMappingService] [ILEy7pa] [.monitoring-es-2-2017.03.27/vKBUK49EQVma5V022VaVAA] update_mapping [indices_stats]
[2017-03-27T05:59:41,028][INFO ][o.e.c.m.MetaDataMappingService] [ILEy7pa] [.monitoring-es-2-2017.03.27/vKBUK49EQVma5V022VaVAA] update_mapping [index_stats]
[2017-03-27T05:59:41,050][INFO ][o.e.c.m.MetaDataMappingService] [ILEy7pa] [.monitoring-es-2-2017.03.27/vKBUK49EQVma5V022VaVAA] update_mapping [cluster_stats]

[prehor]

You used only two file layers in your Dockerfile (one RUN and one COPY) which is running fine in my environment too. My problem started with four file layers in my image (any combination of four RUN or COPY commands in Dockerfile).

Dockerfile which I posted is only proof of concept for testing the problem. You can also use this Dockerfile with files created outside of Elasticsearch installation so permissions doesn't matter:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
RUN echo test 0 > /tmp/test0
RUN echo test 1 > /tmp/test1
RUN echo test 2 > /tmp/test2
RUN echo test 3 > /tmp/test3

I think that problem is connected to first raised exception:

Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_92-internal]
	at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:340) ~[elasticsearch-5.2.2.jar:5.2.2]

which seems to be related to the number of stacked filesystems. Maybe it's related to OpenJDK or musl libc in Alpine Linux, because Alpine based image from Docker Hub is also affected but Debian based image from Docker Hub is not affected.

And yes, my docker environment was pruned and I used freshly downloaded image from Elastic.

[dliappis]

@prehor I tried with your Dockerfile and still can't reproduce the problem on Linux (with docker 1.13.1):

Do you think you could re-try your test on a Linux machine/VM or another workstation?

$ cat >Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
RUN echo test 0 > /tmp/test0
RUN echo test 1 > /tmp/test1
RUN echo test 2 > /tmp/test2
RUN echo test 3 > /tmp/test3

$ docker build . -t myes
Sending build context to Docker daemon 3.072 kB
Step 1/5 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
 ---> 89315294a341
Step 2/5 : RUN echo test 0 > /tmp/test0
 ---> Running in 7e5b88726a8b
 ---> 9dee8be3c610
Removing intermediate container 7e5b88726a8b
Step 3/5 : RUN echo test 1 > /tmp/test1
 ---> Running in 841b1e4ead9e
 ---> 996bd7b12158
Removing intermediate container 841b1e4ead9e
Step 4/5 : RUN echo test 2 > /tmp/test2
 ---> Running in 57b5294a7dd4
 ---> d4febab17a72
Removing intermediate container 57b5294a7dd4
Step 5/5 : RUN echo test 3 > /tmp/test3
 ---> Running in cc6293f09243
 ---> 71779b7ed65d
Removing intermediate container cc6293f09243
Successfully built 71779b7ed65d

$ docker run --rm myes
[2017-03-30T09:27:42,565][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-30T09:27:42,640][INFO ][o.e.e.NodeEnvironment    ] [ZQJQ1Ww] using [1] data paths, mounts [[/ (/dev/mapper/docker-253:1-9175146-9542a2626355405b98d7a8df5901398253b2e5e41da3c60b874ef344aee9d179)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [possibly], types [xfs]
[2017-03-30T09:27:42,640][INFO ][o.e.e.NodeEnvironment    ] [ZQJQ1Ww] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-03-30T09:27:42,641][INFO ][o.e.n.Node               ] node name [ZQJQ1Ww] derived from node ID [ZQJQ1WwrTfWuuBvjV2XhNw]; set [node.name] to override
[2017-03-30T09:27:42,643][INFO ][o.e.n.Node               ] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.9.10-100.fc24.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [aggs-matrix-stats]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [ingest-common]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [lang-expression]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [lang-groovy]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [lang-mustache]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [lang-painless]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [percolator]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [reindex]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [transport-netty3]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded module [transport-netty4]
[2017-03-30T09:27:43,687][INFO ][o.e.p.PluginsService     ] [ZQJQ1Ww] loaded plugin [x-pack]
[2017-03-30T09:27:44,144][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-03-30T09:27:45,894][INFO ][o.e.n.Node               ] initialized
[2017-03-30T09:27:45,894][INFO ][o.e.n.Node               ] [ZQJQ1Ww] starting ...
[2017-03-30T09:27:46,089][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: a8:51:56:50:89:30:2a:6f
[2017-03-30T09:27:46,144][INFO ][o.e.t.TransportService   ] [ZQJQ1Ww] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2017-03-30T09:27:46,149][INFO ][o.e.b.BootstrapChecks    ] [ZQJQ1Ww] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-03-30T09:27:49,209][INFO ][o.e.c.s.ClusterService   ] [ZQJQ1Ww] new_master {ZQJQ1Ww}{ZQJQ1WwrTfWuuBvjV2XhNw}{-6yhYW44RNW0JHNxDQQEzQ}{172.17.0.2}{172.17.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-03-30T09:27:49,236][INFO ][o.e.h.HttpServer         ] [ZQJQ1Ww] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2017-03-30T09:27:49,236][INFO ][o.e.n.Node               ] [ZQJQ1Ww] started
[2017-03-30T09:27:49,312][INFO ][o.e.g.GatewayService     ] [ZQJQ1Ww] recovered [0] indices into cluster_state
[2017-03-30T09:27:49,695][INFO ][o.e.l.LicenseService     ] [ZQJQ1Ww] license [96130472-89ae-421f-9f55-0615758b0607] mode [trial] - valid
[2017-03-30T09:27:56,113][INFO ][o.e.c.m.MetaDataCreateIndexService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [_default_, node, shards, index_stats, index_recovery, cluster_state, cluster_stats, node_stats, indices_stats]
[2017-03-30T09:27:56,210][INFO ][o.e.c.m.MetaDataCreateIndexService] [ZQJQ1Ww] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [node, _default_, logstash, kibana, cluster_info]
[2017-03-30T09:27:56,401][INFO ][o.e.c.m.MetaDataMappingService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30/_0FPahRvSeebGKx9eNqjow] update_mapping [cluster_stats]
[2017-03-30T09:27:56,437][INFO ][o.e.c.m.MetaDataMappingService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30/_0FPahRvSeebGKx9eNqjow] update_mapping [node_stats]
[2017-03-30T09:28:06,550][INFO ][o.e.c.m.MetaDataMappingService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30/_0FPahRvSeebGKx9eNqjow] update_mapping [indices_stats]
[2017-03-30T09:28:06,588][INFO ][o.e.c.m.MetaDataMappingService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30/_0FPahRvSeebGKx9eNqjow] update_mapping [index_stats]
[2017-03-30T09:28:06,619][INFO ][o.e.c.m.MetaDataMappingService] [ZQJQ1Ww] [.monitoring-es-2-2017.03.30/_0FPahRvSeebGKx9eNqjow] update_mapping [cluster_stats]

[athollon]

Hello,

I got the same issue using @dliappis test dockerfile on docker for windows:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.2.2
RUN echo test 0 > /tmp/test0
RUN echo test 1 > /tmp/test1
RUN echo test 2 > /tmp/test2
RUN echo test 3 > /tmp/test3

uname -a
MINGW64_NT-10.0 LENOV766DSY 2.5.0(0.295/5/3) 2016-03-31 18:47 x86_64 Msys

Here is my docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1018
Server Version: 17.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.12-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 3.837 GiB
Name: moby
ID: QRPC:2AQX:6HMG:MNID:46XM:CWMY:KYAB:ZNT5:LHB4:J6F3:NRUN:GRIG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 29
 Goroutines: 44
 System Time: 2017-03-30T14:32:16.6494708Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
Live Restore Enabled: false

And here are the log :

[2017-03-30T14:27:52,620][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-30T14:27:52,682][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:260) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:239) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more
Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_92-internal]
	at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:340) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:108) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:227) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.node.Node.<init>(Node.java:232) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more

[dliappis]

@artholl I see your system has MINGW installed; I would avoid running any docker build from mingw (or cygwin) terminal. Additionally it should be noted that building non Windows images from a Windows host can cause issues, as indicated by the SECURITY WARNING message (see below).

While me and @Jarpy both run Windows workstations as well, we run Docker builds on Linux workstations or VMs to avoid such issues.

Despite all that, I was successful building a Docker image on top of the 5.3.0 image on a typical Windows 10 Pro workstation:

> type Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
RUN echo test 0 > /tmp/test0
RUN echo test 1 > /tmp/test1
RUN echo test 2 > /tmp/test2
RUN echo test 3 > /tmp/test3

> docker build . -t myes
Sending build context to Docker daemon 5.632 kB
Step 1/5 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
 ---> ccec59a7dd84
Step 2/5 : RUN echo test 0 > /tmp/test0
 ---> Running in 57f5be76e7d0
 ---> 11bc52d1d31c
Removing intermediate container 57f5be76e7d0
Step 3/5 : RUN echo test 1 > /tmp/test1
 ---> Running in be948c7ac4a0
 ---> f39d0a1be93b
Removing intermediate container be948c7ac4a0
Step 4/5 : RUN echo test 2 > /tmp/test2
 ---> Running in 30b0bd1d9f4d
 ---> 355c4611169a
Removing intermediate container 30b0bd1d9f4d
Step 5/5 : RUN echo test 3 > /tmp/test3
 ---> Running in c543433c71bf
 ---> 9f8f433c39c1
Removing intermediate container c543433c71bf
Successfully built 9f8f433c39c1
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

> docker run --rm myes
[2017-03-30T19:27:21,733][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-30T19:27:21,926][INFO ][o.e.e.NodeEnvironment    ] [k6Gp-Ae] using [1] data paths, mounts [[/ (none)]], net usable_space [51.7gb], net total_space [55.1gb], spins? [possibly], types [aufs]
[2017-03-30T19:27:21,927][INFO ][o.e.e.NodeEnvironment    ] [k6Gp-Ae] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-03-30T19:27:21,933][INFO ][o.e.n.Node               ] node name [k6Gp-Ae] derived from node ID [k6Gp-AekTLm2oIUyiGeX6A]; set [node.name] to override
[2017-03-30T19:27:21,934][INFO ][o.e.n.Node               ] version[5.3.0], pid[1], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/4.9.12-moby/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-03-30T19:27:26,906][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [aggs-matrix-stats]
[2017-03-30T19:27:26,906][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [ingest-common]
[2017-03-30T19:27:26,907][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [lang-expression]
[2017-03-30T19:27:26,907][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [lang-groovy]
[2017-03-30T19:27:26,907][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [lang-mustache]
[2017-03-30T19:27:26,907][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [lang-painless]
[2017-03-30T19:27:26,908][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [percolator]
[2017-03-30T19:27:26,908][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [reindex]
[2017-03-30T19:27:26,909][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [transport-netty3]
[2017-03-30T19:27:26,909][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded module [transport-netty4]
[2017-03-30T19:27:26,911][INFO ][o.e.p.PluginsService     ] [k6Gp-Ae] loaded plugin [x-pack]
[2017-03-30T19:27:33,270][INFO ][o.e.n.Node               ] initialized
[2017-03-30T19:27:33,270][INFO ][o.e.n.Node               ] [k6Gp-Ae] starting ...
[2017-03-30T19:27:33,548][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 65:a8:29:f7:84:a8:96:d5
[2017-03-30T19:27:33,664][INFO ][o.e.t.TransportService   ] [k6Gp-Ae] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2017-03-30T19:27:33,675][INFO ][o.e.b.BootstrapChecks    ] [k6Gp-Ae] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-03-30T19:27:35,005][INFO ][o.e.m.j.JvmGcMonitorService] [k6Gp-Ae] [gc][young][1][5] duration [980ms], collections [1]/[1.5s], total [980ms]/[2s], memory [97.1mb]->[107.5mb]/[1.9gb], all_pools {[young] [58.5mb]->[33.9mb]/[133.1mb]}{[survivor] [16.6mb]->[14.4mb]/[16.6mb]}{[old] [21.9mb]->[59.6mb]/[1.8gb]}
[2017-03-30T19:27:35,008][WARN ][o.e.m.j.JvmGcMonitorService] [k6Gp-Ae] [gc][1] overhead, spent [980ms] collecting in the last [1.5s]
[2017-03-30T19:27:36,770][INFO ][o.e.c.s.ClusterService   ] [k6Gp-Ae] new_master {k6Gp-Ae}{k6Gp-AekTLm2oIUyiGeX6A}{VHO1WsjhTd2_lKsAD4mcuw}{172.17.0.2}{172.17.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-03-30T19:27:36,834][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [k6Gp-Ae] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2017-03-30T19:27:36,840][INFO ][o.e.n.Node               ] [k6Gp-Ae] started
[2017-03-30T19:27:36,965][INFO ][o.e.g.GatewayService     ] [k6Gp-Ae] recovered [0] indices into cluster_state
[2017-03-30T19:27:38,025][INFO ][o.e.l.LicenseService     ] [k6Gp-Ae] license [7e24f59a-3464-465a-b3aa-44006000acbb] mode [trial] - valid
[2017-03-30T19:27:43,554][INFO ][o.e.c.m.MetaDataCreateIndexService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [shards, _default_, node, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats]
[2017-03-30T19:27:43,879][INFO ][o.e.c.m.MetaDataCreateIndexService] [k6Gp-Ae] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [logstash, _default_, node, kibana, cluster_info]
[2017-03-30T19:27:44,489][INFO ][o.e.c.m.MetaDataMappingService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30/GVuO-x30ST2Iav8pWk4iUw] update_mapping [cluster_stats]
[2017-03-30T19:27:45,610][WARN ][o.e.m.j.JvmGcMonitorService] [k6Gp-Ae] [gc][young][11][6] duration [1s], collections [1]/[1.5s], total [1s]/[3s], memory [192.2mb]->[110.3mb]/[1.9gb], all_pools {[young] [118.1mb]->[2.4mb]/[133.1mb]}{[survivor] [14.4mb]->[11.5mb]/[16.6mb]}{[old] [59.6mb]->[96.4mb]/[1.8gb]}
[2017-03-30T19:27:45,610][WARN ][o.e.m.j.JvmGcMonitorService] [k6Gp-Ae] [gc][11] overhead, spent [1s] collecting in the last [1.5s]
[2017-03-30T19:27:45,747][INFO ][o.e.c.m.MetaDataMappingService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30/GVuO-x30ST2Iav8pWk4iUw] update_mapping [node_stats]
[2017-03-30T19:27:53,427][INFO ][o.e.c.m.MetaDataMappingService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30/GVuO-x30ST2Iav8pWk4iUw] update_mapping [index_stats]
[2017-03-30T19:27:53,501][INFO ][o.e.c.m.MetaDataMappingService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30/GVuO-x30ST2Iav8pWk4iUw] update_mapping [cluster_stats]
[2017-03-30T19:27:53,557][INFO ][o.e.c.m.MetaDataMappingService] [k6Gp-Ae] [.monitoring-es-2-2017.03.30/GVuO-x30ST2Iav8pWk4iUw] update_mapping [indices_stats]

[prehor]

Today I have upgraded my Docker for mac to 17.03.1-ce-mac5 (16048) and reset my docker host to factory default (I'm sorry but I don't have any other docker host available now). Unfortunately the problem still exists:

> cat Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:%%BASE_IMAGE_TAG%%
RUN echo test 0 > /tmp/ftest0
RUN echo test 1 > /tmp/ftest1
RUN echo test 2 > /tmp/ftest2
RUN echo test 3 > /tmp/ftest3

> docker build -t es-test .
Sending build context to Docker daemon  16.9 kB
Step 1/5 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
5.3.0: Pulling from elasticsearch/elasticsearch
3690ec4760f9: Pull complete
f52154c3d3fc: Pull complete
4075cc5db14a: Pull complete
2d52f619a2d6: Pull complete
8bb7b04253b3: Pull complete
1cb5a2a95e15: Pull complete
e2f60cff0f2f: Pull complete
9eb2cf345f74: Pull complete
ff68f5fa0401: Pull complete
90f6e7841041: Pull complete
Digest: sha256:56ac964338bc74f3874d63271433f6555648d55405a89c96f56a18dee48456eb
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.3.0
 ---> ccec59a7dd84
Step 2/5 : RUN echo test 0 > /tmp/ftest0
 ---> Running in b784a0ebafba
 ---> 879153c89167
Removing intermediate container b784a0ebafba
Step 3/5 : RUN echo test 1 > /tmp/ftest1
 ---> Running in ccd049d13239
 ---> afb0028249b2
Removing intermediate container ccd049d13239
Step 4/5 : RUN echo test 2 > /tmp/ftest2
 ---> Running in 774a8ea8bd63
 ---> d99b7a9a755e
Removing intermediate container 774a8ea8bd63
Step 5/5 : RUN echo test 3 > /tmp/ftest3
 ---> Running in d00844f41d93
 ---> 500b42458aab
Removing intermediate container d00844f41d93
Successfully built 500b42458aab

> > docker run es-test
[2017-03-30T20:01:51,750][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-30T20:01:51,829][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.0.jar:5.3.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:260) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:238) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.0.jar:5.3.0]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:239) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:238) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.0.jar:5.3.0]
	... 6 more
Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_92-internal]
	at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:340) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:108) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:227) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:238) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.0.jar:5.3.0]
	... 6 more

I have tried to squash my docker image and it helps:

> docker build -t es-test --squash .
Sending build context to Docker daemon  16.9 kB
Step 1/5 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
 ---> ccec59a7dd84
Step 2/5 : RUN echo test 0 > /tmp/ftest0
 ---> Using cache
 ---> 879153c89167
Step 3/5 : RUN echo test 1 > /tmp/ftest1
 ---> Using cache
 ---> afb0028249b2
Step 4/5 : RUN echo test 2 > /tmp/ftest2
 ---> Using cache
 ---> d99b7a9a755e
Step 5/5 : RUN echo test 3 > /tmp/ftest3
 ---> Using cache
 ---> 500b42458aab
Successfully built 500b42458aab

> docker run es-test
[2017-03-30T20:03:14,585][INFO ][o.e.n.Node               ] [] initializing ...
[2017-03-30T20:03:15,226][INFO ][o.e.e.NodeEnvironment    ] [5oozheL] using [1] data paths, mounts [[/ (overlay)]], net usable_space [58.2gb], net total_space [62.7gb], spins? [possibly], types [overlay]
[2017-03-30T20:03:15,227][INFO ][o.e.e.NodeEnvironment    ] [5oozheL] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-03-30T20:03:15,228][INFO ][o.e.n.Node               ] node name [5oozheL] derived from node ID [5oozheL1SgK7vIY9uIk17A]; set [node.name] to override
[2017-03-30T20:03:15,229][INFO ][o.e.n.Node               ] version[5.3.0], pid[1], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/4.9.13-moby/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-03-30T20:03:19,691][INFO ][o.e.p.PluginsService     ] [5oozheL] loaded module [aggs-matrix-stats]
[2017-03-30T20:03:19,692][INFO ][o.e.p.PluginsService     ] [5oozheL] loaded module [ingest-common]
...

This reproduces for me on Docker for Windows version 17.03.0-ce-win1, (run from Powershell).

It does not reproduce on Docker for Linux version 17.03.0-ce.

Interesting. I think the Docker VM, as used by OSX and Windows is introducing the problem.

The error is also at run time, not at build time. The same image runs on my Linux host, but not on my "Windows" host. I suspect a limit, that can hopefully be adjusted.

Ah. My Linux host is using aufs, the Windows/Alpine host is using overlay2. This is almost certainly an overlay problem.

See: https://bugs.openjdk.java.net/browse/JDK-8165852

The exception is thrown from here, a function which, among things, iterates over the contents of /proc/mounts. Just look at the size of the entry for the overlay filesystem:

bash-4.3$ cat /proc/mounts  | grep overlay
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/A3AFSC7UMLGP3RSJ3GLYU5QISA:/var/lib/docker/overlay2/l/U5DS5KSEVFLCSMRMQHSP4WKPV2:/var/lib/docker/overlay2/l/JB44VJRVYXDEUFSPQBQZC5KSTD:/var/lib/docker/overlay2/l/6FNQPK5TDM4O64MBPAI4WSH4VT:/var/lib/docker/overlay2/l/B53IOX2SGG24VUNC4DQPW4FIUO:/var/lib/docker/overlay2/l/GSSGTPYU2NTJQADDFGKJSLIX6F:/var/lib/docker/overlay2/l/3F5GHL6JSC3IQEOLJ6UOLR3ALD:/var/lib/docker/overlay2/l/ENNR2KKWTLXFQZRKBROP226564:/var/lib/docker/overlay2/l/NSFVF5QLCE5SUVVKZ3KQ6K7L4Y:/var/lib/docker/overlay2/l/XANLMD5PTSTIUK2LUQ5IOJZ4TY:/var/lib/docker/overlay2/l/CXHNN3LK73XRON5BLFGEPOIJIS:/var/lib/docker/overlay2/l/UZAEVH44OCMYQA3UUPMMWVPE6B:/var/lib/docker/overlay2/l/D2WZ6SYTS5FWUYP2LASIC3OYQ5:/var/lib/docker/overlay2/l/V4T4HTM6GAN3CVBNFP7HGLKZE7:/var/lib/docker/overlay2/l/63NJ5LBFI3WM4AGSJFMCLG7KMK,upperdir=/var/lib/docker/overlay2/46ae7377ee511513605cc359236c3ee8eab96491f69f244f220329ab0b469899/diff,workdir=/var/lib/docker/overlay2/46ae7377ee511513605cc359236c3ee8eab96491f69f244f220329ab0b469899/work 0 0

With four layers in the test Dockerfile, but not with three, it's a little over 1 KiB. Very suspicious.

Sorry to bug you, @jasontedor, but it's been 10 years since I wrote any Java, so I have no idea if I'm on the right track. Do you see anything obvious in findMountEntry() or getMountEntries() that would make them choke on a large line in /proc/mounts? Thanks.

[prehor]

Excellent work @Jarpy. Docker for mac use overlay2 storage driver too (docker/for-mac#516).

I have switched storage driver to aufs and the problem is gone (but some other may arise).

Thanks @prehor. We'll be switching to glibc soon, so this should get solved properly!

Actually, my money is on the native call to getmntent(3).

Oh look! What's the chances we don't have this patch in our musl libc?

https://git.musl-libc.org/cgit/musl/commit/src/misc/mntent.c?id=05973dc3bbc1aca9b3c8347de6879ed72147ab3b