Reduce the surface area of the Elasticsearch Docker image

[jasontedor]

Today our Docker image is based on the centos:7 base image. This leaves a large surface area of binaries and libraries that we don’t need, but exposes us to noisy vulnerability scans (with issues that don’t actually impact the security of the image). We haven’t made much of an effort to slim this surface area down.

One reason we chose this image over others (e.g., Ubuntu-derived images) is perceived better support of the JDK, because Red Hat has long been heavily involved in OpenJDK. This reason is a non-factor, now that we use the bundled JDK in the images. There was also a desire to have consistency with other images in the stack. I’m less convinced of the value of this compared to other factors but it is something to keep in our minds.

Note that a non-goal for this issue is to reduce the physical size of the image. While that is something to consider and will likely result from reducing the surface area of the image, it is separate to this issue to consider reducing the physical size of the image.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[Code-Glitch]

Would using Alpine Linux make sense? A tiny distro with security in mind.

Here's some comments about performance:
https://nickjanetakis.com/blog/benchmarking-debian-vs-alpine-as-a-base-docker-image

[jasontedor]

Our images use to be based on the official Alpine Linux images. We had significant challenges with JVM bugs on the musl C library, and some other issues, so moved away. As of JDK 13 (the latest major release of the JDK), there is not official support for the JVM on Alpine Linux.

[pugnascotia]

docker-slim looks promising here. It analyzes a running image to see what files are actually used, and removes everything else. Unfortunately, slimtoolkit/slim#26 gets in the way here. I'm attempting to work around it, because the reduction in surface area (and therefore image size) is significant.

[droberts195]

docker-slim looks promising here.

If it literally looks at the loaded dependencies of the running processes at a moment in time then please make sure an ML job is opened at that moment in time, otherwise it may well remove dependencies that ML needs.

[pugnascotia]

Yeah, I am keeping ML in mind. I'm just exploring options at the moment.

[pugnascotia]

I also found GoogleContainerTools/distroless, which is interesting but probably unsuitable for us.

• It's Debian-based
• It's super minimal - you don't even get a shell or a package manager

[jasontedor]

Yeah, distroless has been raised for discussion before, but we need a Bash shell.

[probakowski]

Have we considered using distroless and installing bash in it? Should still be lighter than full OS. Or maybe even BusyBox would suffice? It uses ash so unless we have something bash-specific we should be fine

[pugnascotia]

Well, you can run gcr.io/distroless/base-debian10:debug which adds busybox. Might be worth an experiment.

[pugnascotia]

OK, I managed to hack code enough to build an image, but it was pretty awful and in the env Java wouldn't start due to a missing libz.so.1. So I think we can rule that out for now.

I did try their Java version, but it seems unuseable unless you have a fat jar ready to run.

[pugnascotia]

The maintainer(s) of docker-slim are keen to make it work for us, so I'll see how that goes.

I did try hacking something up (1) with a custom Rust / itnotify tool, and then (b) a bunch of bash and the inotifywait tool, but I couldn't get the resulting tar of accessed files to load into an image. I suspect docker-slim is doing something more clever here.