Introduce Docker image that bundles plugins

[pugnascotia]

Related:

• Plugin Installation Permissions Issue Plugin Installation Permissions Issue #40969
• Plugin install fail with Operation not permitted on OpenShift #1801

The Elasticsearch Docker image does not bundle any plugins at the moment. This issue is to discuss whether it would be useful to provide a further Docker image that bundles the official plugins, and provides a mechanism to enable them via e.g. an environment variable.

Such an image would also be useful to Cloud, who currently have to perform their own bundling, which I understand is a cumbersome process. I am assuming that the Stack release process would have an easier time of building the plugins that match the particular release and bundling them into the Docker image.

### Implementation sketch

We could provide images, say elasticsearch/elasticsearch-with-plugins (and an analogue for OSS), which provides all official plugins in /usr/share/elasticsearch/plugins-archive. The bootstrap script would detect an environment variable such as ELASTIC_PLUGINS, and use this to enable the specified plugins before starting Elasticsearch.

Customers who need to bundle their own plugins could simple copy their plugins to the archive directory, and reuse the install mechanism.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Plugins)

[jasontedor]

We discussed this during our weekly team discussion meeting. There are several concerns here:

• Adding plugins to our existing Docker images would entail increasing the image size. This is an area where we already get a lot of concerns from the community. We do think we should investigate reducing the size of our image, and this change would move us in the wrong direction in that regard.
• Thus, it would entail producing yet another image, which raises concerns about maintaining yet another set of images. It would also be a deviation from the fact that we aim to keep our Docker images aligned with the other distributions that we produce (the archive and package distributions) yet we would not want to also build archives and packages that bundle all the plugins.
• We (and most of the Docker community) don't like the idea of modifying the containers at runtime (to install the plugins in the container); we recommend that users of our Docker containers instead derive an image that installs the necessary plugins (to create a new image), and then deploy that.
• Traditionally, the reasons for bundling all the plugins in the Cloud base image are two fold:
  • slowness of installing plugins in Cloud delaying container bootstrap
  • offline installs in ECE
• We think that there are other solutions worth considering for these problems.
  • In the past we had discussed removing CPU limits in Cloud during container bootstrap, something that we understand might be in place today, can @pmoust confirm this?
  • Separately, perhaps we need to reconsider a bulk mode to install multiple plugins at once, an idea from the past that is worth reconsidering in this context.
  • Perhaps for ECE, we could ship a lightweight proxy that serves the plugins locally.

So, rather than producing another image as part of our build that raises some concerns for us, we would like that we revisit the initial motivations for producing a with plugins image.

[pmoust]

In the past we had discussed removing CPU limits in Cloud during container bootstrap, something that we understand might be in place today, can @pmoust confirm this?

I confirm that this was implemented in 0bcd13415431696242dc52759613f1753f99d623 and in ESS since March 2019. That said, it took a bit over 60s to install 11 plugins (enabling every eligible choice) in a 1G RAM 7.5.1 single node deployment.

Pinging @nachogiljaldo @pboutes in case I am missing something on that.

[nachogiljaldo]

@pmoust yes, that boosting has been in place for a long time (longer than March 2019 I believe). Of course the idea of allowing installing several plugins at once would be very nice as it would speed up the startup time when plugins are selected.

However, IMHO, the main problem falls under the goal/idea of having Cloud run the stack docker images and the current need for us to have the plugins as part of the image. As mentioned before, stack docker images do not bundle any plugins, but we need certain plugins to be bundled.
Downloading them dynamically is currently not a feasible alternative due to:

offline installs in ECE

On an air-gapped ECE installation, the script would try to download the plugins and it would fail and with it the startup of the deployment.

A proxy, and accordingly added firewall rules, would fix the issue, but it would also complicate the installation of ECE and the infrastructure itself.
I am not sure that is a desirable price to pay for using the exact same image. If there is no agreement, I would rather vote for us extending the stack image to add the plugins (and only that, moving all the rest to the stack).

FWIW, there are other things that we could do:

• bundle the plugins with the stackpack (only mentioned for completeness, I think we shouldn't go this way because it would also complicate Cloud's infrastructure)
• modifying the air-gapped environment installation docs/tooling to provide a script that would, in that case, generate a new image downloading the necessary plugins, etc. This one, with some good tooling could be an option to consider.

[jasontedor]

Of course the idea of allowing installing several plugins at once would be very nice as it would speed up the startup time when plugins are selected.

I opened #50924.

However, IMHO, the main problem falls under the goal/idea of having Cloud run the stack docker images and the current need for us to have the plugins as part of the image.

That's the piece that we are challenging (the need to have the plugins as part of the image).

A proxy, and accordingly added firewall rules, would fix the issue, but it would also complicate the installation of ECE and the infrastructure itself.

As mentioned in the initial comment, the idea is that the proxy would serve them locally, not reach out to the Internet.

modifying the air-gapped environment installation docs/tooling to provide a script that would, in that case, generate a new image downloading the necessary plugins, etc. This one, with some good tooling could be an option to consider.

This is fine with us too.

[mieciu]

I think it's worth implementing multiple plugins removal as well. Not only it'd be super useful for Cloud, but also we'd have consistent interface for plugin management.

@jasontedor WDYT?

[jasontedor]

@mieciu Sorry for the delay. Can you expand on why it'd be super-useful for Cloud? One of the challenges with batching plugin removal into a single invocation of the elasticsearch-plugin tool is that it's difficult to do in a transactional way.