Make docker environment variable parsing available in CLI tools

[tvernum]

Our docker entry point script has special logic to convert environment variables into -E options when starting Elasticsearch.

https://github.com/elastic/elasticsearch/blob/v7.3.0/distribution/docker/src/docker/bin/docker-entrypoint.sh#L41-L62

However this logic is not available when executing CLI commands via docker exec (or other similar commands like kubectl exec, etc) which surprises people, and makes it difficult to run some commands within a container.

e.g. https://discuss.elastic.co/t/how-to-use-elasticsearch-setup-passwords-with-tls-and-selfsigned-certificates/193795/7

I definitely don't think we should enable this environment parsing by default, but I wonder if there's value in having some sort of option for this.

A couple of ideas:

1. Have a --docker (or --parse-env) option to our CLI tools (this could be handled in elasticsearch-cli or elasticsearch-env)
2. Have a elasticsearch-parse-env script that generates -E options as output so that something like the below would work:

bin/elasticsearch-setup-passwords auto $(elasticsearch-parse-env)

I don't love any of this, but it's causing confusion right now so I think it's at least worth a discussion.

The CLI tools that I know of that make use of ES settings are (there may be others, I didn't dig too deeply):

• elasticsearch-node (node roles)
• elasticsearch-saml-metadata (realm settings)
• elasticsearch-setup-passwords (HTTP URL, HTTP SSL settings)
• elasticsearch-users (password hashing algorithm)

[elasticmachine]

Pinging @elastic/es-core-infra

[dliappis]

Wanted to provide some initial feedback here.

The Docker entrypoint script is only used when the container starts and is not involved in subsequent operations like docker exec'ing commands. (see also this SO comment).

We discussed on alternative in our weekly meeting which is to generalize the entrypoint script so that it can be used as a launcher for adhoc commands as well. Then it would be sufficient to run the following:

docker exec -ti <containername> docker-entrypoint.sh bin/elasticsearch-setup-passwords auto

[pugnascotia]

I had a quick look at some of the scripts. It looks like they all eventually source elasticsearch-env to tailor the environment, so if we moved the Docker-specific logic there, not only would we share it between the CLI tools, but there would be no need explicitly call an entrypoint i.e. docker exec would just work. Am I missing something, or does that sound reasonable?

[pugnascotia]

I suppose a downside of this would be that anyone running ES not in a container would need to watch their environment variables more closely, though the same applies to e.g. JAVA_HOME already.

[rjernst]

anyone running ES not in a container would need to watch their environment variables

We definitely don't want that. Perhaps we could conditionally do this based on ES_DISTRIBUTION_TYPE, which is set to "docker" already for docker images?

[pugnascotia]

Oh that's interesting, I wasn't aware of that env var. @dliappis and I taked about it and concluded that the explicit entrypoint approach was more appropriate, but actually being able to tailor the env var handling while consolidating the code would be a +1 from me.

[dliappis]

I am possibly missing something; my question is even if we can do env var parsing in elasticsearch-env how would this help as the cli tools just source it but still need the actual parameters to get passed as -E ... style cli args? This is ultimately what the entrypoint does, parsing env vars that seem related to elasticsearch and convert them to -E ... cli args.

[rjernst]

I believe the idea is we would move the conversion of the environment variabls into cli args into elasticsearch-env. We would need an env var that contains these converted settings which clis would then pass through. My suggestion was about making sure this only happens insider docker.