Investigate using UTF-8 console output in docker

[albertzaharovits]

This issue is only concerned with the encoding of the stdout/stderr streams for docker run -it.

The JVM has internal logic to deduce the character encoding to use for its display to stdout/stderr, which is influenced by the system locale configuration. Some information on this logic is available at JEP 400: UTF-8 by Default.
On every platform there are probably good (or obscure) reasons to default to non-UTF-8. We shoud not mess with that.
But, from my testing, the stdout encoding for the JVM inside docker run -it is US-ASCII even if it is UTF-8 if the same JVM (ES node) is run on the host (I can force the encoding in docker to UTF-8 by setting -e "LC_ALL=C.UTF-8").

I would like to propose that we affect the docker JVM's encoding based on the host's terminal preferred encoding (locale setting).

Rationale for UTF-8:
In some circumstances it is desirable to output flashy Unicode characters to highlight auto-generated configuration in the terminal, ala Display security auto-configuration with fancy unicode

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

I'm flagging Core/Infra here as well since changing the JVM encoding has broad implications that goes beyond just packaging and the output from our Docker image.

[albertzaharovits]

changing the JVM encoding

TBC the JVM does its thing to deduce the encoding to use for the console. I'm not proposing to mess with that.

What I'm proposing is that when attaching the terminal to the docker container (docker run -it), the JVM works as it would when started from the host terminal, wrt the console encoding.
More generally, the docker "locale" environment should reflect the host "locale" environment when docker run -it.

[pugnascotia]

More generally, the docker "locale" environment should reflect the host "locale" environment when docker run -it.

I'm not sure it's possible to do this automatically. The user would need to explicitly pass the locale when executing the container e.g. with an env var.

Note also that the ubuntu:20.04 image ships with no extra locale information. We can add e.g. en-US.UTF-8, but currently it's POSIX.

[albertzaharovits]

Note also that the ubuntu:20.04 image ships with no extra locale information. We can add e.g. en-US.UTF-8, but currently it's POSIX.

I see that it ships with C.UTF-8, and I can start the container with docker run -it -e "LC_ALL=C.UTF-8" elasticsearch:test and get the expected behavior (the JVM uses UTF-8 to encode its stdout, rather than ASCII).

It might not be desirable for other reasons, or, as you say, even possible, to do it automatically based on the host's locale (which is what I haven't investigated).

[mark-vieira]

How does this play out with JEP 400. As per our typical process, we'll bump the bundled JDK to Java 18 on release, which if I understand correctly, will mean we'll use UTF-8 encoding by default on all platforms unless otherwise overriden in jvm.options. We wouldn't be using the "host" value here, but we'd be using a consistent one across all distributions.