Expose logs to systemd users

[DaveCTurner]

We get more than a few posts on the forums like this one complaining about Elasticsearch stopping shortly after starting and quoting the output from systemd:

 elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-08-26 09:39:21 EST; 5min ago
     Docs: http://www.elastic.co
  Process: 881 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 881 (code=exited, status=78)

Aug 26 09:39:09 p2e systemd[1]: Started Elasticsearch.
Aug 26 09:39:10 p2e elasticsearch[881]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be r
Aug 26 09:39:21 p2e systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 26 09:39:21 p2e systemd[1]: elasticsearch.service: Unit entered failed state.
Aug 26 09:39:21 p2e systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Often (as in the post linked above) this is after adjusting network.host and the reason Elasticsearch is shutting down is due to a bootstrap check that now fails. However there is no indication here that there is more information in some other logs, and volunteers on the forum must spend time helping users to find this extra information.

Can we improve this? For instance, can we expose the location of the Elasticsearch logs in this output? It looks like stderr is exposed above, because of the OpenJDK 64-Bit Server VM warning line. Could we emit a message on stderr that points the user to the log file? Or maybe we can expose stdout here too?

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

Or maybe we can expose stdout here too?

Relates #17220

[pugnascotia]

Interestingly, if I replicate the problem of setting network.host and thereby causing the bootstrap checks to fail, I get the following printed to STDERR (on master):

$ ./bin/elasticsearch > /dev/null
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

Whereas in the snippet above we only see the JVM warning line. I should point out however, that I get this from running bin/elasticsearch directory from a tarball, so the situation may be different when running through systemd.

[jasontedor]

@pugnascotia Note that in our systemd startup script we pass --quiet which closes the standard streams:

elasticsearch/server/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java

Lines 314 to 323 in a2ccb52

	final boolean closeStandardStreams = (foreground == false) || quiet;
	try {
	if (closeStandardStreams) {
	final Logger rootLogger = LogManager.getRootLogger();
	final Appender maybeConsoleAppender = Loggers.findAppender(rootLogger, ConsoleAppender.class);
	if (maybeConsoleAppender != null) {
	Loggers.removeAppender(rootLogger, maybeConsoleAppender);
	}
	closeSystOut();
	}

and

elasticsearch/server/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java

Lines 344 to 346 in a2ccb52

	if (closeStandardStreams) {
	closeSysError();
	}

[jasontedor]

I think closing system error in any situation is a mistake. 🤷‍♀

[pugnascotia]

So we could just stop closing system error, and that should let any errors through to e.g. systemd. We could also print a message about the logs dir, for completeness.

[jasontedor]

Yes, I don't think we should passthrough stdout to the journal, only stderr.