systemd service opendistro-performance-analyzer.service has the wrong binary path configured

[perkons]

OS, OpenDistro and Java version used:
Used the documentation here https://opendistro.github.io/for-elasticsearch-docs/docs/install/rpm/ to install rmp packages from repo https://d3g5vo6xdbdb9a.cloudfront.net/yum/opendistroforelasticsearch-artifacts.repo

OS: CentOS Linux release 8.2.2004
OpenDistro: opendistroforelasticsearch-1.11.0-1.noarch
Performance Analyzer: opendistro-performance-analyzer-1.11.0.0-1.noarch
Java: java-11-openjdk-11.0.8.10-0.el8_2.x86_64

First this problem appeared in opendistroforelasticsearch-1.10.1 (opendistro/for-elasticsearch#94)

What happened:
After upgrading OpenDistro from previous to 1.10.1 or 1.11.0 Performance Analyzer systemd service Failed to start.

This is a rpm package build issue as it seems that the binary path has changed from /usr/share/elasticsearch/bin/performance-analyzer-agent-cli to /usr/share/elasticsearch/plugins/opendistro_performance_analyzer/bin/performance-analyzer-agent-cli
but the systemd service file has not been updated.

The file /usr/lib/systemd/system/opendistro-performance-analyzer.service looks like this after install:

[Unit]
Description=Opendistro for Elasticsearch Performance Analyzer
PartOf=elasticsearch.service

[Service]
ExecStart=/usr/share/elasticsearch/bin/performance-analyzer-agent-cli
Restart=on-failure
User=elasticsearch
Group=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
WorkingDirectory=/usr/share/elasticsearch

[Install]
WantedBy=elasticsearch.service

but /usr/share/elasticsearch/bin/performance-analyzer-agent-cli does not exist.
It has been moved to /usr/share/elasticsearch/plugins/opendistro_performance_analyzer/bin/performance-analyzer-agent-cli
also it is not executable.

What you expected to happen:
Opendistro for Elasticsearch Performance Analyzer starts.

[ktkrg]

Thanks for finding this issue @perkons ! If you have a PR ready, we're happy to review and accept it. Contributions welcome!

[perkons]

@ktkrg I made PR #254

After this fix the service still fails with the error:
Dec 7 16:46:07 esnode1 performance-analyzer-agent-cli[1840903]: Error: Could not find or load main class com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp Dec 7 16:46:07 esnode1 performance-analyzer-agent-cli[1840903]: Caused by: java.lang.ClassNotFoundException: com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp

Also, I am not sure what is done with the package build as the postrm script removes everything that postinst copies. Please look here https://github.com/opendistro-for-elasticsearch/performance-analyzer/blob/master/packaging/rpm/postrm#L13
I think someone needs to take a proper look at the pacakge build as I do not quite understand the reasoning behind some of the script parts.

Please note that I also only changed rpm build, I did not touch deb.

[opoplawski]

Remember that %postuninstall runs on package upgrade as well. See https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/ for more information. Not sure if this is the issue or not.

rpm verify fails as well:

# rpm -V opendistro-performance-analyzer-1.11.0.0-1.noarch
missing     /usr/share/elasticsearch/bin/performance-analyzer-agent-cli

[perkons]

@opoplawski the package build in this case is pretty incorrect, there should be no use of mv and cp, instead the spec file should be used with something like this:

Source1: opendistro-performance-analyzer.service
Source2: performance-analyzer-agent-cli
%install
install -D -m 644 %{SOURCE1} %{buildroot}%{_unitdir}/opendistro-performance-analyzer.service
install -D -m 755 %{SOURCE2} /usr/share/elasticsearch/bin/performance-analyzer-agent-cli

I just upgraded to 1.12.0 and the issue is still there, the binary is not here /usr/share/elasticsearch/bin/performance-analyzer-agent-cli. And even if I copy it manually and set exacutable, I get:

Dec 15 15:09:29 es1 performance-analyzer-agent-cli[2042984]: Error: Could not find or load main class com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp
Dec 15 15:09:29 es1 performance-analyzer-agent-cli[2042984]: Caused by: java.lang.ClassNotFoundException: com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp

Also I do not understand there reason behind this https://github.com/opendistro-for-elasticsearch/performance-analyzer/blob/master/packaging/rpm/postrm as it removes everything.

[gferrette]

Hello!

I'm getting the same error:

Failed at step EXEC spawning /usr/share/elasticsearch/bin/performance-analyzer-agent-cli: No such file or directory

Is there any workaround?

Thanks.

[perkons]

@gferrette you could copy the file manually and set exec:

# cp /usr/share/elasticsearch/plugins/opendistro_performance_analyzer/bin/performance-analyzer-agent-cli  /usr/share/elasticsearch/bin/
# chmod +x /usr/share/elasticsearch/bin/performance-analyzer-agent-cli

unfortunatelly there is another problem. You will get a new error:

Dec 15 15:09:29 es1 performance-analyzer-agent-cli[2042984]: Error: Could not find or load main class com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp
Dec 15 15:09:29 es1 performance-analyzer-agent-cli[2042984]: Caused by: java.lang.ClassNotFoundException: com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp

[perkons]

I just did redeploy my lab cluster. That means fresh installed CentOS 8 and Opendistro 1.12.0 and Performance Analyzer is working just fine. This proves my conclusion on the package build. It is wrong as you cannot upgrade the cluster without breaking components/plugins.

[LukaszWasko]

We have also the same error after upgrading to v.1.12:

May  5 16:30:03 dn109.krk.prod.elk.icore performance-analyzer-agent-cli: Error: Could not find or load main class com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp
May  5 16:30:03 dn109.krk.prod.elk.icore performance-analyzer-agent-cli: Caused by: java.lang.ClassNotFoundException: com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp

[Khadarbaba09]

Reinstalling the opendistro-performance-analyzer package fixed the issue for me.

After reinstallation, performance-analyzer-agent-cli is moved back to /usr/share/elasticsearch/bin/ which fixed the issue.