[CI] DockerTests test021InstallPlugin failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-packaging #3143 / rhel-7_packaging-tests-unix
• elasticsearch-periodic-packaging #3127 / centos-7_packaging-tests-unix
• elasticsearch-pull-request #26557 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #26553 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-periodic-packaging #3119 / oraclelinux-8_packaging-tests-unix
• elasticsearch-pull-request #26489 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #26470 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #26452 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #26422 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #26410 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample

Reproduction Line:

null

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.packaging.util.Shell$ShellException: Command was not successful: [bash -c docker exec --tty bc157673d677dca6c36ebf08e4daac0e0a48bc3623c0334777c1268ceefee40b /usr/share/elasticsearch/bin/elasticsearch-plugin list]
   result: exitCode = [137] stdout = [analysis-icu] stderr = []

Issue Reasons:

• [main] 10 failures in test test021InstallPlugin (1.2% fail rate in 842 executions)
• [main] 4 failures in step ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample (3.7% fail rate in 109 executions)
• [main] 3 failures in step rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample (2.8% fail rate in 108 executions)
• [main] 3 failures in pipeline elasticsearch-periodic-packaging (33.3% fail rate in 9 executions)
• [main] 7 failures in pipeline elasticsearch-pull-request (6.4% fail rate in 109 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

The underlying error is the following:

java.lang.IllegalStateException: failed to load plugin analysis-icu while checking for jar hell
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:353)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:294)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeServiceProvider.newPluginService(NodeServiceProvider.java:55)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.createEnvironment(NodeConstruction.java:454)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.Node.<init>(Node.java:192)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/.installing-10801102274982928062/analysis-icu-8.15.0-SNAPSHOT.jar
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1858)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1491)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:251)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:180)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:345)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:316)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:255)
	at org.elasticsearch.base@8.15.0-SNAPSHOT/org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:221)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:349)
	... 9 more

@elastic/es-core-infra I'm not familiar with this .installing file and why it would be missing. As far as the logs show, the plugin was successfully installed. Has anything changes regarding plugin installation recently?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

When a plugin is installed the zip is extracted into a temporary .installing directory. We check jarhell on the plugin there before moving it to the final plugin directory.

This test failed because when we tried to access a jar from the plugin the file system said it doesn't exist. Yet the way we know about the jar in the first place is we listed the extracted plugin dir. So something is messed up with the filesystem such that the file was there, and then it wasn't. From the plugin installer point of view, there is nothing we can do about that.

[mark-vieira]

I have a suspicion this has something to do with some disk space issues in CI we're seeing. Let's get those sorted out and see if these failures continue.