[CI] Numerous :build-tools:integTest fail in darwin builds

[droberts195]

There have been several darwin builds where many integration tests of the build tools have failed with little indication of the cause:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/350/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+multijob-darwin-compatibility/124/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+multijob-darwin-compatibility/125/console

I tried to reproduce this locally on my mac using ./gradlew :build-tools:integTest on both the 7.0 and master branches but that command was successful for both branches.

The errors in the build log are not very expansive about what went wrong:

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testViolationFoundAndCompileOnlyIgnored FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testWithEmptyRules FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testJarHellWithJDK FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testClassNotFoundAndCompileOnlyIgnored FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.JarHellTaskIT > testJarHellDetected FAILED
    java.lang.AssertionError at JarHellTaskIT.java:34

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testElasticsearchIgnoredWithViolations FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.ThirdPartyAuditTaskIT > testElasticsearchIgnored FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at ThirdPartyAuditTaskIT.java:32

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testNamingConvention FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:50

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testValidSetupWithBaseClass FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at TestingConventionsTasksIT.java:103

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testValidSetupWithoutBaseClass FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure at TestingConventionsTasksIT.java:96

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testTaskNotImplementBaseClass FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:83

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testNoEmptyTasks FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:63

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testTestsInMain FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:111

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testAllTestTasksIncluded FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:73

org.elasticsearch.gradle.precommit.TestingConventionsTasksIT > testInnerClasses FAILED
    java.lang.AssertionError at TestingConventionsTasksIT.java:36

For example ThirdPartyAuditTaskIT.java:32 is the .build() line in:

        // Build the sample jars
        getGradleRunner("thirdPartyAudit")
            .withArguments("build", "-s")
            .build();

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

The root cause looks to be:

Starting process 'Gradle Test Executor 1'. Working directory: /private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/junit12486434234967271624/build/testrun/test Command: /var/lib/jenkins/.java/java11/bin/java -Dcompiler.java=12 -Des.scripting.update.ctx_in_params=false -Dgradle.dist.lib=/private/var/lib/jenkins/.gradle/wrapper/dists/gradle-5.3-all/a2o9bs8pjyp10tdbmwhtqkmvn/gradle-5.3/lib -Dgradle.user.home=/private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/.gradle-test-kit-jenkins -Dgradle.worker.jar=/private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/.gradle-test-kit-jenkins/caches/5.3/workerMain/gradle-worker.jar -Djava.awt.headless=true -Djna.nosys=true -Dorg.gradle.native=false -Druntime.java=11 -Dtests.artifact=junit12486434234967271624 -Dtests.gradle=true -Dtests.logger.level=WARN -Dtests.security.manager=true -Dtests.seed=8203BB77DE43B000 -Dtests.task=:test -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/junit12486434234967271624/build/heapdump --illegal-access=warn -esa @/private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/gradle-worker-classpath1409477605981861779txt -Xms512m -Xmx512m -Dfile.encoding=UTF-8 -Djava.io.tmpdir=./temp -Duser.country=US -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Error: Could not find or load main class worker.org.gradle.process.internal.worker.GradleWorkerMain
Caused by: java.lang.ClassNotFoundException: worker.org.gradle.process.internal.worker.GradleWorkerMain

and

* What went wrong:
Failed to deserialize script metadata extracted for settings file '/private/var/lib/jenkins/workspace/elastic+elasticsearch+7.0+multijob-darwin-compatibility/buildSrc/src/testKit/thirdPartyAudit/settings.gradle'
> /private/var/folders/48/bbwtd6jn6dd63rmw5xmjy92r0000ln/T/.gradle-test-kit-jenkins/caches/5.2.1/scripts-remapped/settings_asgr2bfdslrv0xqfddiv0axyf/5g7v71l2e5s5r61n300m5k8ew/cp_settings3ca90766b0adfce53d4b035e7e9dc5fe/metadata/metadata.bin (No such file or directory)

All of the failing jobs seem to be on worker-c07904vtjrv2.
We should try cleaning up the Gradle cache, I'we seen errors like this when it somehow gets in a corrupt state.

[davidkyle]

Another failure https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-darwin-compatibility/138/console

Caused by: java.lang.IllegalStateException: Failed to deserialize script metadata extracted for build file '/private/var/lib/jenkins/workspace/elastic+elasticsearch+6.7+multijob-darwin-compatibility/buildSrc/src/testKit/elasticsearch-build-resources/build.gradle'

Caused by: java.io.FileNotFoundException: /private/var/folders/90/q3z4wpxd7y3333wk4mg4xrxm0000ln/T/.gradle-test-kit-jenkins/caches/5.2.1/scripts-remapped/build_5yhxgcluc261tm999dpeyto2/egk3o6tjremzgcy273hqq3zg8/cp_proj3ca90766b0adfce53d4b035e7e9dc5fe/metadata/metadata.bin (No such file or directory)

It appears the original failure was resolved by cleaning the ~/.gradle in elastic/infra#11061 I will follow up with infra

[danielmitterdorfer]

We have several more of those test failures (e.g.) in:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-darwin-compatibility/150/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+multijob-darwin-compatibility/150/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-darwin-compatibility/149/console

The details remain the same as in the comments above so I will not duplicate already known information. I'll also follow-up with infra to clean the Gradle cache on the affected worker as an immediate measure.

[alpar-t]

seem related to #gradle/gradle#4506
Work around seems to be to clean up the .gradle-test-kit directory, not sure where this is stored on macs will have to search for it.

[benwtrent]

More issues, all following the same pattern as previously mentioned

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-darwin-compatibility/153/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/379/console

Another build failure. Exactly the same errors.

[mark-vieira]

Work around seems to be to clean up the .gradle-test-kit directory, not sure where this is stored on macs will have to search for it.

This is going to be relative to the project directory. So it should generally never be reused. You can explicitly specify this though. So we could use a new temp dir for every test.

[davidkyle]

Another with the 'failed to deserialize...' error

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-darwin-compatibility/154/console

Worker https://elasticsearch-ci.elastic.co/computer/worker-c07905ngjrt9

[droberts195]

This failed today in 6.7: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-darwin-compatibility/177/console

The 6.7 branch will soon be obsolete, so there is no point fixing this in that branch. However, there will be a benefit in fixing this in 6.8, otherwise this problem will plague us for the next year or more.

I see there is a PR open to fix this, but it is currently not labelled with 6.8. Would it be possible to backport the fix to the 6.8 branch too?