[CI] RankEvalIT failing on ARM workers

[mark-vieira]

In the process of getting tests running on ARM I'm seeing failures in RankEvalIT for our rest client.

org.elasticsearch.client.RankEvalIT > testMetrics FAILED
    java.lang.AssertionError: expected:<1.6408962261063627> but was:<1.6408962261063629>
        at __randomizedtesting.SeedInfo.seed([43A5ADA58EB0903B:BDB77F0792BAE788]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:553)
        at org.junit.Assert.assertEquals(Assert.java:683)
        at org.elasticsearch.client.RankEvalIT.testMetrics(RankEvalIT.java:132)

The Elasticsearch node logs are full of errors like this:

» [2021-02-11T18:27:16,545][ERROR][o.e.x.m.j.p.a.JobModelSnapshotUpgrader] [asyncIntegTest-0] [test-upgrade-job-model-snapshot] Unexpected exception writing to process
»  org.elasticsearch.ElasticsearchException: [test-upgrade-job-model-snapshot] Unexpected death of autodetect: Failed to find valid JSON
»  Expected persisted state but no state exists
»  Failed to restore detectors
»  Failed to restore state
»  ML anomaly detector job failed
»  
»  	at org.elasticsearch.xpack.ml.job.process.autodetect.JobModelSnapshotUpgrader$Executor.checkProcessIsAlive(JobModelSnapshotUpgrader.java:342) ~[x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.xpack.ml.job.process.autodetect.JobModelSnapshotUpgrader$Executor$1.doRun(JobModelSnapshotUpgrader.java:332) [x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectWorkerExecutorService.start(AutodetectWorkerExecutorService.java:95) [x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
»  	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:669) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
»  	at java.lang.Thread.run(Thread.java:834) [?:?]

Perhaps something is misconfigured here? Any thoughts?

https://gradle-enterprise.elastic.co/s/rrggex4kkpph6/console-log/raw?task=:client:rest-high-level:asyncIntegTest

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

I think there are two separate problems here. The failure in RankEvalIT is not ML. The difference is in the last significant figure and will be due to very slight floating point differences between ARM and x86. We did coincidentally see similar things in the ML C++ unit tests. Assertions on doubles on ARM need to be to a maximum of 14 significant figures, not 15, as the results differ from x86 in the last significant figure. I think the correct area label for this test is :Search/Ranking.

The ML problem in the server side logs is something different. We will have to look at which test that relates to.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[mark-vieira]

Thanks for investigating, David!

[droberts195]

The ML problem in the server side logs is something different. We will have to look at which test that relates to.

I had a look and that error message from ML is not actually a problem. It relates to the discussion in #64665 (comment). Testing that the ML model snapshot upgrader works is so complicated and time consuming that we didn't want to duplicate it in the HLRC tests just to test that the client APIs work as expected. So the tests of the model snapshot upgrader in the HLRC tests actually test an invalid upgrade. This still proves that the communications between the HLRC and server work as expected, but avoid the need for the test to somehow create valid ML model state from a previous version of the product. So I'll remove the ML labels from this issue.

The RankEvalIT problem is caused by using Double.MIN_VALUE as the comparison tolerance in

elasticsearch/client/rest-high-level/src/test/java/org/elasticsearch/client/RankEvalIT.java

Line 132 in 985ae9a

assertEquals(expectedScores[i], response.getMetricScore(), Double.MIN_VALUE);

If that's replaced with 1e-14 then the test should pass. Double.MIN_VALUE is of the order of 1e-308, so far too low to have any impact on comparisons around a value of 1. I am not sure if that pattern has been used widely through the codebase but if it has then there could be a lot of places that need changing for the tests to pass on ARM. Anywhere where Double.MIN_VALUE has been used for a double precision assertion tolerance it really should be replaced with a tolerance that means that at most the 14 most significant figures of expected and actual get compared.

[cbuescher]

Double.MIN_VALUE is of the order of 1e-308, so far too low to have any impact on comparisons

Agreed, it doesn't make sense for this test to have such a strict comparison. I will lower the bound and open a PR.