[CI] QueueResizingEsThreadPoolExecutor assertion fails if task time not available

[droberts195]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/3245/console failed due to an assertion failure:

apr. 23, 2019 4:15:30 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[elasticsearch[mock_ node][search][T#1],5,TGRP-NodeTests]
java.lang.AssertionError: expected task to always take longer than 0 nanoseconds, got: -1
	at __randomizedtesting.SeedInfo.seed([5F8BB618AC0348C7]:0)
	at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.afterExecute(QueueResizingEsThreadPoolExecutor.java:154)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

The assertion that tripped is:

        assert taskExecutionNanos >= 0 : "expected task to always take longer than 0 nanoseconds, got: " + taskExecutionNanos;

The implementation of the method that is used to get taskExecutionNanos is:

    long getTotalExecutionNanos() {
        if (startTimeNanos == -1 || finishTimeNanos == -1) {
            // There must have been an exception thrown, the total time is unknown (-1)
            return -1;
        }
        return Math.max(finishTimeNanos - startTimeNanos, 1);
    }

So it looks like that assertion will fail if an exception occurs that results in the total time of a task being unknown.

[elasticmachine]

Pinging @elastic/es-core-infra

[andyb-elastic]

I muted org.elasticsearch.node.NodeTests.testCloseOnInterruptibleTask which was affected by this

[jaymode]

We discussed this in the core infra meeting and believe that the assertion is faulty and needs to take the exceptional situation into account.