[CI] BindTransportException Failed to resolve host - getFlags failes

[henningandersen]

A PR build failed: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12738/ . The actual test that this happened to seems unrelated, so did not mute anything. Could be an infra, a build infra issue or some random issue. I did try to run the specific tests 1000s of times, but did not reproduce.

This seems to be the primary problem:

org.elasticsearch.analysis.common.HighlighterWithAnalyzersTests > testPhrasePrefix FAILED
    java.lang.RuntimeException: failed to start nodes

        Caused by:
        java.util.concurrent.ExecutionException: BindTransportException[Failed to resolve host []]; nested: SocketException[No such device (getFlags() failed)];

            Caused by:
            BindTransportException[Failed to resolve host []]; nested: SocketException[No such device (getFlags() failed)];

                Caused by:
                java.net.SocketException: No such device (getFlags() failed)

consoleText.txt.gz

[elasticmachine]

Pinging @elastic/es-core-infra

[albertzaharovits]

Another blip here https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/13111/

Caused by: java.util.concurrent.ExecutionException: BindTransportException[Failed to resolve host []]; nested: SocketException[No such device (getFlags() failed)];
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.elasticsearch.test.InternalTestCluster.startAndPublishNodesAndClients(InternalTestCluster.java:1674)
	... 43 more
Caused by: BindTransportException[Failed to resolve host []]; nested: SocketException[No such device (getFlags() failed)];
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:331)
	at org.elasticsearch.transport.nio.MockNioTransport.doStart(MockNioTransport.java:110)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:230)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)
	at org.elasticsearch.node.Node.start(Node.java:661)
	at org.elasticsearch.test.InternalTestCluster$NodeAndClient.startNode(InternalTestCluster.java:961)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: java.net.SocketException: No such device (getFlags() failed)
	at java.net.NetworkInterface.isUp0(Native Method)
	at java.net.NetworkInterface.isUp(NetworkInterface.java:399)
	at org.elasticsearch.common.network.NetworkUtils.getLoopbackAddresses(NetworkUtils.java:157)
	at org.elasticsearch.common.network.NetworkService.resolveInternal(NetworkService.java:212)
	at org.elasticsearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:192)
	at org.elasticsearch.common.network.NetworkService.resolveBindHostAddresses(NetworkService.java:108)
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:329)
	... 12 more

[henningandersen]

One more here:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12724/testReport/org.elasticsearch.index.search.nested/NestedSortingTests/testEmpty/

BindTransportException[Failed to resolve publish address]; nested: SocketException[No such device (getFlags() failed)];
	at org.elasticsearch.transport.TcpTransport.createBoundTransportAddress(TcpTransport.java:413)
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:348)
	at org.elasticsearch.transport.nio.MockNioTransport.doStart(MockNioTransport.java:109)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:230)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59)
	at org.elasticsearch.node.Node.start(Node.java:663)
	at org.elasticsearch.test.ESSingleNodeTestCase.newNode(ESSingleNodeTestCase.java:225)
	at com.carrotsearch.randomizedtesting.RandomizedContext.runWithPrivateRandomness(RandomizedContext.java:187)
	at com.carrotsearch.randomizedtesting.RandomizedContext.runWithPrivateRandomness(RandomizedContext.java:211)
	at org.elasticsearch.test.ESSingleNodeTestCase.startNode(ESSingleNodeTestCase.java:81)
	at org.elasticsearch.test.ESSingleNodeTestCase.setUp(ESSingleNodeTestCase.java:120)
	at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketException: No such device (getFlags() failed)
	at java.base/java.net.NetworkInterface.isUp0(Native Method)
	at java.base/java.net.NetworkInterface.isUp(NetworkInterface.java:446)
	at org.elasticsearch.common.network.NetworkUtils.getLoopbackAddresses(NetworkUtils.java:157)
	at org.elasticsearch.common.network.NetworkService.resolveInternal(NetworkService.java:207)
	at org.elasticsearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:187)
	at org.elasticsearch.common.network.NetworkService.resolvePublishHostAddresses(NetworkService.java:143)
	at org.elasticsearch.transport.TcpTransport.createBoundTransportAddress(TcpTransport.java:411)
	... 45 more

[alpar-t]

This does not seem to be specific to any platform, failures are across the board.
Also started happening in CI only after 24th of April, https://scans.gradle.com/s/34tn7dovywvym being the first occurrence.
There's no change in neither infra nor elasticsearch around that time that sticks out.

[alpar-t]

We enabled --parallel the day before so that's likely the primary suspect right now.

[jaymode]

We discussed this in yesterday's core infra meeting. @jasontedor did some investigation and thinks that maybe a network interface appeared then disappeared during startup of the node. One suggestion was to change the logic and only check if the interface is up if the interface has a loopback address. Additionally, the exception message can be enhanced to include the network interface to help in understanding this more if it continues to happed.