[CI] UpgradeClusterClientYamlTestSuiteIT failure

[bizybot]

Build Link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/296/console

Reproduces locally with command

./gradlew :qa:rolling-upgrade:v6.6.0#oldClusterTestRunner \
 -Dtests.seed=C8ADCC320FE24AF4 \
 -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
 -Dtests.method="test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade}" \
 -Dtests.security.manager=true \
 -Dtests.locale=es-MX \
 -Dtests.timezone=Zulu \
 -Dcompiler.java=11 \
 -Druntime.java=8 \
 -Dtests.rest.suite=old_cluster

  1> [2018-11-23T22:28:44,531][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "error" : {
  1>         "root_cause" : [
  1>           {
  1>             "type" : "repository_verification_exception",
  1>             "reason" : "[my_repo] path  is not accessible on master node",
  1>             "stack_trace" : "RepositoryVerificationException[[my_repo] path  is not accessible on master node]; nested: IOException[snapshot.test];
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:647)
  1>    at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:221)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627)
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  1>    at java.lang.Thread.run(Thread.java:834)
  1> Caused by: java.io.IOException: snapshot.test
  1>    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
  1>    at java.net.Socket.connect(Socket.java:591)
  1>    at java.net.Socket.connect(Socket.java:540)
  1>    at sun.net.NetworkClient.doConnect(NetworkClient.java:182)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
  1>    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:341)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:362)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1242)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1075)
  1>    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1009)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
  1>    at java.net.URL.openStream(URL.java:1117)
  1>    at java.security.AccessController.doPrivileged(Native Method)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.getInputStream(URLBlobContainer.java:118)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.readBlob(URLBlobContainer.java:104)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:816)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:807)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:633)
  1>    ... 5 more
  1> "
  1>           }
  1>         ],
  1>         "type" : "repository_verification_exception",
  1>         "reason" : "[my_repo] path  is not accessible on master node",
  1>         "caused_by" : {
  1>           "type" : "i_o_exception",
  1>           "reason" : "snapshot.test",
1>           "stack_trace" : "java.io.IOException: snapshot.test
  1>    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
  1>    at java.net.Socket.connect(Socket.java:591)
  1>    at java.net.Socket.connect(Socket.java:540)
  1>    at sun.net.NetworkClient.doConnect(NetworkClient.java:182)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
  1>    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:341)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:362)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1242)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1075)
  1>    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1009)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
  1>    at java.net.URL.openStream(URL.java:1117)
  1>    at java.security.AccessController.doPrivileged(Native Method)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.getInputStream(URLBlobContainer.java:118)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.readBlob(URLBlobContainer.java:104)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:816)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:807)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:633)
  1>    at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:221)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627)
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  1>    at java.lang.Thread.run(Thread.java:834)
  1> "
  1>         },
  1>         "stack_trace" : "RepositoryVerificationException[[my_repo] path  is not accessible on master node]; nested: IOException[snapshot.test];
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:647)
  1>    at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:221)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627)
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  1>    at java.lang.Thread.run(Thread.java:834)
  1> Caused by: java.io.IOException: snapshot.test
  1>    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
  1>    at java.net.Socket.connect(Socket.java:591)
  1>    at java.net.Socket.connect(Socket.java:540)
  1>    at sun.net.NetworkClient.doConnect(NetworkClient.java:182)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
  1>    at sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
  1>    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:341)
  1>    at sun.net.www.http.HttpClient.New(HttpClient.java:362)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1242)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
  1>    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1075)
  1>    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1009)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1581)
  1>    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
  1>    at java.net.URL.openStream(URL.java:1117)
  1>    at java.security.AccessController.doPrivileged(Native Method)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.getInputStream(URLBlobContainer.java:118)
  1>    at org.elasticsearch.common.blobstore.url.URLBlobContainer.readBlob(URLBlobContainer.java:104)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:816)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:807)
  1>    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:633)
  1>    ... 5 more
  1> "
  1>       },
  1>       "status" : 500
  1>     }
  1>   }
  1> }]
1> [2018-11-23T22:28:44,580][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] [p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade]

[original-brownbear]

This seems to be failing 100% of the time now (on Jenkins as well as locally). Trying to fix it now

[original-brownbear]

@ywelsch I think this commit to 6.x 10796ea is causing this (you made a change to the blob repo verification there that is not throwing in the master BwC tests).

[original-brownbear]

Maybe we could merge #35876 to get builds green again until this is fixed in 6.x?

[original-brownbear]

Muted in master for now via 05620cc

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

I've looked at the cause of this: The problem with this failure was that it only occurred once the PR #35731 was backported to 6.6, as the rolling-upgrade tests on master would now be doing something illegal with a 6.6 node. This is unfortunately impossible to catch with our standard PR testing, as we typically don't test the combination of master + backport. I have taken the fix by @original-brownbear and backported that fix to 6.x (6.6) as well, as it will otherwise break again once we have 6.6.1 or 6.7 running rolling-upgrade tests from 6.6.