failed to create S3 Repository using dockerized ES 6.2.2 due to SDK connect timed out

[jlamillan]

I've been getting a number of errors when trying to create an S3 repository on my local box using Docker. Note, the same steps appear to work when elasticsearch is run outside of Docker (using brew).

Here is the latest:

Elasticsearch version: 6.2.2

Plugins installed: [ingest-geoip, ingest-user-agent, repository-s3, discovery-file, x-pack]

JVM version: openjdk version "1.8.0_161"

OS version: Docker / CentOS Linux release 7.4.1708 (Core)

Description of the problem including expected versus actual behavior:
Getting an error when trying to create a S3 Repository.

Steps to reproduce:

1. Install the repository-s3 plugin, start the elasticsearch container:

mac$ docker run -e AWS_ACCESS_KEY_ID=XXXXXXXXXXX -e AWS_SECRET_KEY=XXXXXXXXXXX docker.elastic.co/elasticsearch/elasticsearch:6.2.2 /bin/sh -c "elasticsearch-plugin install --batch repository-s3 && /usr/local/bin/docker-entrypoint.sh"

mac$ curl http://127.0.0.1:9200/_cat/health
1520277749 19:22:29 docker-cluster green 1 1 1 1 0 0 0 0 - 100.0%

2. Set the s3 credentials in the elasticsearch keystore (default client):

container# elasticsearch-keystore add s3.client.default.access_key XXXXXXXXXXX
container# elasticsearch-keystore add s3.client.default.secret_key XXXXXXXXXXX

container# elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

3. Attempt to create an S3 Repository (default client):

container# curl -XPUT "http://localhost:9200/_snapshot/my_s3_repository1" -H'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-index-storage-amazon"
  }
}'

Provide logs (if relevant):

Error from create S3 Repository PUT (i.e. step 3 from above):

{"error":{"root_cause":[{"type":"repository_exception","reason":"[my_s3_repository1] failed to create repository"}],"type":"repository_exception","reason":"[my_s3_repository1] failed to create repository","caused_by":{"type":"sdk_client_exception","reason":"Unable to load credentials from service endpoint","caused_by":{"type":"socket_timeout_exception","reason":"connect timed out"}}},"status":500}[root@cb5e3dd9d478 

From server:

[2018-03-03T23:58:48,611][INFO ][o.e.c.m.MetaDataCreateIndexService] [hrK-wmC] [.monitoring-es-6-2018.03.03] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2018-03-03T23:58:48,946][INFO ][o.e.c.r.a.AllocationService] [hrK-wmC] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.03.03][0]] ...]).
[2018-03-04T00:00:08,439][INFO ][o.e.c.m.MetaDataCreateIndexService] [hrK-wmC] [.monitoring-es-6-2018.03.04] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2018-03-04T00:00:08,475][INFO ][o.e.c.r.a.AllocationService] [hrK-wmC] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.03.04][0]] ...]).
[2018-03-04T00:06:43,320][WARN ][o.e.r.RepositoriesService] [hrK-wmC] failed to create repository [s3][my_s3_repository1]
com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	... 44 more
[2018-03-04T00:06:43,345][WARN ][o.e.r.RepositoriesService] [hrK-wmC] failed to create repository [my_s3_repository1]
org.elasticsearch.repositories.RepositoryException: [my_s3_repository1] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:389) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
[2018-03-04T00:06:43,383][WARN ][r.suppressed             ] path: /_snapshot/my_s3_repository1, params: {repository=my_s3_repository1}
org.elasticsearch.repositories.RepositoryException: [my_s3_repository1] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:389) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more

[javanna]

cc @elastic/es-distributed

[ywelsch]

Problem and solution should be explained in https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/
Gist of it: All the modifications to the keystore take affect only after restarting Elasticsearch.

[jlamillan]

Sounds like a bug.

[windowsrefund]

Can anyone confirm the following link as being the fix for this? #30652

[ebuildy]

Using version 7.15.X, still see java.net.SocketTimeoutException: connect timed out wrong error message, whereas this was a credentials missing issue