[BUG] Registering MinIO (S3) snapshot repository fails with "Connect timed out" #16305
Description
Describe the bug
I’m running OpenSearch as part of Graylog Helm installation under Kubernetes. I’m trying to register a snapshot endpoint with MinIO. I’m following this document: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/
When I try to register the repository with curl (using the REST API), I get "Connect timed out" error. Using tcpdump I can see that no connection to provided IP address is attempted. When I manually test the connection to MinIO with curl, it works. (I.e. it’s not a network issue.)
If I remove s3.client.default.endpoint setting, I can see OpenSearch connecting to Amazon servers. (Which is not what I want.)
I suspect this might be just a misconfiguration, but no matter what I try, I get the same results.
Related component
Plugins
To Reproduce
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore create
An opensearch keystore already exists. Overwrite? [y/N]y
Created opensearch keystore in /usr/share/opensearch/config/opensearch.keystore
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:
[opensearch@opensearch-cluster-master-0 ~]$ grep s3.client.default config/opensearch.yml
s3.client.default.protocol: "http"
s3.client.default.endpoint: "http://1.2.3.4:9000/"
s3.client.default.path_style_access: "true"
Did steps above on all 3 cluster members.
[opensearch@opensearch-cluster-master-0 ~]$ curl -X POST "http://localhost:9200/_nodes/reload_secure_settings"
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"opensearch-cluster","nodes":{"Ug2a4ZiqS_6sNDvKlFRNbg":{"name":"opensearch-cluster-master-2"},"zi7xQcAsT0WyPEXLozMEJQ":{"name":"opensearch-cluster-master-0"},"R6I3MgjqRrS85OjyIWHCaw":{"name":"opensearch-cluster-master-1"}}}[opensearch@opensearch-cluster-master-0 ~]$
[opensearch@opensearch-cluster-master-0 ~]$ curl -X PUT "http://localhost:9200/_snapshot/minio-repo?pretty" -H 'Content-Type: application/json' -d '
{
"type": "s3",
"settings": {
"bucket": "opensearch",
"base_path": "opensearch/snapshot/"}
}'
{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[minio-repo] path [opensearch/snapshot/] is not accessible on cluster-manager node"
}
],
"type" : "repository_verification_exception",
"reason" : "[minio-repo] path [opensearch/snapshot/] is not accessible on cluster-manager node",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [opensearch/snapshot//tests-nZNGJ5szRh-Pd5gX3q44dA/master.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
"status" : 500
}
tcpdump shows no traffic to MinIO
Test if the Minio endpoint is reachable:
[opensearch@opensearch-cluster-master-0 ~]$ curl http://1.2.3.4:9000/
AccessDeniedAccess Denied./minio17FE44A7FEAD5E72dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8[opensearch@opensearch-cluster-master-0 ~]$
tcpdump shows connection with MinIO was established
Expected behavior
Snapshot endpoint should be successfully registered, allowing me to make snapshots and recoveries.
Additional Details
Plugins
plugins:
enabled: true
installList:
- repository-s3
Host/Environment (please complete the following information):
- OS: Ubuntu Server
- Version: 22.04, kernel 5.15.0-102-generic
Additional context
Kubernetes: v1.28.14
Containerd: 1.7.2-0ubuntu1~22.04.1
Docker image: opensearchproject/opensearch:2.4.0
Helm chart: graylog-2.3.10 - uses https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch