[BUG] repository-s3 plugin is broken in mainline

[rishabh6788]

Describe the bug

Getting below error while trying to register an s3 repository using latest build for 3.0.0-alpha1

[ec2-user@ip-172-31-23-58 default]$ curl -X PUT "http://opense-clust-***********-692e187bb21cd8d5.elb.us-east-1.amazonaws.com/_snapshot/benchmark-workloads-repo-3x" -H "Content-Type: application/json" -d '
> {
>   "type": "s3",
>   "settings": {
>     "bucket": "benchmark-***********",
>     "base_path": "workload-*******",
>     "region": "*****"
>   }
> }'

{"error":{"root_cause":[{"type":"repository_exception","reason":"[benchmark-workloads-repo-3x] failed to create repository"}],"type":"repository_exception","reason":"[benchmark-workloads-repo-3x] failed to create repository","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.repositories.s3.async.AsyncExecutorContainer.getStreamReader()\" because \"this.normalExecutorBuilder\" is null"}},"status":500}[

Related component

Plugins

To Reproduce

1. Download the latest tar for 3.0.0-alpha1 version.
2. Install respository-s3 plugin and make sure the required credentials and permissions are configured.
3. register a snapshot repository
4. See error

Expected behavior

The cluster should return 200 response.

Additional Details

Use tarball https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-alpha1-SNAPSHOT/opensearch-min-3.0.0-alpha1-SNAPSHOT-linux-x64-latest.tar.gz

[prudhvigodithi]

Hey @rishabh6788 in past I have implemented snapshot capabilities for the OpenSearch K8s Operator opensearch-project/opensearch-k8s-operator#278, this issue has some details related to IAM role and commands, so AFAIK the snapshot repo registration and creating of snapshot is for sure working in past.

[reta]

@rishabhmaurya I suspect it is caused by #16962, could you please double check? thank you wrong pointers, sorry about that

[rishabh6788]

@reta Above PR was merged yesterday, I am experiencing the same error on a tarball created on Feb-10th as well.

[rishabh6788]

Hey @rishabh6788 in past I have implemented snapshot capabilities for the OpenSearch K8s Operator opensearch-project/opensearch-k8s-operator#278, this issue has some details related to IAM role and commands, so AFAIK the snapshot repo registration and creating of snapshot is for sure working in past.

it was working fine up until 10-days back. Last successful run was with commit-id b823d1f794c27f92c257128b62b955ae10cf7f6c. The next run which started failing had commit -d d0a65d3b457e11d514171ecc0a77450fdd7d25bb.
Something broke in between these two commits.

[rishabh6788]

For more context, I encountered this in our nightly benchmark workflow, that runs on the latest build of mainline daily.
The cluster ec2 has instance role attached that has appropriate policy required for respository-s3 plugin to work.

[rishabh6788]

@peterzhuamazon The repository-s3 plugin on all alpha builds is broken.

[peterzhuamazon]

Hi @reta @andrross @expani any lead on fixing this issue?

cc: @getsaurabh02

Thanks!

[andrross]

@peterzhuamazon Are we properly pulling the alpha1 version of the repository plugin when doing opensearch-plugin install respository-s3? I'm not able to reproduce this issue when I install the plugin from a zip file directly build from source, but I can reproduce when just installing by name where it downloads the zip.